Training data collection platform, method and system for graphical user interface agent models

Abstract

This application provides a training data acquisition platform, method, and system for a graphical user interface (GUI) proxy model, relating to the field of artificial intelligence technology. The data acquisition platform includes at least a first task generation module, a sample data generation module, and a sample data evaluation module. After the first task generation module acquires a large number of user tasks from the network side, the sample data generation module calls a first model and interacts with the testing device through the first model to construct sample data corresponding to the user tasks. For each set of constructed sample data, the correctness of the operation sequence in the sample data is evaluated using a data sample evaluation model to determine whether the sample data can be used as training data for the GUI proxy model. Compared to the traditional method of relying on manual operation to construct training data, this significantly improves the efficiency of model training data acquisition and the quality of the training data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training data acquisition platform, method and system for a graphical user interface agent model. Background Technology

[0002] The application of intelligent agents based on multimodal large models is becoming increasingly widespread in the field of automated device interaction. Multimodal large models can understand and process various types of data input, such as text and images. In the field of graphical user interfaces (GUIs), intelligent agents invoke multimodal large models to parse interface elements, understand user intent, and achieve automated interaction with device applications, automatically completing user tasks. For example, if a user inputs the task "Find flight tickets from destination 1 to destination 2 tomorrow on application A," the intelligent agent invokes the multimodal large model, executes the operation instructions output by the model, and completes the user task.

[0003] The accurate execution of user tasks by intelligent agents relies on the training of a large multimodal model, which requires a large amount of high-quality operation trajectory data. Operation trajectory data can be understood as a collection of continuous operation records (e.g., clicks, swipes, inputs) and behavioral paths generated when the intelligent agent executes user tasks on the GUI. A complete operation trajectory data usually corresponds to the complete process of completing a user task.

[0004] Currently, the methods for collecting trajectory data for training mainly rely on manual operation, which is difficult to meet actual needs, especially in terms of covering high-frequency business paths, ensuring data consistency, and improving collection efficiency. Summary of the Invention

[0005] This application provides a training data acquisition platform, method, and system for a graphical user interface (GUI) agent model, which improves the efficiency of model training data acquisition and the quality of training data.

[0006] In a first aspect, embodiments of this application propose a training data acquisition platform for a GUI agent model (hereinafter referred to as the data acquisition platform), comprising: a first task generation module, a sample data generation module, and a sample data evaluation module; the sample data generation module is connected to the first task generation module and the sample data evaluation module, respectively.

[0007] The first task generation module is used to obtain user tasks from multiple application servers; the sample data generation module is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model; send operation instructions to the test device and obtain the interface data corresponding to the operation instructions from the test device; construct sample data based on the user task, operation sequence and the interface data corresponding to each operation instruction in the operation sequence; the sample data evaluation module is used to evaluate the correctness of the operation sequence in the sample data and determine whether the sample data is positive example data; if the sample data is positive example data, the sample data is used as training data for the GUI proxy model.

[0008] In this embodiment, the GUI proxy model is used to generate operation instructions and operation instruction descriptions corresponding to the task based on the task input by the user.

[0009] In this embodiment, the multiple application servers include servers for multiple target applications. Target applications can be applications with a higher number of user visits (or downloads) within a preset time period (e.g., the past six months or the past month) than a preset number (i.e., popular applications), or applications with a higher average number of user visits per day within a preset time period (e.g., daily) than a preset number (i.e., applications with high daily user activity). It is understood that since user tasks are obtained from multiple application servers, the user tasks involve various high-frequency business scenarios.

[0010] In this embodiment, the operation sequence corresponding to the user task includes one or more operation instructions. Each time the sample data generation module calls the first model, it can obtain one operation instruction from the operation sequence corresponding to the user task. By repeatedly calling the first model, all operation instructions in the operation sequence corresponding to the user task are obtained until the user task is completed. The interface data corresponding to each operation instruction in the operation sequence includes at least the interface data after the test device executes the operation instruction, and the interface data includes a screenshot. In some embodiments, the interface data also includes an interface description document, which is used to indicate the interface structure, layout, interface element attributes, and hierarchical relationships, etc.

[0011] In this embodiment, the sample data is positive example data, which indicates that the user task and operation sequence in the sample data are matched with each other, that is, the user task can be successfully completed by executing the operation sequence through the test device.

[0012] The data acquisition platform described above, after acquiring a large number of user tasks from the network side, calls a pre-trained first model. This first model interacts with the testing device to construct sample data corresponding to the user tasks. For each constructed set of sample data, the correctness of the operation sequences within the sample data is evaluated to determine whether the sample data can be used as training data for a GUI proxy model to optimize model performance. Compared to the traditional method of relying on manual operation to construct training data, this significantly improves the efficiency of model training data acquisition and the quality of the training data.

[0013] In an optional embodiment of the first aspect, the first model includes a pre-trained GUI agent model; the sample data generation module is used to obtain operation instructions from the operation sequence corresponding to the user task by invoking the first model, including:

[0014] The sample data generation module is used to call the GUI proxy model and obtain the operation instructions and operation instruction descriptions from the operation sequence corresponding to the user task output by the GUI proxy model.

[0015] In one optional embodiment of the first aspect, the input of the GUI agent model includes user tasks, historical operations, and a screenshot of the current interface, and the output of the GUI agent model includes at least one operation instruction and an operation instruction description corresponding to at least one operation instruction.

[0016] In some embodiments, the model parameters of the GUI proxy model include a first parameter, which indicates the diversity of operation instructions output by the GUI proxy model. If the first parameter is set to a third value, the output of the GUI proxy model includes one operation instruction and a corresponding operation instruction description. If the first parameter is set to a fourth value, the output of the GUI proxy model includes at least two operation instructions and corresponding operation instruction descriptions for each operation instruction.

[0017] In one optional embodiment of the first aspect, the first model includes a large language model and a pre-trained first instruction generation model; the sample data generation module is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model, including: the sample data generation module is used to obtain the operation instruction description corresponding to the user task output by the large language model by calling the large language model; input the operation instruction description into the first instruction generation model, and obtain the operation instruction corresponding to the operation instruction description output by the first instruction generation model.

[0018] For example, large language models include generative pre-trained transformer (GPT) models.

[0019] In some embodiments, the data acquisition platform includes a pre-trained first instruction generation model, which corresponds to the instruction generation model 41 described below.

[0020] In the above scheme, the data acquisition platform first obtains the operation instruction description corresponding to the user task by calling an external large language model, and then calls a locally pre-trained first instruction generation model to obtain the operation instruction corresponding to the operation instruction description. This operation instruction is the operation instruction in the operation sequence corresponding to the user task. The above calling process is repeated to obtain other operation instructions in the operation sequence corresponding to the user task until the user task is completed.

[0021] In an optional embodiment of the first aspect, the first model includes a pre-trained instruction description generation model and a pre-trained second instruction generation model; the sample data generation module is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model, including:

[0022] The sample data generation module is used to call the instruction description generation model to obtain the operation instruction description corresponding to the user task output by the instruction description generation model; input the operation instruction description into the second instruction generation model to obtain the operation instruction corresponding to the operation instruction description output by the second instruction generation model.

[0023] In some embodiments, the data acquisition platform includes an instruction description generation model and a second instruction generation model. The second instruction generation model may correspond to instruction generation model 52 described later.

[0024] In the above scheme, the data acquisition platform first obtains the operation instruction description corresponding to the user task by calling a locally pre-trained instruction description generation model. Then, it calls a locally pre-trained second instruction generation model to obtain the operation instruction corresponding to the operation instruction description. This operation instruction is the operation instruction in the operation sequence corresponding to the user task. The above calling process is repeated to obtain other operation instructions in the operation sequence corresponding to the user task until the user task is completed.

[0025] In one alternative embodiment of the first aspect, the first model includes a GUI agent model, a large language model, a pre-trained first instruction generation model, a pre-trained instruction description generation model, and a pre-trained second instruction generation model.

[0026] The sample data generation module is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model, including: the sample data generation module calls the GUI proxy model to obtain the first operation instruction corresponding to the user task; and sequentially calls the large language model and the first instruction generation model to obtain the second operation instruction corresponding to the user task; and sequentially calls the instruction description generation model and the second instruction generation model to obtain the third operation instruction corresponding to the user task.

[0027] The sample data generation module is also used to select the operation instruction with the highest confidence level from the first operation instruction, the second operation instruction, and the third operation instruction as the final nth operation instruction; the confidence level is used to indicate the probability value that the operation instruction is a correct operation instruction.

[0028] Among them, the first operation instruction, the second operation instruction, and the third operation instruction are all candidate instructions for the nth operation instruction in the operation sequence corresponding to the user task, where n is a positive integer.

[0029] In the above scheme, the data acquisition platform obtains different model call paths (including, for example, multiple locally pre-trained models combined with calls to external large language models) by using multiple pre-trained models locally. Figure 6 The optional operation instructions (including, for example, those shown in paths 1, 2, and 3) for user tasks are as follows: Figure 6 The operation instructions shown (1, 2, and 3) determine the target operation instruction from multiple optional operation instructions, aiming to improve the quality of the constructed sample data by increasing data diversity.

[0030] In an optional embodiment of the first aspect, the sample data evaluation module is used to evaluate the correctness of the operation sequence in the sample data and determine whether the sample data is positive example data, including:

[0031] The sample data evaluation module is used to obtain the first evaluation result of the first evaluation model for each operation instruction in the operation sequence by calling the pre-trained first evaluation model; and to obtain the second evaluation result of the second evaluation model for the operation sequence as a whole by calling the pre-trained second evaluation model.

[0032] The sample data evaluation module is used to determine whether the sample data is positive example data based on the first evaluation result and the second evaluation result.

[0033] For example, the first evaluation model can correspond to the process reward model (PRM) described later, and the second evaluation model can correspond to the outcome reward model (ORM) described later.

[0034] In the above scheme, the data acquisition platform does not directly use each set of sample data to train the GUI agent model. Instead, it calls the first evaluation model and the second evaluation model to evaluate each set of sample data, filter out sample data that is judged as negative, retain sample data that is judged as positive, and use the sample data judged as positive as the training data for the subsequent GUI agent model, thereby improving the data quality of the constructed training data and continuously optimizing the model performance.

[0035] In one optional embodiment of the first aspect, the first evaluation result includes a first score of each operation instruction in the operation sequence by the first evaluation model, the first score being used to indicate whether the operation instruction is progressing in the direction of completing the user task; the second evaluation result is used to indicate whether the final execution result of the operation sequence matches the user task.

[0036] The sample data evaluation module is used to determine whether the sample data is positive example data based on the first evaluation result and the second evaluation result, including: the sample data evaluation module is used to determine the first proportion of the number of operation instructions with the first score as the first value to the total number of operation instructions in the operation sequence based on the first evaluation result; and to determine whether the sample data is positive example data based on the relationship between the first proportion and the first threshold, and the second evaluation result.

[0037] The first score is the highest value, used to indicate that the operation instructions are progressing in the direction of completing the user's task.

[0038] In this embodiment, the first evaluation result includes the evaluation result of the first evaluation model for each operation instruction in the operation sequence corresponding to the user task (which may correspond to evaluation result 1 below). The evaluation result of the first evaluation model for each operation instruction in the operation sequence can be a binary evaluation result. For example, if the first score of an operation instruction is a first value (e.g., 1), it indicates that the operation instruction is progressing in the direction of completing the user task (i.e., the operation instruction is a correct operation instruction). If the first score of an operation instruction is a second value (e.g., 0), it indicates that the operation instruction is not progressing in the direction of completing the user task (i.e., the operation instruction is an incorrect operation instruction).

[0039] In this embodiment, the second evaluation result (which may correspond to evaluation result 2 below) can be a binary evaluation result. If the second evaluation result is a first value (e.g., 1), it indicates that the final execution result of the operation sequence corresponding to the user task matches the user task. If the second evaluation result is a second value (e.g., 0), it indicates that the final execution result of the operation sequence corresponding to the user task does not match the user task.

[0040] In some embodiments, if the first proportion is greater than or equal to the first threshold and the second evaluation result is the first value, the sample data is determined to be positive data. The first threshold can be 0.6.

[0041] In some embodiments, if the first proportion is less than or equal to the second threshold, and the second evaluation result is the second value, the sample data is determined to be negative data. The second threshold can be 0.4.

[0042] The above scheme shows how to determine whether a sample data is positive or negative based on the evaluation results of the first evaluation model and the second evaluation model. The first evaluation model is mainly used to evaluate the correctness of each operation instruction in the sample data, and the second evaluation model is mainly used to evaluate the correctness of the operation instructions in the sample data after overall execution.

[0043] In one optional embodiment of the first aspect, the data acquisition platform further includes a second task generation module; the second task generation module is connected to the sample data evaluation module.

[0044] The sample data evaluation module is used to send negative sample data to the second task generation module when the sample data is determined to be negative sample data. The second task generation module is used to obtain a new user task that matches the operation sequence in the negative sample data by calling the pre-trained task synthesis model. The sample data generation module is also used to take the operation sequence in the negative sample data, the interface data corresponding to each operation instruction in the operation sequence, and the new user task as a new set of sample data.

[0045] In some embodiments, the data acquisition platform includes a task synthesis model.

[0046] In the above scheme, the data acquisition platform can generate corresponding user tasks based on the operation sequences and interface data in the negative example data by calling the task synthesis model, thereby constructing a new set of sample data. Compared with direct negative example data, this can improve the utilization rate of negative example data and further expand the sample data for model training.

[0047] In an optional embodiment of the first aspect, the sample data evaluation module is further configured to evaluate the correctness of the new sample data by calling a pre-trained second evaluation model to determine whether the new sample data is positive data; if the new sample data is positive data, the new sample data is used as training data for the GUI agent model.

[0048] The above scheme shows that after constructing new sample data, the data acquisition platform performs a secondary evaluation on the newly constructed sample data through a sample data evaluation model to determine whether the newly constructed sample data can be used as training data.

[0049] Secondly, embodiments of this application provide a method for collecting training data for a GUI proxy model, comprising: obtaining user tasks from multiple application servers; obtaining operation instructions in the operation sequence corresponding to the user tasks by calling a first model; sending operation instructions to a test device; obtaining interface data corresponding to the operation instructions from the test device; constructing sample data based on the user tasks, operation sequences, and interface data corresponding to each operation instruction in the operation sequence; evaluating the correctness of the operation sequences in the sample data to determine whether the sample data is positive example data; and if the sample data is positive example data, using the sample data as training data for the GUI proxy model.

[0050] In an optional embodiment of the second aspect, the first model includes a pre-trained GUI agent model; by invoking the first model, operation instructions in the operation sequence corresponding to the user task are obtained, including:

[0051] By calling the GUI proxy model, the operation instructions and descriptions of the operation instructions in the operation sequence corresponding to the user task output by the GUI proxy model are obtained;

[0052] In an optional embodiment of the second aspect, the input of the GUI agent model includes user tasks, historical operations, and a screenshot of the current interface, and the output of the GUI agent model includes at least one operation instruction and an operation instruction description corresponding to at least one operation instruction.

[0053] In an optional embodiment of the second aspect, the first model includes a large language model and a pre-trained first instruction generation model; by invoking the first model, operation instructions in the operation sequence corresponding to the user task are obtained, including:

[0054] By calling the large language model, the operation instruction description corresponding to the user task output by the large language model is obtained; the operation instruction description is input into the first instruction generation model, and the operation instruction corresponding to the operation instruction description output by the first instruction generation model is obtained.

[0055] In an alternative embodiment of the second aspect, the first model includes a pre-trained instruction description generation model and a pre-trained second instruction generation model;

[0056] By calling the first model, the operation instructions in the operation sequence corresponding to the user task are obtained, including: by calling the instruction description generation model, obtaining the operation instruction description corresponding to the user task output by the instruction description generation model; inputting the operation instruction description into the second instruction generation model, obtaining the operation instruction corresponding to the operation instruction description output by the second instruction generation model.

[0057] In an optional embodiment of the second aspect, the first model includes a GUI agent model, a large language model, a pre-trained first instruction generation model, a pre-trained instruction description generation model, and a pre-trained second instruction generation model.

[0058] By calling the first model, the operation instructions in the operation sequence corresponding to the user task are obtained, including: calling the GUI proxy model to obtain the first operation instruction corresponding to the user task; and calling the large language model and the first instruction generation model in sequence to obtain the second operation instruction corresponding to the user task; and calling the instruction description generation model and the second instruction generation model in sequence to obtain the third operation instruction corresponding to the user task.

[0059] From the first, second, and third operation instructions, select the operation instruction with the highest confidence level as the final nth operation instruction; the confidence level is used to indicate the probability value that the operation instruction is correct.

[0060] The first operation instruction, the second operation instruction, and the third operation instruction are all candidate instructions for the nth operation instruction in the operation sequence corresponding to the user task, where n is a positive integer.

[0061] In an optional embodiment of the second aspect, the correctness evaluation of the operation sequence in the sample data and the determination of whether the sample data is positive example data include: obtaining the first evaluation result of the first evaluation model for each operation instruction in the operation sequence by calling the pre-trained first evaluation model; and obtaining the second evaluation result of the second evaluation model for the operation sequence as a whole by calling the pre-trained second evaluation model; and determining whether the sample data is positive example data based on the first evaluation result and the second evaluation result.

[0062] In an optional embodiment of the second aspect, the first evaluation result includes a first score from the first evaluation model for each operation instruction in the operation sequence, the first score indicating whether the operation instruction is progressing in the direction of completing the user task; the second evaluation result indicating whether the final execution result of the operation sequence matches the user task.

[0063] Based on the first evaluation result and the second evaluation result, determine whether the sample data is positive example data, including: based on the first evaluation result, determine the first proportion of the number of operation instructions with the first score as the first value to the total number of operation instructions in the operation sequence; based on the relationship between the first proportion and the first threshold, and the second evaluation result, determine whether the sample data is positive example data; the first score as the first value is used to indicate that the operation instructions are developing in the direction of completing the user task.

[0064] In an optional embodiment of the second aspect, the method further includes: when the sample data is determined to be negative sample data, by calling a pre-trained task synthesis model, obtaining a new user task output by the task synthesis model that matches the operation sequence in the negative sample data; and taking the operation sequence in the negative sample data, the interface data corresponding to each operation instruction in the operation sequence, and the new user task as a new set of sample data.

[0065] In an optional embodiment of the second aspect, the method further includes: evaluating the correctness of the new sample data by invoking a pre-trained second evaluation model to determine whether the new sample data is positive data; if the new sample data is positive data, using the new sample data as training data for the GUI agent model.

[0066] In an alternative embodiment of the second aspect, obtaining user tasks from multiple application servers includes: obtaining user tasks from multiple application servers by executing crawler tasks and / or invoking a large language model.

[0067] Thirdly, this application provides a data acquisition system, comprising: a first device and at least one second device; the first device being communicatively connected to the at least one second device; the first device being configured to perform the method as described in the second aspect or any optional embodiment of the second aspect. The second device is a test device as described in the second aspect.

[0068] Fourthly, embodiments of this application provide an apparatus including a processor and a memory, the memory being used to store execution instructions, and the processor being used to run the execution instructions stored in the memory to perform the method described in the second aspect or any optional embodiment of the second aspect.

[0069] Fifthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed on a device, cause the device to perform the method described in the second aspect or any optional embodiment of the second aspect.

[0070] In a sixth aspect, embodiments of this application provide a program product, the program product including a program, which, when run, causes a device to perform the method described in the second aspect or any optional embodiment of the second aspect.

[0071] In a seventh aspect, this application provides a chip or chip system including at least one processor and a communication interface, the communication interface and at least one processor being interconnected via a line, and the at least one processor being used to run a program or instructions to perform the method described in the second aspect or any optional embodiment of the second aspect.

[0072] The communication interface in the chip can be an input / output interface, pins, or circuits.

[0073] In an alternative embodiment of the seventh aspect, the chip or chip system further includes at least one memory storing instructions. The memory may be an internal storage unit of the chip, such as a register or cache, or it may be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0074] It should be understood that the second to seventh aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding optional embodiments are similar, and will not be described again. Attached Figure Description

[0075] Figure 1 A schematic diagram illustrating the interaction between the GUI proxy model provided in this application and a mobile phone to execute a user task;

[0076] Figure 2 A flowchart illustrating a training data acquisition method for a GUI agent model provided in this application;

[0077] Figure 3 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 1 ;

[0078] Figure 4 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 2 ;

[0079] Figure 5 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 3 ;

[0080] Figure 6 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 4 ;

[0081] Figure 7 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 5 ;

[0082] Figure 8 A flowchart illustrating another method for collecting training data for a GUI agent model provided in this application. Detailed Implementation

[0083] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0084] 1. A web crawler task refers to the process of using an automated program (i.e., a web crawler or spider robot) to retrieve specified information and data from the Internet in batches according to preset rules.

[0085] 2. The number of model parameters is one of the key indicators for measuring the size and capability of a model. All other things being equal, the more parameters a model has, the more complex information it can learn and store, and the better its performance will be. The unit for the number of model parameters can be billions (B). For example, a model with 7B parameters means that it has 7 billion parameters.

[0086] 3. Other terms

[0087] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0088] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0089] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0090] 4. Electronic devices can be referred to as user equipment (UE), terminals, etc. For example, electronic devices can be mobile phones, tablets, personal computers (PCs), wearable electronic devices such as smartwatches, as well as various teaching aids (such as learning machines and early education machines), portable robots, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, in-vehicle equipment, wireless terminals in industrial control, and wireless terminals in smart homes, etc.

[0091] With the increasing application of multimodal large models in the field of graphical user interfaces, intelligent agents in electronic devices are gradually becoming the core tools for automated device interaction. Taking mobile phones as an example, the intelligent agent of a mobile phone calls multimodal large models to parse screenshots of mobile phone screens, historical operations, and user task intentions, and gradually generates and executes operation instructions to complete complex tasks.

[0092] The intelligent agent can also be called an intelligent agent, such as the YOYO agent. User tasks can be such as "finding flights from destination 1 to destination 2 tomorrow on application A, with two adults and one child, and filtering by airline 1", "searching for store 1 on application B and entering the store", "opening a navigation app and asking for a ride to the company", "opening application C and sharing the link to this page with friend Li", etc. It is evident that user tasks involve various high-frequency business scenarios such as e-commerce, travel, and social networking.

[0093] The following example illustrates the process by which electronic devices execute user tasks through intelligent agents.

[0094] For example, Figure 1 This diagram illustrates the interaction between the GUI proxy model provided in this application and a mobile phone to execute a user task. Taking a mobile phone as an example, see below. Figure 1 The phone displays interface 10, which is the desktop. The user wakes up the intelligent agent via voice and inputs the user task as "search for store 1 on application B and enter the store". In response to the input of the user task, the intelligent agent calls the GUI agent model. The GUI agent model outputs a series of operation instructions according to the user task until the user task is completed.

[0095] Specifically, in response to the user's input task, the smart agent invokes the GUI agent model. The GUI agent model generates operation instruction 1 based on the user task and the current interface 10. The content of operation instruction 1 is "Open application B". The smart agent executes operation instruction 1 and obtains the interface 11 returned by application B. Interface 11 is the homepage of application B.

[0096] Subsequently, the intelligent agent calls the GUI agent model. Based on interface 11, user task, and operation instruction 1, the GUI agent model generates operation instruction 2. The content of operation instruction 2 is "Enter store 1 in the search bar". The intelligent agent executes operation instruction 2 and obtains interface 12 returned by application B. Interface 12 is the search interface of application B.

[0097] Subsequently, the intelligent agent invokes the GUI agent model. Based on interface 12, the user task, operation instruction 1, and operation instruction 2, the GUI agent model generates operation instruction 3. The content of operation instruction 3 is "Click to search". The intelligent agent executes operation instruction 3 and obtains interface 13 returned by application B. Interface 13 is the search display interface of application B.

[0098] Subsequently, the intelligent agent invokes the GUI agent model. Based on interface 13, the user task, operation instruction 1, operation instruction 2, and operation instruction 3, the GUI agent model generates operation instruction 4. The content of operation instruction 4 is "Click to enter store 1". The intelligent agent executes operation instruction 4 and obtains interface 14 returned by application B. Interface 14 is the homepage of store 1 in application B.

[0099] It should be noted that while the GUI agent model outputs operation command 4, it also outputs indication information. The indication information is used to indicate that the user task has been completed. The intelligent agent displays interface 14 based on the indication information.

[0100] It should be noted that the GUI proxy model can be regarded as a multimodal large model applied to the GUI field. The GUI proxy model can be deployed on the edge or the cloud, and this application embodiment does not limit this.

[0101] The quality and diversity of training data for GUI agent models directly affect their generalization ability, task completion efficiency, and accuracy. Training data for GUI agent models includes a large number of task samples and datasets of a series of continuous operations during the execution of each task sample. These datasets include, but are not limited to, operation records, operation instructions, and interface data. Alternatively, training data for GUI agent models includes a large amount of operation trajectory data.

[0102] In related technologies, the methods for collecting training data for GUI agent models mainly rely on manual operation or automated tools based on random walk and breadth-first search (BFS).

[0103] The two data collection methods will be described below.

[0104] 1) Manual operation refers to developers generating trajectory data by manually simulating user actions on a mobile phone. This data collection method is not only time-consuming and labor-intensive, but also struggles to cover all possible operational scenarios, resulting in limited data scale. Current statistics show that a single person can collect approximately 20 trajectory data points per day, demonstrating extremely low efficiency in manual trajectory data collection, especially when a large amount of diverse operational data is needed. The speed of manual collection cannot keep up with the increasing data demands of model training. The long data collection cycle severely restricts the iteration and optimization speed of the model.

[0105] Furthermore, due to the subjectivity and uncertainty of manual operation, the collected trajectory data varies greatly in quality and consistency. This difference may lead to biases during model training, affecting the stability and performance of the model.

[0106] 2) Automated tools based on random walks mainly construct training data through the following steps:

[0107] Step 1: Obtain various types of apps from the app store and install them in the emulator environment.

[0108] Step 2: Run different types of installed applications and explore them using a random walk approach.

[0109] In this step, by simulating operations (such as clicking, scrolling, and inputting) on ​​each interactive element in the application interface in sequence, the jump relationship between interfaces is recorded. The interface is used as a node and the operation is used as an edge to construct a directed graph, and screenshots of the interface in the entire operation trajectory are saved.

[0110] Step 3: Use Breadth-First Search (BFS) to explore new interfaces with shorter operation sequences, while more comprehensively covering the various functions of the application, usually including some less frequently used interfaces (such as advertising pages, settings pages, etc.).

[0111] The above methods achieve comprehensive coverage of application functions to a certain extent, but they have the following significant drawbacks:

[0112] 1) Difficulty in covering common tasks in the business. Because random walks cannot simulate the task logic of real users, the proportion of operation trajectory data for core businesses (such as login, ticket purchase, and order placement) is relatively low.

[0113] 2) Poor data targeting. All interfaces are treated equally, resulting in a large number of low-value, low-frequency interfaces (such as advertising pages and settings pages). Although a lot of data is collected, its relevance to actual business needs is weak, and there is a lack of focused coverage of key business paths for core functions.

[0114] 3) Low data collection efficiency. Due to the lack of clear target path planning, it is easy to repeatedly explore peripheral pages, resulting in wasted resources and a prolonged data collection cycle.

[0115] Therefore, this application provides a method for collecting training data for a GUI agent model, applied to a data acquisition platform. Through automated task generation and dynamic evaluation mechanisms, it achieves efficient collection and quality screening of training data. This method mainly involves the following technical points:

[0116] Key Point 1: By calling large language models (such as generative pre-trained transformer (GPT) models), or by combining web crawling technology with calling large language models, a large number of user tasks in high-frequency business scenarios can be obtained from the network side. The collected user task data can cover high-frequency business scenarios of users' real needs, thereby improving the data quality of the training dataset.

[0117] Furthermore, based on the performance of the GUI proxy model, target business scenarios with insufficient model capabilities are identified. Using the aforementioned methods, user tasks in the target business scenarios are obtained as supplementary data for training the GUI proxy model, thereby enhancing the performance of the GUI proxy model.

[0118] Key Point 2: For a specific user task collected, the data acquisition platform sequentially obtains the operation instructions and descriptions corresponding to the user task by calling a pre-trained GUI proxy model. Through multiple interactions with test devices (such as developer devices, which have a large number of popular applications installed), the platform obtains the operation trajectory data corresponding to the user task. The operation trajectory data includes multiple operation instructions and the interface data after executing the operation instructions. The user task and the operation trajectory data corresponding to the user task are used as a set of sample data (not necessarily used for training the GUI proxy model).

[0119] Subsequently, pre-trained data evaluation models, such as process reward models (PRM) and outcome reward models (ORM), are used to determine whether each set of sample data is positive or negative. Positive data can be added to a positive example library for subsequent training of the GUI agent model.

[0120] By introducing a data evaluation model, the quality of the model training data can be improved.

[0121] Point 3: For sample data identified as negative examples, a pre-trained task synthesis model can be used to update the user tasks in the negative example data, generating a new set of sample data. Subsequently, a pre-trained data evaluation model (e.g., an ORM model) evaluates this new sample data. If the new sample data is determined to be positive, it can be added to the positive example library for subsequent training of the GUI agent model.

[0122] Compared to directly discarding negative examples, this method can improve the utilization rate of negative examples.

[0123] Points 1 to 3 above can be referred to Figure 2 The illustrated embodiment.

[0124] Point 4: By adjusting the model parameters of the pre-trained GUI proxy model, the diversity of the GUI proxy model's output results is increased. Then, combined with the PRM's scoring of the GUI proxy model's output results, the final output result is determined, thereby improving the quality of the constructed sample data. See details for further information. Figure 3 The illustrated embodiment.

[0125] Point 5: For a specific user task collected, the data acquisition platform calls both the large language model and the pre-trained instruction generation model to sequentially obtain the operation instructions corresponding to the user task. Through multiple interactions with the test device, it acquires the operation trajectory data corresponding to the user task, and combines the user task and its corresponding operation trajectory data as a set of sample data. See details for further information. Figure 4 Example.

[0126] Point 6: For a specific user task collected, the data acquisition platform calls both the pre-trained instruction description generation model and the pre-trained instruction generation model to sequentially obtain the operation instructions corresponding to the user task. Through multiple interactions with the test device, it acquires the operation trajectory data corresponding to the user task, and combines the user task and its corresponding operation trajectory data as a set of sample data. See details for further information. Figure 5 Example.

[0127] Point 7: For a specific user task being collected, the data acquisition platform can combine at least two of the following: the GUI proxy model in Point 4; or the large language model and instruction generation model in Point 5; or the instruction description generation model and instruction generation model in Point 6, to sequentially obtain the operation instructions corresponding to the user task. Through multiple interactions with the test device, the platform can obtain the operation trajectory data corresponding to the user task, and use the user task and its corresponding operation trajectory data as a set of sample data. See details for further information. Figure 6 Example.

[0128] The following section provides a detailed description of the data collection method for the GUI proxy model, using specific examples.

[0129] For example, Figure 2 This is a flowchart illustrating a method for collecting training data for a GUI agent model provided in this application. This method is used to construct training data for the GUI agent model, and the execution entity of this method is a data collection platform. (Refer to...) Figure 2 The data acquisition platform 20 includes a first task generation module 21, a pre-trained GUI agent model 22, a pre-trained data evaluation model 23, and a pre-trained task synthesis model 24.

[0130] The following sections will introduce the various modules and models of the data acquisition platform in sequence according to the data construction process.

[0131] (a) The first task generation module is used to automatically generate a large number of user tasks according to the preset business scenario requirements. The model training data includes user tasks.

[0132] The first task generation module can generate user tasks in at least one of the following ways:

[0133] Method 1: The first task generation module obtains user task templates corresponding to the user's preset business scenario requirements. Next, the first task generation module calls an open-source large language model to obtain a large number of slot contents related to the business scenario requirements. Combined with the user task templates corresponding to the business scenario requirements, it generates a large number of user tasks. Here, "user" refers to developers, and "slot" refers to the variable information that needs to be filled in the task, such as time, location, product name, destination, etc.

[0134] For example, taking the travel scenario of application A as an example, the user task template for the travel scenario can be "Search for a one-way train ticket from <departure location> to <destination location> in application A", "Purchase a train ticket from <departure location> to <destination location> with <seat type> for <date> in application A", "Purchase a train ticket from <departure location> to <destination location> for <traveler name> with <date> in application A", "Search for the latest train ticket from <departure location> to <destination location> in application A", etc. Here, "<>" represents a slot. By calling the Big Oracle model to populate the slots in the user task template, a large number of user tasks are generated.

[0135] This approach leverages the massive database of a large language model to retrieve slot content relevant to business scenarios, making the generated user tasks more specific and aligned with actual user needs.

[0136] Method 2: The first task generation module generates corresponding user tasks by executing web crawler tasks to obtain users' hot topics and common operational needs from the Internet.

[0137] Optionally, the first task generation module extracts user-focused hot topics and common operational needs from platforms with high user activity on the Internet by executing web crawler tasks, and generates corresponding user tasks.

[0138] Optionally, the first task generation module can further expand the user's task volume by executing crawler tasks and calling large language models to obtain more hot issues and scenario operation requirements that users are concerned about.

[0139] This approach utilizes web crawling technology combined with large language models to obtain user tasks in high-frequency business scenarios, ensuring that the constructed training data covers high-frequency business scenarios that reflect real user needs.

[0140] Method 3: The first task generation module generates a large number of user tasks required for specific business scenarios by executing crawler tasks and / or calling large language models, serving as supplementary training data. Specific business scenarios refer to those where the execution performance of the GUI proxy model is insufficient (e.g., user task execution fails or errors occur).

[0141] This approach collects user tasks for business scenarios where the model's functionality is insufficient, serving as supplementary user tasks to help the model be strengthened and trained in specific business scenarios.

[0142] (ii) A pre-trained GUI agent model is used to generate a series of operation instructions in sequence according to the user task.

[0143] After the first task generation module generates user tasks, these tasks can be stored in the task library of the data acquisition platform. The task library contains a large number of user tasks.

[0144] The data acquisition platform retrieves a user task from the task library and determines whether the user task is a user task in the positive example library. If not, the data acquisition platform calls a pre-trained GUI agent model, which interacts with the test device (e.g., the developer's mobile phone) multiple times to collect the operation trajectory data corresponding to the user task.

[0145] The positive example library includes user tasks that have been successfully executed and whose operation trajectory data has been collected, as well as the corresponding operation trajectory data for these user tasks. In other words, the positive example library stores training data that can be used for the model.

[0146] Specifically, the data acquisition platform invokes the GUI proxy model, which generates the first operation command and its description corresponding to the user task. This first command is sent to the test device (e.g., the developer's mobile phone). After executing the command, the test device returns the current interface data, such as a screenshot and corresponding structured document, to the data acquisition platform, completing the first interaction. The data acquisition platform then invokes the GUI proxy model again, inputting the historical operation commands (e.g., the first command), the current interface screenshot, and the user task. After processing, the GUI proxy model obtains the second operation command and its description, and sends it to the test device. After executing the second command, the test device returns the latest interface data to the data acquisition platform, completing the second interaction. This process continues until the user task is completed. Based on the user task, the corresponding operation commands, and the interface data, the data acquisition platform constructs a set of sample data.

[0147] It should be understood that the number of interactions between the platform and the testing device varies depending on the user's task, and the number of interactions may be greater than or equal to 1.

[0148] In some embodiments, the input to the GUI agent model includes historical operations (which may correspond to the aforementioned operation instructions), a current screenshot of the phone, and the user task. The output of the GUI agent model includes an operation instruction description and the operation instructions themselves.

[0149] The functionality of the GUI proxy model will be explained below with a specific example.

[0150] Example input for a GUI agent model: "You are a professional smartphone assistant capable of completing navigation tasks step-by-step based on user needs. The system will provide a current screenshot, a description of the user's task, and historical actions. Please make a decision based on the following context:"

[0151] Screenshot from mobile phone: ;

[0152] User task: "Find flights for tomorrow from destination 1 to destination 2 on application A";

[0153] Historical actions: "Open application A, click on flight tickets";

[0154] Please answer in Chinese. Please include a description of the operating instructions in the following place: <describe>< / describe> Inside, the operation instructions are placed <answer>< / answer> Inside."

[0155] Example of output from the GUI proxy model:

[0156] “ <describe> Select Destination 1< / describe> <answer> tap(504,190)< / answer> ".

[0157] In this example, the data acquisition platform can send the operation command "tap(504,190)" output by the GUI agent model to the test device so that the test device can execute the operation command.

[0158] It should be noted that in some embodiments, the input of the GUI agent model also includes manually labeled information, which is a description by the developer of a series of operation instructions corresponding to the user task, as detailed below.

[0159] (iii) Data evaluation model, used to determine whether the constructed sample data can be used for training the GUI agent model. In other words, the data evaluation model is used to determine whether the constructed sample data is positive or negative data, and positive data can be used for subsequent training of the GUI agent model.

[0160] In this embodiment, the data evaluation model can also be described as a reward model. In some embodiments, the data evaluation model includes a process reward model (PRM) and an outcome reward model (ORM).

[0161] The following sections will introduce PRM and ORM respectively.

[0162] (1) PRM

[0163] Referring to Part (II), in the process of constructing a set of sample data, PRM is used to score each operation instruction output by the GUI agent model, and to determine whether it is moving in the direction of completing the user task based on the score, that is, to judge the correctness and rationality of the operation steps corresponding to the operation instruction.

[0164] In one example, PRM provides a binary evaluation result, where the PRM output evaluation result includes 0 or 1, where 0 indicates that the operation step is incorrect and 1 indicates that the operation step is correct.

[0165] In some embodiments, the inputs to the PRM include a screenshot of the current mobile phone interface, historical operations, user tasks, current operation content, and a description of the current operation instruction. The current operation content is the current operation instruction output by the GUI agent model, and the current operation instruction description is a description of the current operation instruction. The output of the PRM includes a binary evaluation result (0 or 1).

[0166] The following example illustrates the functionality of PRM.

[0167] Example PRM input: "You are a trained process reward model that helps users determine whether an agent's actions are correct. Given the current smartphone screenshot, user task, and historical actions, please determine whether the given operation instruction description and operation content are correct. You will receive the following dynamic information:"

[0168] Screenshot from mobile phone: ;

[0169] User task: "Open the video app, find the most popular video of my followed user 1, and pause it."

[0170] Historical steps: "Open the video app, tap 'Me,' tap 'Follow,' swipe up to find User 1, and tap User 1."

[0171] Current operation instruction description: "Click on video 1 posted by user 1";

[0172] Current operation: "tap(228,659)"

[0173] User tasks typically require multiple steps to complete. You need to determine the correctness of the current operation command description and content based on screenshots, user tasks, and historical operations. A score of 1 indicates all correct actions, while a score of 0 indicates any errors. <answer> and< / answer> Output the final judgment in between.

[0174] PRM output example: <answer> 0< / answer> An output of 0 indicates that the PRM has determined the current operation to be incorrect.

[0175] By introducing PRM, the quality of the operation sequence (i.e., a series of operation instructions) corresponding to the user task can be controlled in a fine-grained manner. From the constructed sample data, sample data that causes the overall task to fail due to single-step operation errors can be removed, ensuring the purity of positive examples in the data used for model training. At the same time, it supports the GUI proxy model to correct errors in a timely manner during execution, enhancing the robustness of the model.

[0176] (2) ORM

[0177] When a user task is completed, the GUI agent model usually generates a task completion instruction. The data acquisition platform can input the sample data built based on the user task into the ORM. The ORM is used to determine whether the user task has been successfully completed based on the sample data, that is, to evaluate the correctness of the final execution result of a series of operation instructions.

[0178] In one example, the ORM provides a binary evaluation result, where the ORM outputs an evaluation result of 0 or 1, where 0 indicates that the user task was executed incorrectly and 1 indicates that the user task was executed correctly.

[0179] In some embodiments, the input to the ORM includes the user task, historical actions, and screenshots of the last N steps, where N is a positive integer, such as N=3. The output of the ORM includes a binary evaluation result (0 or 1).

[0180] The following example illustrates the functionality of ORM.

[0181] Example input for ORM: "You are a trained outcome reward model that helps users determine whether an agent's actions are correct. Given the user's task, historical actions, and screenshots of the last few steps on the phone, please determine whether the agent's actions are correct. You will receive the following dynamic information:"

[0182] Third to last step: Take a screenshot on your phone. ;

[0183] Second to last step: Take a screenshot on your phone. ;

[0184] Final screenshot: ;

[0185] User task: "Help me hail a ride to the company using the navigation app";

[0186] Historical steps: "Open the navigation app, enter the company name in the input box, click confirm, click hailing a taxi, enter the pick-up location, select express car, and click hailing a taxi now";

[0187] User tasks typically require multiple steps to complete. Please use screenshots of the final few steps as the basis for judgment; historical operations are for reference only. A score of 1 indicates that the user task requirements are fully met, and 0 indicates that there are errors. Finally, in <answer>< / answer> Output the final judgment.

[0188] Output example: <answer> 1< / answer> An output of 1 indicates that the ORM has determined the user's task to have been successfully completed.

[0189] By introducing ORM, objective evaluation of user task execution can be achieved, thereby automating the screening of training data, reducing manual review workload, and ensuring the quality of training data. Furthermore, based on the evaluation results of a large amount of sample data using ORM, the data acquisition platform can identify business scenarios where the GUI agent model's capabilities are insufficient, thus guiding model optimization strategies. For example, by collecting more user tasks from target business scenarios, constructing model training data for those target business scenarios, and further training the GUI agent model, the model's functionality can be optimized.

[0190] Based on the foregoing, after a user task is completed, the data acquisition platform obtains sample data constructed based on that user task, and by calling the data evaluation model, obtains the evaluation result of the data evaluation model on the sample data, and determines whether the sample data is positive or negative data.

[0191] In some embodiments, the data evaluation model includes PRM and ORM. It is assumed that the evaluation results output by both PRM and ORM are binary evaluation results. The data acquisition platform obtains the evaluation result 1 of PRM for each operation instruction in the sample data, and determines the proportion of operation instructions with evaluation result 1 as the first value (e.g., the first value is 1) to the total number of operation instructions in the sample data; based on this proportion and the evaluation result 2 of ORM for the operation sequence (including at least one operation instruction) of the sample data, it determines whether the sample data is positive or negative.

[0192] Among them, the evaluation result 1 of the PRM for the operation instruction is the first value, which means that the operation steps corresponding to the operation instruction are correct, or that the operation instruction is moving in the direction of completing the user's task.

[0193] In one example, if evaluation result 2 is the first value, and the proportion of the number of operation instructions with evaluation result 1 as the first value to the total number of operation instructions in the sample data is greater than or equal to the first threshold (e.g., the first threshold is 0.6), the data acquisition platform determines that the sample data is positive example data.

[0194] In one example, if evaluation result 2 is the second value (e.g., the second value is 0), and the proportion of the number of operation instructions with evaluation result 1 as the first value to the total number of operation instructions in the sample data is less than or equal to the second threshold (e.g., the second threshold is 0.4), the data acquisition platform determines the sample data to be negative example data.

[0195] In some embodiments, after the data acquisition platform determines that the sample data is positive data based on the evaluation results of the data evaluation model, it still needs to be manually confirmed to avoid the data acquisition platform misjudging negative data as positive data and to improve the quality of positive data in the positive data library.

[0196] In some embodiments, for sample data that the data acquisition platform determines to be positive examples and that is also manually confirmed as positive examples, the data acquisition platform stores it in a positive example library for subsequent training of the GUI agent model. The data acquisition platform marks the status of the user task in this sample data as successfully executed to avoid the data acquisition platform repeatedly executing the same user task.

[0197] In some embodiments, sample data that is determined to be positive by the data acquisition platform but is ultimately determined to be negative after manual verification can be fed back to the data evaluation model for further training and updating of the data evaluation model (e.g., PRM and ORM) to improve the performance of the data evaluation model.

[0198] (iv) Task synthesis model, used to generate user tasks that match the operation sequence in the sample data for sample data that are judged as negative examples.

[0199] In some embodiments, after the data acquisition platform determines that the sample data is negative based on the evaluation result of the data evaluation model, the data acquisition platform inputs the sample data into the task synthesis model. After analysis and processing by the task synthesis model, the target task that matches the operation sequence in the sample data is obtained.

[0200] In this embodiment, the target task can be understood as a new user task generated by the task synthesis model based on the operation sequence in the sample data. The target task is different from the user task in the sample data. If the sample data is negative, it indicates that the operation sequence in the sample data has not completed the user task in the sample data, that is, the user task in the sample data does not match the operation sequence. By introducing a pre-trained task synthesis model, the output of positive sample data is further improved, avoiding the direct discarding of sample data judged as negative, and improving the utilization rate of sample data judged as negative.

[0201] In some embodiments, the input to the task synthesis model includes an operation sequence (e.g., an operation sequence from sample data judged as negative), a screenshot of the target application's homepage, and screenshots of the interface corresponding to the last M steps in the operation sequence. The target application is the application mentioned in the user task within the sample data. M is a positive integer, for example, M = 3. The output of the task synthesis model includes the target task that matches the operation sequence in the sample data.

[0202] The following example illustrates the functionality of the task synthesis model.

[0203] Example input for a task synthesis model: "You are a specially trained operation instruction inference model that can understand the mobile application name, historical operations, and multiple screenshots, and infer the user task of reaching the last screenshot. Input information includes:"

[0204] Mobile application name: Application 1;

[0205] Screenshot of the homepage of application 1: ;

[0206] Third to last step: Take a screenshot on your phone. ;

[0207] Second to last step: Take a screenshot on your phone. ;

[0208] Final step: Take a screenshot on your phone. ;

[0209] Historical steps: "Open application 1, tap Me, tap My Orders, tap Confirm Receipt for Order 1 in My Orders, tap Confirm";

[0210] Please place the final generated user task completely and accurately in [the specified location]. <answer>< / answer>Within the tag.

[0211] Example of output from any synthetic model: <answer> Sign up for order 1 in application 1< / answer> ".

[0212] Based on the foregoing, the task synthesis model infers user tasks through operation sequences and interface data, while the GUI agent model infers operation instructions by combining user tasks with interface data. Therefore, the reasoning process of the task synthesis model can be viewed as the inverse process of the GUI agent model's reasoning.

[0213] In some embodiments, the data acquisition platform can acquire new sample data through a task synthesis model. This new sample data includes operation sequences, interface data, and a new user task (i.e., the user task inferred by the task synthesis model). The data acquisition platform calls a data evaluation model (e.g., ORM) to determine whether the execution results of the operation sequences in the new sample data match the new user task. If the data evaluation model outputs a first value, and the sample data is manually confirmed as positive example data (optional), the data acquisition platform adds the new sample data to the positive example library for subsequent training of the GUI agent model.

[0214] In some embodiments, after the data acquisition platform determines that the sample data is negative based on the evaluation results of the data evaluation model, the data acquisition platform can first push the negative sample data to the developers so that the developers can manually annotate it, marking the correct operation path of the user task in the negative sample data. The data acquisition platform can then input the manually annotated user task and its correct operation path into the task library so that the data acquisition platform can collect data again for that user task, improving the success rate of training data acquisition.

[0215] In addition, by collecting negative example data, the data acquisition platform can determine which business areas the GUI agent model is inadequate in, triggering the first task generation module to further collect user tasks in specific business areas, thereby constructing training data for that specific business area and continuously improving the performance of the GUI agent model.

[0216] It should be pointed out that, Figure 2 The data acquisition platform shown in the embodiment can be used not only for training data to build the GUI agent model, but also for dynamically evaluating the performance of the GUI agent model.

[0217] When evaluating the performance of a GUI agent model, quantifiable metrics are required. Traditional methods typically use static test sets for evaluation, which involves splitting the collected task trajectory data into multiple single-step data points, organizing them into samples that conform to the input format of the GUI agent model, and then comparing the consistency between the model output and the ground truth.

[0218] However, static testing methods have significant drawbacks, such as the multi-path problem: in real-world tasks, there may be multiple feasible paths to complete the same user task, while static testing typically only considers outputs that perfectly match the preset truth value as correct, ignoring other equally reasonable paths, leading to biased and unfair evaluation results. Therefore, relying solely on static testing may not fully and accurately reflect the model's true performance.

[0219] The data acquisition platform shown in this application embodiment can automatically evaluate the execution results of the collected test data (including the evaluation of different execution paths) through pre-trained PRM and ORM. Combined with simple manual verification, it can quickly and accurately quantify model performance, significantly improving the fairness and practicality of the test.

[0220] Figure 2 In this embodiment, each output of the pre-trained GUI agent model includes an operation instruction and a description of that instruction. To improve the diversity of data sampling, the diversity of the GUI agent model's output results can be increased by adjusting the model parameters within the GUI agent model.

[0221] In some embodiments, the model parameters of the GUI agent model include a first parameter, which indicates the diversity of operation instructions output by the GUI agent model. The larger the value of the first parameter, the higher the diversity of operation instructions output by the GUI agent model; the smaller the value of the first parameter, the lower the diversity of operation instructions output by the GUI agent model.

[0222] For example, if the first parameter is the third value, the operation instructions output by the GUI agent model include, for instance, 10 operation instructions, all of which are identical, resulting in low diversity of operation instructions output by the GUI agent model. If the first parameter is the fourth value, which is greater than the third value, the operation instructions output by the GUI agent model still include 10 operation instructions, but only 2 of these 10 operation instructions are identical, resulting in high diversity of operation instructions output by the model.

[0223] For example, Figure 3 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 1 .exist Figure 2 Based on the data acquisition platform shown, and referring to Figure 3 The data acquisition platform 20 also includes a post-processing module 25, which is used to determine the output results of the GUI agent model 22 each time, determine the target operation instruction from multiple operation instructions, and send the target operation instruction to the test device for execution.

[0224] Specifically, the output of the GUI agent model 33 each time includes multiple operation instructions, such as... Figure 3 The operation instructions 1, 2, and 3 shown are used by the post-processing module to determine the target operation instruction from among the multiple operation instructions based on the evaluation results of the PRM, and then send the target operation instruction to the test device for execution.

[0225] The Probability Response Model (PRM) is used to evaluate multiple operation commands output by the GUI agent model. The PRM evaluation result for each operation command includes the probability value that the operation command is correct and the probability value that the operation command is incorrect. The post-processing module selects the operation command with the highest probability value indicating correctness from the evaluation results as the target operation command.

[0226] By introducing a post-processing module, which combines the evaluation data (e.g., probability values) of multiple outputs of the GUI agent model from the PRM, the final output can be determined from the multiple outputs of the GUI agent model, thereby improving the quality of the constructed sample data.

[0227] In some embodiments, the data acquisition platform can also obtain the operation instruction description corresponding to the collected user task by calling an external large language model, such as the GPT model, and combine it with a pre-trained instruction generation model to obtain the machine-recognizable operation instruction corresponding to the operation instruction description.

[0228] For example, Figure 4 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 2 . Reference Figure 4 The data acquisition platform 40 includes a first task generation module 21, a pre-trained instruction generation model 41, a pre-trained data evaluation model 23, and a pre-trained task synthesis model 24. The instruction generation model 41 is a single-step model used to generate machine-recognizable operation instructions based on the textual description of the operation instructions.

[0229] based on Figure 4 The data acquisition platform shown is, and Figure 2 The difference in this embodiment is that the data acquisition platform calls an external large language model, inputting user tasks, historical operations, and a current mobile phone screenshot into the large language model. After processing by the large language model, it obtains the operation instruction description returned by the large language model. Subsequently, the data acquisition platform generates the operation instruction corresponding to the operation instruction description through an instruction generation model and sends the operation instruction to the test device for execution. It should be noted that in this embodiment, the first task generation module, data evaluation model, and task synthesis model in the data acquisition platform are all different from those in this embodiment. Figure 2 The implementation methods are similar, and you can refer to the previous text for details, which will not be repeated here.

[0230] The data acquisition platform shown in this embodiment can efficiently and effectively construct training data for GUI agent models, thereby accelerating the iteration and performance improvement of GUI agent models.

[0231] In some embodiments, the data acquisition platform can also obtain the operation instruction description and operation instruction corresponding to the collected user task through a pre-trained instruction description generation model and a pre-trained instruction generation model, respectively.

[0232] For example, Figure 5 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 3 . Reference Figure 5 The data acquisition platform 50 includes a first task generation module 21, a pre-trained instruction description generation model 51, a pre-trained instruction generation model 52, a pre-trained data evaluation model 23, and a task synthesis model 24.

[0233] based on Figure 5 The data acquisition platform shown is, and Figure 2 The difference in this embodiment is that the data acquisition platform calls the instruction description generation model, inputting user tasks, historical operations, and the current mobile phone screenshot into the instruction description generation model. After processing by the instruction description generation model, it obtains the operation instruction description. Subsequently, the data acquisition platform inputs the operation instruction description into the instruction generation model to obtain the corresponding operation instruction, and sends the operation instruction to the test device for execution. It should be noted that in this embodiment, the first task generation module, data evaluation model, and task synthesis model in the data acquisition platform are all different from those in this embodiment. Figure 2 The implementation methods are similar, and you can refer to the previous text for details, which will not be repeated here.

[0234] The data acquisition platform shown in this embodiment can efficiently and effectively construct training data for GUI agent models, thereby accelerating the iteration and performance improvement of GUI agent models.

[0235] In conjunction with the foregoing embodiments, in some embodiments, reference is made to Figure 6 The data acquisition platform 60 includes a first task generation module 21, a data evaluation model 23, a post-processing module 25, a task synthesis model 24, and at least two of the following models: instruction generation model 41; or, instruction description generation model 51 and instruction generation model 52; or, GUI agent model 22.

[0236] For example, Figure 6 Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 4 . Reference Figure 6The data acquisition platform 60 includes a first task generation module 21, a pre-trained GUI agent model 22, an instruction generation model 41, an instruction description generation model 51, an instruction generation model 52, a post-processing module 25, a data evaluation model 23, and a task synthesis model 24. In one example, the GUI agent model 22 has 7 bytes of parameters, the instruction description generation model 51 has 32 bytes of parameters, and both the instruction generation model 41 and the instruction generation model 52 are single-step models.

[0237] based on Figure 6 The data acquisition platform shown is, and Figure 2 The difference in this embodiment is that the data acquisition platform obtains user tasks to be executed from the task library, and acquires the operation instructions output by different execution paths through the following three execution paths:

[0238] Execution Path 1: Input the user task, historical operations, and current mobile phone screenshot into the pre-trained GUI agent model to obtain at least one operation instruction and its description. The operation instruction output by Path 1 is denoted as Operation Instruction 1;

[0239] Execution path 2: Call an external large language model, input the user task, historical operations, and current mobile phone screenshot into the large language model to obtain the operation instruction description; then, use the pre-trained instruction generation model 41 to obtain the operation instruction corresponding to the operation instruction description. The operation instruction output by path 2 is denoted as operation instruction 2;

[0240] Execution path 3: Using a pre-trained instruction description generation model, the user task, historical operations, and current mobile phone screenshot are input into the instruction description generation model to obtain the operation instruction description; subsequently, the pre-trained instruction generation model 52 is used to obtain the operation instruction corresponding to the operation instruction description. The operation instruction in path 3 is denoted as operation instruction 3.

[0241] Next, the post-processing module of the data acquisition platform can select the target operation instruction with the highest probability of correctness from the three operation instructions based on the evaluation results of the data evaluation model (such as RPM), for example, operation instruction 2, and send operation instruction 2 to the test device for execution, completing one interaction with the test device. In one example, PRM evaluates the correctness of each operation instruction and outputs the evaluation result for each operation instruction. The evaluation result may include the confidence level of the operation instruction, which indicates the probability value that the operation instruction is a correct operation instruction.

[0242] Figure 6 The data acquisition scheme shown in the embodiment can improve the quality of the constructed sample data by calling different models to increase the diversity of sample data.

[0243] For example, Figure 7Schematic diagram of the training data acquisition platform for the GUI agent model provided in this application Figure 5 . Reference Figure 7 The data acquisition platform of this embodiment includes: a first task generation module 71, a sample data generation module 72, and a sample data evaluation module 73; the sample data generation module 75 is connected to the first task generation module 71 and the sample data evaluation module 73 respectively.

[0244] The first task generation module 71 is used to obtain user tasks from multiple application servers; the sample data generation module 72 is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model; send operation instructions to the test device and obtain the interface data corresponding to the operation instructions from the test device; construct sample data based on the user task, operation sequence and the interface data corresponding to each operation instruction in the operation sequence; the sample data evaluation module 73 is used to evaluate the correctness of the operation sequence in the sample data and determine whether the sample data is positive example data; if the sample data is positive example data, the sample data is used as training data for the GUI proxy model.

[0245] In this embodiment, the multiple application servers include servers for multiple target applications. Target applications can be applications with a higher number of user visits (or downloads) within a preset time period (e.g., the past six months or the past month) than a preset number (i.e., popular applications), or applications with a higher average number of user visits per day within a preset time period (e.g., daily) than a preset number (i.e., applications with high daily user activity). It is understood that since user tasks are obtained from multiple application servers, the user tasks involve various high-frequency business scenarios.

[0246] In this embodiment, the operation sequence corresponding to the user task includes one or more operation instructions. Each time the sample data generation module calls the first model, it can obtain one operation instruction from the operation sequence corresponding to the user task. By repeatedly calling the first model, all operation instructions in the operation sequence corresponding to the user task are obtained until the user task is completed. The interface data corresponding to each operation instruction in the operation sequence includes at least the interface data after the test device executes the operation instruction, and the interface data includes a screenshot. In some embodiments, the interface data also includes an interface description document, which is used to indicate the interface structure, layout, interface element attributes, and hierarchical relationships, etc.

[0247] In this embodiment, the sample data is positive example data, which indicates that the user task and operation sequence in the sample data are matched with each other, that is, the user task can be successfully completed by executing the operation sequence through the test device.

[0248] In this embodiment, the GUI proxy model is used to generate operation instructions and operation instruction descriptions corresponding to the task based on the task input by the user.

[0249] The data acquisition platform described above, after acquiring a large number of user tasks from the network side, calls a pre-trained first model. This first model interacts with the testing device to construct sample data corresponding to the user tasks. For each constructed set of sample data, the correctness of the operation sequences within the sample data is evaluated to determine whether the sample data can be used as training data for a GUI proxy model to optimize model performance. Compared to the traditional method of relying on manual operation to construct training data, this significantly improves the efficiency of model training data acquisition and the quality of the training data.

[0250] In one optional embodiment, the first model includes a pre-trained GUI agent model; the sample data generation module is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model, including:

[0251] The sample data generation module is used to call the GUI proxy model and obtain the operation instructions and operation instruction descriptions from the operation sequence corresponding to the user task output by the GUI proxy model.

[0252] In one optional embodiment, the input of the GUI agent model includes user tasks, historical operations, and a screenshot of the current interface, and the output of the GUI agent model includes at least one operation instruction and an operation instruction description corresponding to at least one operation instruction.

[0253] In some embodiments, the model parameters of the GUI proxy model include a first parameter, which indicates the diversity of operation instructions output by the GUI proxy model. If the first parameter is set to a third value, the output of the GUI proxy model includes one operation instruction and a corresponding operation instruction description. If the first parameter is set to a fourth value, the output of the GUI proxy model includes at least two operation instructions and corresponding operation instruction descriptions for each operation instruction.

[0254] In one optional embodiment, the first model includes a large language model and a pre-trained first instruction generation model; the sample data generation module is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model, including: the sample data generation module is used to obtain the operation instruction description corresponding to the user task output by the large language model by calling the large language model; input the operation instruction description into the first instruction generation model, and obtain the operation instruction corresponding to the operation instruction description output by the first instruction generation model.

[0255] For example, large language models include, for instance, the GPT model.

[0256] In some embodiments, the data acquisition platform includes a pre-trained first instruction generation model, which may correspond to the aforementioned instruction generation model 41.

[0257] In the above scheme, the data acquisition platform first obtains the operation instruction description corresponding to the user task by calling an external large language model, and then calls a locally pre-trained first instruction generation model to obtain the operation instruction corresponding to the operation instruction description. This operation instruction is the operation instruction in the operation sequence corresponding to the user task. The above calling process is repeated to obtain other operation instructions in the operation sequence corresponding to the user task until the user task is completed.

[0258] In one optional embodiment, the first model includes a pre-trained instruction description generation model and a pre-trained second instruction generation model; the sample data generation module is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model, including:

[0259] The sample data generation module is used to call the instruction description generation model to obtain the operation instruction description corresponding to the user task output by the instruction description generation model; input the operation instruction description into the second instruction generation model to obtain the operation instruction corresponding to the operation instruction description output by the second instruction generation model.

[0260] In some embodiments, the data acquisition platform includes an instruction description generation model and a second instruction generation model. The second instruction generation model may correspond to the aforementioned instruction generation model 52.

[0261] In the above scheme, the data acquisition platform first obtains the operation instruction description corresponding to the user task by calling a locally pre-trained instruction description generation model. Then, it calls a locally pre-trained second instruction generation model to obtain the operation instruction corresponding to the operation instruction description. This operation instruction is the operation instruction in the operation sequence corresponding to the user task. The above calling process is repeated to obtain other operation instructions in the operation sequence corresponding to the user task until the user task is completed.

[0262] In one alternative embodiment, the first model includes a GUI agent model, a large language model, a pre-trained first instruction generation model, a pre-trained instruction description generation model, and a pre-trained second instruction generation model.

[0263] The sample data generation module is used to obtain operation instructions in the operation sequence corresponding to the user task by calling the first model, including: the sample data generation module calls the GUI proxy model to obtain the first operation instruction corresponding to the user task; and sequentially calls the large language model and the first instruction generation model to obtain the second operation instruction corresponding to the user task; and sequentially calls the instruction description generation model and the second instruction generation model to obtain the third operation instruction corresponding to the user task.

[0264] The sample data generation module is also used to select the operation instruction with the highest confidence level from the first operation instruction, the second operation instruction, and the third operation instruction as the final nth operation instruction; the confidence level is used to indicate the probability value that the operation instruction is a correct operation instruction.

[0265] Among them, the first operation instruction, the second operation instruction, and the third operation instruction are all candidate instructions for the nth operation instruction in the operation sequence corresponding to the user task, where n is a positive integer.

[0266] In the above scheme, the data acquisition platform obtains different model call paths (including, for example, multiple locally pre-trained models combined with calls to external large language models) by using multiple pre-trained models locally. Figure 6 The optional operation instructions (including, for example, those shown in paths 1, 2, and 3) for user tasks are as follows: Figure 6 The operation instructions shown (1, 2, and 3) determine the target operation instruction from multiple optional operation instructions, aiming to improve the quality of the constructed sample data by increasing data diversity.

[0267] In one optional embodiment, the sample data evaluation module is used to evaluate the correctness of the operation sequence in the sample data and determine whether the sample data is positive example data, including:

[0268] The sample data evaluation module is used to obtain the first evaluation result of the first evaluation model for each operation instruction in the operation sequence by calling the pre-trained first evaluation model; and to obtain the second evaluation result of the second evaluation model for the operation sequence as a whole by calling the pre-trained second evaluation model.

[0269] The sample data evaluation module is used to determine whether the sample data is positive example data based on the first evaluation result and the second evaluation result.

[0270] For example, the first evaluation model can correspond to the aforementioned process reward model (PRM), and the second evaluation model can correspond to the aforementioned outcome reward model (ORM).

[0271] In the above scheme, the data acquisition platform does not directly use each set of sample data to train the GUI agent model. Instead, it calls the first evaluation model and the second evaluation model to evaluate each set of sample data, filter out sample data that is judged as negative, retain sample data that is judged as positive, and use the sample data judged as positive as the training data for the subsequent GUI agent model, thereby improving the data quality of the constructed training data and continuously optimizing the model performance.

[0272] In one optional embodiment, the first evaluation result includes a first score from the first evaluation model for each operation instruction in the operation sequence, the first score indicating whether the operation instruction is progressing in the direction of completing the user task; the second evaluation result indicates whether the final execution result of the operation sequence matches the user task.

[0273] The sample data evaluation module is used to determine whether the sample data is positive example data based on the first evaluation result and the second evaluation result, including: the sample data evaluation module is used to determine the first proportion of the number of operation instructions with the first score as the first value to the total number of operation instructions in the operation sequence based on the first evaluation result; and to determine whether the sample data is positive example data based on the relationship between the first proportion and the first threshold, and the second evaluation result.

[0274] The first score is the highest value, used to indicate that the operation instructions are progressing in the direction of completing the user's task.

[0275] In this embodiment, the first evaluation result includes the evaluation result of the first evaluation model for each operation instruction in the operation sequence corresponding to the user task (which may correspond to the aforementioned evaluation result 1). The evaluation result of the first evaluation model for each operation instruction in the operation sequence can be a binary evaluation result. For example, if the first score of an operation instruction is a first value (e.g., 1), it indicates that the operation instruction is progressing in the direction of completing the user task (i.e., the operation instruction is a correct operation instruction). If the first score of an operation instruction is a second value (e.g., 0), it indicates that the operation instruction is not progressing in the direction of completing the user task (i.e., the operation instruction is an incorrect operation instruction).

[0276] In this embodiment, the second evaluation result (which may correspond to the aforementioned evaluation result 2) can be a binary evaluation result. If the second evaluation result is a first value (e.g., 1), it indicates that the final execution result of the operation sequence corresponding to the user task matches the user task. If the second evaluation result is a second value (e.g., 0), it indicates that the final execution result of the operation sequence corresponding to the user task does not match the user task.

[0277] In some embodiments, if the first proportion is greater than or equal to the first threshold and the second evaluation result is the first value, the sample data is determined to be positive data. The first threshold can be 0.6.

[0278] In some embodiments, if the first proportion is less than or equal to the second threshold, and the second evaluation result is the second value, the sample data is determined to be negative data. The second threshold can be 0.4.

[0279] The above scheme shows how to determine whether a sample data is positive or negative based on the evaluation results of the first evaluation model and the second evaluation model. The first evaluation model is mainly used to evaluate the correctness of each operation instruction in the sample data, and the second evaluation model is mainly used to evaluate the correctness of the operation instructions in the sample data after overall execution.

[0280] In one optional embodiment, the data acquisition platform further includes a second task generation module; the second task generation module is connected to the sample data evaluation module.

[0281] The sample data evaluation module is used to send negative sample data to the second task generation module when the sample data is determined to be negative sample data. The second task generation module is used to obtain a new user task that matches the operation sequence in the negative sample data by calling the pre-trained task synthesis model. The sample data generation module is also used to take the operation sequence in the negative sample data, the interface data corresponding to each operation instruction in the operation sequence, and the new user task as a new set of sample data.

[0282] In some embodiments, the data acquisition platform includes a task synthesis model.

[0283] In the above scheme, the data acquisition platform can generate corresponding user tasks based on the operation sequences and interface data in the negative example data by calling the task synthesis model, thereby constructing a new set of sample data. Compared with direct negative example data, this can improve the utilization rate of negative example data and further expand the sample data for model training.

[0284] In one optional embodiment, the sample data evaluation module is further configured to evaluate the correctness of the new sample data by calling a pre-trained second evaluation model, and determine whether the new sample data is positive data; if the new sample data is positive data, the new sample data is used as training data for the GUI agent model.

[0285] The above scheme shows that after constructing new sample data, the data acquisition platform performs a secondary evaluation on the newly constructed sample data through a sample data evaluation model to determine whether the newly constructed sample data can be used as training data.

[0286] For example, Figure 8 This is a flowchart illustrating another method for acquiring training data for a GUI agent model provided in this application. This method can be applied to any data acquisition device, see below. Figure 8 The method may include the following steps:

[0287] S801 retrieves user tasks from multiple application servers;

[0288] S802, by calling the first model, obtain the operation instructions in the operation sequence corresponding to the user task;

[0289] S803 sends operation commands to the test equipment;

[0290] S804 receives interface data corresponding to operation commands from the test equipment;

[0291] S805, construct sample data based on user tasks, operation sequences, and interface data corresponding to each operation instruction in the operation sequence;

[0292] S806, evaluate the correctness of the operation sequence in the sample data to determine whether the sample data is positive example data;

[0293] In some embodiments, if the sample data is positive data, the following can be performed:

[0294] S807 uses sample data as training data for the GUI agent model.

[0295] In this embodiment, the GUI proxy model is used to generate operation instructions and descriptions corresponding to the task based on the user's input. The test device can be a developer's mobile phone, etc., and various types of popular applications are installed on the test device.

[0296] In the above method, a large number of user tasks are first obtained from the network side. Then, by calling the first model and interacting with the test device, sample data corresponding to the user tasks is obtained. The sample data corresponding to the user tasks includes the user tasks, operation sequences, and interface data corresponding to each operation instruction in the operation sequence. For each set of sample data, the correctness of the operation sequences in the sample data is evaluated to determine whether the sample data can be used as training data for the GUI proxy model to optimize model performance. Compared with the traditional method of relying on manual operation to collect training data, this method can significantly improve the efficiency of model training data collection and the quality of training data.

[0297] In one optional embodiment, the first model includes a pre-trained GUI proxy model; obtaining operation instructions in the operation sequence corresponding to the user task by calling the first model includes: obtaining operation instructions and operation instruction descriptions in the operation sequence corresponding to the user task output by the GUI proxy model by calling the GUI proxy model.

[0298] In one optional embodiment, the input of the GUI agent model includes user tasks, historical operations, and a screenshot of the current interface, and the output of the GUI agent model includes at least one operation instruction and an operation instruction description corresponding to at least one operation instruction.

[0299] In one optional embodiment, the first model includes a large language model and a pre-trained first instruction generation model; by invoking the first model, the operation instructions in the operation sequence corresponding to the user task are obtained, including:

[0300] By calling the large language model, the operation instruction description corresponding to the user task output by the large language model is obtained; the operation instruction description is input into the first instruction generation model, and the operation instruction corresponding to the operation instruction description output by the first instruction generation model is obtained.

[0301] In one alternative embodiment, the first model includes a pre-trained instruction description generation model and a pre-trained second instruction generation model;

[0302] By calling the first model, the operation instructions in the operation sequence corresponding to the user task are obtained, including: by calling the instruction description generation model, obtaining the operation instruction description corresponding to the user task output by the instruction description generation model; inputting the operation instruction description into the second instruction generation model, obtaining the operation instruction corresponding to the operation instruction description output by the second instruction generation model.

[0303] In one optional embodiment, the first model includes a GUI agent model, a large language model, a pre-trained first instruction generation model, a pre-trained instruction description generation model, and a pre-trained second instruction generation model.

[0304] By calling the first model, the operation instructions in the operation sequence corresponding to the user task are obtained, including: calling the GUI proxy model to obtain the first operation instruction corresponding to the user task; and calling the large language model and the first instruction generation model in sequence to obtain the second operation instruction corresponding to the user task; and calling the instruction description generation model and the second instruction generation model in sequence to obtain the third operation instruction corresponding to the user task.

[0305] From the first, second, and third operation instructions, select the operation instruction with the highest confidence level as the final nth operation instruction; the confidence level is used to indicate the probability value that the operation instruction is correct.

[0306] The first operation instruction, the second operation instruction, and the third operation instruction are all candidate instructions for the nth operation instruction in the operation sequence corresponding to the user task, where n is a positive integer.

[0307] In one optional embodiment, the correctness evaluation of the operation sequence in the sample data to determine whether the sample data is positive example data includes: obtaining the first evaluation result of the first evaluation model for each operation instruction in the operation sequence by calling the pre-trained first evaluation model; and obtaining the second evaluation result of the second evaluation model for the operation sequence as a whole by calling the pre-trained second evaluation model; and determining whether the sample data is positive example data based on the first evaluation result and the second evaluation result.

[0308] In one optional embodiment, the first evaluation result includes a first score from the first evaluation model for each operation instruction in the operation sequence, the first score indicating whether the operation instruction is progressing in the direction of completing the user task; the second evaluation result indicates whether the final execution result of the operation sequence matches the user task.

[0309] Based on the first evaluation result and the second evaluation result, determine whether the sample data is positive example data, including: based on the first evaluation result, determine the first proportion of the number of operation instructions with the first score as the first value to the total number of operation instructions in the operation sequence; based on the relationship between the first proportion and the first threshold, and the second evaluation result, determine whether the sample data is positive example data; the first score as the first value is used to indicate that the operation instructions are developing in the direction of completing the user task.

[0310] In an optional embodiment, the method further includes: when the sample data is determined to be negative sample data, by calling a pre-trained task synthesis model, obtaining a new user task output by the task synthesis model that matches the operation sequence in the negative sample data; and taking the operation sequence in the negative sample data, the interface data corresponding to each operation instruction in the operation sequence, and the new user task as a new set of sample data.

[0311] In one alternative embodiment, the method further includes: evaluating the correctness of the new sample data by invoking a pre-trained second evaluation model to determine whether the new sample data is positive data; if the new sample data is positive data, using the new sample data as training data for the GUI agent model.

[0312] In one alternative embodiment, obtaining user tasks from multiple application servers includes: obtaining user tasks from multiple application servers by executing crawler tasks and / or invoking a large language model.

[0313] It should be noted that in the above embodiments, a "module" can be a software program, a hardware circuit, or a combination of both to implement the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuits, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functions.

[0314] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0315] It should be noted that the user information and data involved in the embodiments of this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0316] This application provides a data acquisition system, including: a first device and at least one second device; the first device and at least one second device are communicatively connected; the first device is used to execute the technical solution of the above method embodiments. The second device is a test device in the above method embodiments, such as a developer's mobile phone. Exemplarily, the data acquisition system may refer to... Figure 2 .

[0317] This application provides a device including a processor and a memory. The memory is used to store execution instructions, and the processor is used to run the execution instructions stored in the memory to execute the technical solutions in the above method embodiments.

[0318] This application provides a chip or chip system, which includes at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a line. The at least one processor is used to run programs or instructions to execute the technical solutions in the above method embodiments.

[0319] The communication interface in the chip can be an input / output interface, pins, or circuits.

[0320] In one alternative embodiment, the chip or chip system further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0321] This application provides a readable storage medium storing a program or instructions. When the program or instructions are run on a device, the device executes the technical solutions described in the above method embodiments.

[0322] The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over a readable storage medium. A readable storage medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0323] In one possible implementation, a readable storage medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a readable storage medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium.

[0324] This application provides a program product, which includes a program. When the program is run, it causes the device to execute the technical solutions described in the above method embodiments.

[0325] This application describes embodiments of methods, apparatus (systems), and program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0326] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A training data acquisition platform for a graphical user interface (GUI) agent model, characterized in that, include: The first task generation module, sample data generation module, and sample data evaluation module; The sample data generation module is connected to the first task generation module and the sample data evaluation module, respectively. The first task generation module is used to obtain user tasks from multiple application servers; The sample data generation module is used to obtain one or more operation instructions from the operation sequence corresponding to the user task by calling the first model, wherein the multiple operation instructions come from different model call paths in the first model; Send a target operation instruction to the test device, and obtain the interface data corresponding to the target operation instruction from the test device. The target operation instruction is one of the one or more operation instructions. Construct sample data based on the user task, the operation sequence, and the interface data corresponding to each target operation instruction in the operation sequence. The sample data evaluation module is used to evaluate the correctness of the operation sequence in the sample data and determine whether the sample data is positive data; if the sample data is positive data, the sample data is used as training data for the GUI agent model. The data acquisition platform also includes a post-processing module, which is connected to the sample data generation module and the sample data evaluation module respectively. The post-processing module is used to select the operation instruction with the highest confidence from the multiple operation instructions as the target operation instruction based on the evaluation result of each operation instruction by the pre-trained first evaluation model. The confidence is used to indicate the probability value that the operation instruction is a correct operation instruction. The sample data evaluation module is used to evaluate the correctness of the operation sequence in the sample data and determine whether the sample data is positive example data, including: The sample data evaluation module is used to obtain a first evaluation result of the first evaluation model for each operation instruction in the operation sequence by calling the first evaluation model; and to obtain a second evaluation result of the second evaluation model for the entire operation sequence by calling a pre-trained second evaluation model. The sample data evaluation module is used to determine whether the sample data is positive example data based on the first evaluation result and the second evaluation result.

2. The data acquisition platform according to claim 1, characterized in that, The first model includes a pre-trained GUI agent model; the sample data generation module is used to obtain one or more operation instructions from the operation sequence corresponding to the user task by calling the first model, including: The sample data generation module is used to call the GUI proxy model to obtain one or more operation instructions and operation instruction descriptions corresponding to the operation sequence of the user task output by the GUI proxy model.

3. The data acquisition platform according to claim 1, characterized in that, The inputs to the GUI proxy model include the user task, historical operations, and a screenshot of the current interface.

4. The data acquisition platform according to claim 1, characterized in that, The first model includes a large language model and a pre-trained first instruction generation model; the sample data generation module is used to obtain one or more operation instructions from the operation sequence corresponding to the user task by calling the first model, including: The sample data generation module is used to obtain the operation instruction description corresponding to the user task output by the large language model by calling the large language model; input the operation instruction description into the first instruction generation model to obtain the operation instruction corresponding to the operation instruction description output by the first instruction generation model.

5. The data acquisition platform according to claim 1, characterized in that, The first model includes a pre-trained instruction description generation model and a pre-trained second instruction generation model; the sample data generation module is used to obtain one or more operation instructions from the operation sequence corresponding to the user task by calling the first model, including: The sample data generation module is used to obtain the operation instruction description corresponding to the user task output by the instruction description generation model by calling the instruction description generation model; The operation instruction description is input into the second instruction generation model to obtain the operation instruction corresponding to the operation instruction description output by the second instruction generation model.

6. The data acquisition platform according to claim 1, characterized in that, The first model includes the GUI agent model, the large language model, the pre-trained first instruction generation model, the pre-trained instruction description generation model, and the pre-trained second instruction generation model; the sample data generation module is used to obtain one or more operation instructions from the operation sequence corresponding to the user task by calling the first model, including: The sample data generation module is used to call the GUI proxy model to obtain the first operation instruction corresponding to the user task; and to call the large language model and the first instruction generation model in sequence to obtain the second operation instruction corresponding to the user task; and to call the instruction description generation model and the second instruction generation model in sequence to obtain the third operation instruction corresponding to the user task. Wherein, the first operation instruction, the second operation instruction, and the third operation instruction are all candidate instructions for the nth operation instruction in the operation sequence corresponding to the user task, where n is a positive integer; The sample data generation module is further configured to select the operation instruction with the highest confidence level from the first operation instruction, the second operation instruction, and the third operation instruction as the final nth operation instruction; the confidence level is used to indicate the probability value that the operation instruction is a correct operation instruction.

7. The data acquisition platform according to claim 1, characterized in that, The first evaluation result includes a first score from the first evaluation model for each operation instruction in the operation sequence, the first score indicating whether the operation instruction is progressing in the direction of completing the user task; the second evaluation result indicates whether the final execution result of the operation sequence matches the user task. The sample data evaluation module is used to determine whether the sample data is positive example data based on the first evaluation result and the second evaluation result, including: The sample data evaluation module is used to determine, based on the first evaluation result, the first proportion of the number of operation instructions with the first score as the first value to the total number of operation instructions in the operation sequence; Based on the relationship between the first ratio and the first threshold, and the second evaluation result, determine whether the sample data is positive example data; The first score is a first value used to indicate that the operation instructions are progressing in the direction of completing the user's task.

8. The data acquisition platform according to claim 1, characterized in that, The data acquisition platform further includes a second task generation module; the second task generation module is connected to the sample data evaluation module. The sample data evaluation module is used to send the negative sample data to the second task generation module when it is determined that the sample data is negative sample data; The second task generation module is used to obtain a new user task output by the task synthesis model that matches the operation sequence in the negative example data by calling the pre-trained task synthesis model. The sample data generation module is also used to take the operation sequence in the negative example data, the interface data corresponding to each operation instruction in the operation sequence, and the new user task as a new set of sample data.

9. The data acquisition platform according to claim 8, characterized in that, The sample data evaluation module is also used to evaluate the correctness of the new sample data by calling a pre-trained second evaluation model to determine whether the new sample data is positive data; if the new sample data is positive data, the new sample data is used as training data for the GUI agent model.

10. A method for collecting training data for a graphical user interface (GUI) agent model, characterized in that, include: Retrieve user tasks from multiple application servers; By calling the first model, one or more operation instructions are obtained from the operation sequence corresponding to the user task, and the multiple operation instructions come from different model call paths in the first model; Send a target operation instruction to the test equipment, wherein the target operation instruction is one of one or more operation instructions; Obtain the interface data corresponding to the target operation command from the test device; Based on the user task, the operation sequence, and the interface data corresponding to each target operation instruction in the operation sequence, sample data is constructed. The correctness of the operation sequence in the sample data is evaluated to determine whether the sample data is positive example data. If the sample data is positive data, the sample data will be used as the training data for the GUI agent model; The method further includes: selecting the operation instruction with the highest confidence from the plurality of operation instructions as the target operation instruction based on the evaluation result of each operation instruction by the pre-trained first evaluation model, wherein the confidence is used to indicate the probability value that the operation instruction is a correct operation instruction; The step of evaluating the correctness of the operation sequence in the sample data to determine whether the sample data is positive example data includes: By invoking the first evaluation model, a first evaluation result of the first evaluation model for each operation instruction in the operation sequence is obtained; and by invoking the pre-trained second evaluation model, a second evaluation result of the second evaluation model for the entire operation sequence is obtained. Based on the first evaluation result and the second evaluation result, determine whether the sample data is positive example data.

11. The method according to claim 10, characterized in that, The first model includes a pre-trained GUI agent model; the step of obtaining one or more operation instructions from the operation sequence corresponding to the user task by invoking the first model includes: By invoking the GUI proxy model, one or more operation instructions and operation instruction descriptions corresponding to the operation sequence of the user task output by the GUI proxy model are obtained.

12. The method according to claim 10, characterized in that, The inputs to the GUI proxy model include the user task, historical operations, and a screenshot of the current interface.

13. The method according to claim 10, characterized in that, The first model includes a large language model and a pre-trained first instruction generation model; the step of obtaining one or more operation instructions from the operation sequence corresponding to the user task by calling the first model includes: By invoking the large language model, the operation instruction description corresponding to the user task output by the large language model is obtained; the operation instruction description is input into the first instruction generation model, and the operation instruction corresponding to the operation instruction description output by the first instruction generation model is obtained.

14. The method according to claim 10, characterized in that, The first model includes a pre-trained instruction description generation model and a pre-trained second instruction generation model; the step of obtaining one or more operation instructions from the operation sequence corresponding to the user task by calling the first model includes: By invoking the instruction description generation model, the operation instruction description corresponding to the user task output by the instruction description generation model is obtained; the operation instruction description is input into the second instruction generation model, and the operation instruction corresponding to the operation instruction description output by the second instruction generation model is obtained.

15. The method according to claim 10, characterized in that, The first model includes the GUI agent model, the large language model, the pre-trained first instruction generation model, the pre-trained instruction description generation model, and the pre-trained second instruction generation model; The step of obtaining one or more operation instructions from the operation sequence corresponding to the user task by invoking the first model includes: The GUI proxy model is invoked to obtain the first operation instruction corresponding to the user task; and the large language model and the first instruction generation model are invoked in sequence to obtain the second operation instruction corresponding to the user task; and the instruction description generation model and the second instruction generation model are invoked in sequence to obtain the third operation instruction corresponding to the user task; the first operation instruction, the second operation instruction and the third operation instruction are all candidate instructions of the nth operation instruction in the operation sequence corresponding to the user task, where n is a positive integer; From the first operation instruction, the second operation instruction, and the third operation instruction, the operation instruction with the highest confidence level is selected as the final nth operation instruction; the confidence level is used to indicate the probability value that the operation instruction is a correct operation instruction.

16. The method according to claim 10, characterized in that, The first evaluation result includes a first score from the first evaluation model for each operation instruction in the operation sequence, the first score indicating whether the operation instruction is progressing in the direction of completing the user task; the second evaluation result indicates whether the final execution result of the operation sequence matches the user task. The step of determining whether the sample data is positive example data based on the first evaluation result and the second evaluation result includes: Based on the first evaluation result, determine the first proportion of the number of operation instructions with the first score as the first value to the total number of operation instructions in the operation sequence; Based on the relationship between the first ratio and the first threshold, and the second evaluation result, it is determined whether the sample data is positive data; the first score is a first value used to indicate that the operation instructions are developing in the direction of completing the user task.

17. The method according to claim 10, characterized in that, The method further includes: When the sample data is determined to be negative example data, a new user task that matches the operation sequence in the negative example data is obtained by calling a pre-trained task synthesis model. The operation sequence in the negative example data, the interface data corresponding to each operation instruction in the operation sequence, and the new user task are taken as a new set of sample data.

18. The method according to claim 17, characterized in that, The method further includes: By calling the pre-trained second evaluation model, the correctness of the new sample data is evaluated to determine whether the new sample data is positive example data; If the new sample data is positive, the new sample data will be used as the training data for the GUI agent model.

19. The method according to claim 10, characterized in that, The process of obtaining user tasks from multiple application servers includes: User tasks are obtained from the multiple application servers by executing crawler tasks and / or invoking large language models.

20. A data acquisition system, characterized in that, include: A first device and at least one second device; the first device is communicatively connected to the at least one second device; the first device is used to perform the method as described in any one of claims 10 to 19, and the second device is the test device as described in claim 10.

Images