Training data generation method and device suitable for training GUI intelligent agent

Abstract

The invention discloses a training data generation method and device suitable for training a graphical user interface (GUI) agent. Private data of a user side does not need to be used. Wherein a first user instruction of to-be-generated training data is converted into a planning sub-instruction sequence through the large model, the planning sub-instruction sequence is matched with a plurality of sub-instruction sequences in the multiplexing library, and when the first plurality of sub-instructions in the planning sub-instruction sequence are successfully matched with the first sub-instruction sequence, the training data are generated. And obtaining the first action sequence and the page address of the first page. The multiplexing library stores a plurality of sub-instruction sequences, action sequences in the execution process of the sub-instruction sequences and corresponding relations among page addresses obtained after execution. And then, controlling the controlled equipment to display the first page, acquiring a second action sequence from the controlled equipment when the labeling personnel execute the first user instruction on the basis of the first page, and splicing by taking the first action sequence as a front-section action sequence and taking the second action sequence as a rear-section action sequence to obtain a complete action sequence.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating training data suitable for training GUI agents. Background Technology

[0002] A Graphical User Interface (GUI) agent is an artificial intelligence system driven by a multimodal visual model, capable of simulating human operation to complete interactive tasks on a graphical interface of a real device or platform. When training a GUI agent, the training data used is a sequence of GUI action actions. This sequence is a collection of atomic interaction events in a time sequence generated to complete a higher-level user instruction (e.g., "order a coffee"), including page screenshots, touch or mouse events, and the page elements involved in each step. Due to privacy concerns, GUI action sequences cannot be directly collected from the user side. Therefore, the main method for collecting GUI action sequences is currently through manual data annotation. However, manual annotation is inefficient.

[0003] Currently, there is a desire for improved solutions that can more efficiently acquire training data for training GUI agents. Summary of the Invention

[0004] This specification describes one or more embodiments of a method and apparatus for generating training data suitable for training GUI agents, so as to obtain training data for training GUI agents more efficiently. The specific technical solution is as follows.

[0005] In a first aspect, the embodiments provide a method for generating training data suitable for training GUI agents, including:

[0006] The first user instruction to generate training data is input into the large model, and the large model converts the first user instruction into a sequence of planning sub-instructions, which contains several sub-instructions.

[0007] The planning sub-instruction sequence is matched with several sub-instruction sequences in the reuse library. When the first few sub-instructions in the planning sub-instruction sequence are successfully matched with the first sub-instruction sequence, the corresponding first action sequence and the page address of the first page are obtained. The reuse library stores the correspondence between the first sub-instruction sequence, the first action sequence during the execution of the first sub-instruction sequence, and the page address of the first page obtained after the execution of the first sub-instruction sequence.

[0008] The controlled device is controlled to display the first page according to the page address;

[0009] A second action sequence is obtained from the controlled device. The second action sequence is collected when the annotator performs an annotation operation on the first page to execute the first user instruction.

[0010] The first action sequence is used as the preceding action sequence, and the second action sequence is used as the following action sequence. These are then concatenated to obtain the complete action sequence of the first user instruction.

[0011] In one implementation, the step of matching the planning sub-instruction sequence with several sub-instruction sequences in the reuse library includes:

[0012] For any sub-instruction sequence in the reuse library, semantic matching of the sub-instruction sequence with the planning sub-instruction sequence is performed sequentially, starting from the first sub-instruction. When all the sub-instructions in the sub-instruction sequence match the first few sub-instructions in the planning sub-instruction sequence, it is determined that the sub-instruction sequence and the planning sub-instruction sequence are successfully matched.

[0013] Select the first sub-instruction sequence from the successfully matched sub-instruction sequences.

[0014] In one implementation, the step of selecting the first sub-instruction sequence from the successfully matched sub-instruction sequences includes: when there are multiple successfully matched sub-instruction sequences and the sequence lengths of the multiple successfully matched sub-instruction sequences are the same, selecting the sub-instruction sequence with the shortest action sequence from the multiple sub-instruction sequences as the first sub-instruction sequence; when the sequence lengths of the multiple successfully matched sub-instruction sequences are different, selecting the sub-instruction sequence with the longest sequence length from the multiple sub-instruction sequences as the first sub-instruction sequence.

[0015] In one implementation, the reuse library also stores the confidence score of each sub-instruction sequence, which is extracted from the action sequence using a large model. The step of selecting the first sub-instruction sequence from the successfully matched sub-instruction sequences includes: when there are multiple successfully matched sub-instruction sequences, selecting the sub-instruction sequence with the highest confidence score from the multiple successfully matched sub-instruction sequences as the first sub-instruction sequence.

[0016] In one implementation, the method further includes:

[0017] A playback sequence is generated based on the sequentially assembled action sequence and displayed to the annotators;

[0018] Receive quality scores and modification data input by annotation personnel for the playback sequence;

[0019] The confidence level of the first sub-instruction sequence is updated based on the quality score and the modified data.

[0020] In one implementation, the method further includes: when there is no sub-instruction sequence in the reuse library that successfully matches the first few sub-instructions, displaying the first user instruction to the annotator, and obtaining the action sequence of the first user instruction based on the collected data of the annotation operation, wherein the annotation operation is performed by the annotator when executing the first user instruction in the controlled device.

[0021] In one implementation, the correspondence is obtained in the following way:

[0022] The first sub-instruction sequence is extracted from the first action sequence of the second user instruction using a large model; wherein, the first action sequence is collected based on the annotation operations of the annotator.

[0023] Determine the page address of the first page obtained after executing the first action sequence;

[0024] The first sub-instruction sequence, the first action sequence, and the page address of the first page are stored in the reuse library as the correspondence.

[0025] In one implementation, the method further includes:

[0026] When the first sub-instruction sequence contains multiple sub-instructions, a prompt word is constructed based on the first sub-instruction sequence, the first action sequence, and the task instruction;

[0027] The prompt words are input into the large model to obtain a third sub-instruction sequence and a corresponding third action sequence; wherein, the task instruction indicates that the first few sub-instructions are extracted from the first sub-instruction sequence according to semantics to obtain the third sub-instruction sequence, and a third action sequence corresponding to the third sub-instruction sequence is extracted from the first action sequence.

[0028] Determine the page address of the second page obtained after the third action sequence is executed;

[0029] The third sub-instruction sequence, the third action sequence, and the page address of the second page are stored in the reuse library accordingly.

[0030] In one implementation, after obtaining the first action sequence, the method further includes:

[0031] Generate new user instructions based on the remaining sub-instructions in the planning sub-instruction sequence;

[0032] The new user instructions are displayed to the annotation personnel;

[0033] The step of obtaining the second action sequence from the controlled device includes:

[0034] A second action sequence for the new user instruction is obtained from the controlled device. The second action sequence is collected by the annotator when performing annotation operations on the first page to execute the new user instruction.

[0035] In one implementation, the action sequence includes taking a screenshot of a page, performing an atomic operation on the screenshot, and the page element on which the atomic operation acts in the screenshot.

[0036] Secondly, the embodiments provide a training data generation apparatus suitable for training GUI agents, comprising:

[0037] The sub-instruction generation module is configured to input the first user instruction to be generated training data into the large model, and convert the first user instruction into a planned sub-instruction sequence through the large model. The sub-instruction sequence contains several sub-instructions.

[0038] The reuse library matching module is configured to match the planning sub-instruction sequence with several sub-instruction sequences in the reuse library. When the first few sub-instructions in the planning sub-instruction sequence successfully match the first sub-instruction sequence, the corresponding first action sequence and the page address of the first page are obtained. The reuse library stores the correspondence between the first sub-instruction sequence, the first action sequence during the execution of the first sub-instruction sequence, and the page address of the first page obtained after the execution of the first sub-instruction sequence.

[0039] The page display module is configured to control the controlled device to display the first page based on the page address;

[0040] The data acquisition module is configured to acquire a second action sequence from the controlled device. The second action sequence is collected when the annotator performs an annotation operation on the first page to execute the first user instruction.

[0041] The sequence splicing module is configured to splice the first action sequence as the preceding action sequence and the second action sequence as the following action sequence to obtain the complete action sequence of the first user instruction.

[0042] Thirdly, the embodiments provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in any one of the first aspects.

[0043] Fourthly, an embodiment provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in any one of the first aspects.

[0044] In the methods and apparatus provided in the embodiments of this specification, when generating training data, the first user instruction is converted into a theoretical sequence of planned sub-instructions using a large model. This planned sub-instruction sequence is then matched with several sub-instruction sequences in a reuse library. The first action sequence corresponding to the first sub-instruction sequence in the reuse library is reused as the preceding action sequence. This allows the annotator to continue subsequent annotation operations from the first page obtained after executing the first action sequence, thereby obtaining the following action sequence. By reusing action sequences from the reuse library, the embodiments reduce repetitive annotation operations by the annotator. The annotator only needs to annotate the following action sequence, thus reducing the manpower and time costs during data collection and enabling more efficient acquisition of training data for training the GUI agent. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0046] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this application;

[0047] Figure 2 A schematic diagram illustrating the principle by which annotators execute annotation operations based on three user instructions to obtain training data;

[0048] Figure 3 This is a schematic diagram illustrating the optimized annotation process for three user instructions provided in this embodiment.

[0049] Figure 4 A flowchart illustrating a method for generating training data for training GUI agents, provided as an embodiment;

[0050] Figure 5 A schematic diagram illustrating the principle of the matching process between the planning sub-instruction sequence and the reuse library provided in this embodiment;

[0051] Figure 6 A flowchart illustrating the construction of a reusable library is provided for this embodiment.

[0052] Figure 7This is a schematic block diagram of a training data generation device suitable for training GUI agents, provided as an embodiment. Detailed Implementation

[0053] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0054] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this application. It includes an annotation device, a controlled device, a large model, a reuse library, a user device, and a GUI Agent. The annotation device is used to acquire training data for training the GUI agent. The annotation device holds user instructions and inputs them into the large model, which converts the text of the user instructions into the text of a planned sub-instruction sequence. The annotation device matches the planned sub-instruction sequence with multiple sub-instruction sequences in the reuse library to obtain the action sequence and page address corresponding to the sub-instruction sequence that matches the first few sub-instructions of the planned sub-instruction sequence. This action sequence is the initial action sequence in the training data to be acquired. Next, the annotation device controls the controlled device to display the page corresponding to the page address. In this way, the annotator can start performing annotation operations from this page. The controlled device collects the subsequent action sequence based on the annotator's annotation operations and sends it to the annotation device. The annotation device concatenates the initial and subsequent action sequences to obtain the complete action sequence of the user instructions. In this way, the annotation device can obtain action sequences of multiple user instructions as training data. This training data is used to train the GUI agent. After training, the GUI agent can provide online services to execute user commands and control user devices.

[0055] A GUI Agent is an intelligent system based on multimodal large language models and visual language models. It can understand user needs through natural language commands and autonomously analyze GUI interfaces, such as screenshots and user interface (UI) structures, to generate action sequences like clicks, inputs, and drags to complete complex tasks. For example, a user can simply say, "Order an Americano from [store name] on [app name]," and the GUI Agent can automatically recognize and open the app, open [store name], and click on the Americano to complete the order. The execution process of the GUI Agent includes understanding user intent, automatically generating GUI actions, and a closed loop of execution and verification.

[0056] During the training phase, user instructions, also known as task intents or task commands, are instructions provided to the annotation team to obtain training data and for them to execute. During the online execution phase, user instructions are tasks proposed by the user and requested by the GUI agent to be completed. The meaning of user instructions differs between these phases. Annotators and users are distinct. Users are those who execute user instructions online using the GUI Agent. Annotators are those who specifically perform click operations to obtain training data.

[0057] Training data for GUI agents typically consists of interaction trajectories generated by human operators when performing specific tasks. Each training data set includes the following core elements: user commands, interface state sequences, atomic operation sequences, and alignment information between operations and the interface. User commands are task objectives described in natural language, representing high-level task intentions, such as "open xxapp". Interface state sequences are screenshots of the page and / or UI element structures (e.g., control types, text, and positions) at each moment during execution. Atomic operation sequences correspond to the specific actions performed by the human in each interface state, such as clicking, entering text, or swiping upwards. Alignment information between operations and the interface indicates which target page element each operation applies to (e.g., clicking the "Login" button, clicking "coordinates (x, y)", entering the text "123" and marking its position in the screenshot or UI tree path). The training data may also optionally include DOM / control tree information for each page screenshot, as well as network requests (or receipts) and session context (device, resolution, language, etc.) corresponding to the events.

[0058] In other words, a training dataset consists of a user command and its corresponding action sequence. The action sequence, also known as an operation trajectory, can specifically include a series of page screenshots, atomic operations performed on each screenshot, and the page elements affected by these atomic operations within the screenshots. Atomic operations include clicks, double-clicks, drags, and swipes. The action sequence contains data from multiple time steps, organized chronologically. Each time step includes a page screenshot, the atomic operation, and the page element it affects. An atomic operation is the smallest unit of operation.

[0059] Among them, user commands and page screenshots are sample data that require input from the GUI Agent, while atomic operations and the page elements that perform them are labeled data used to determine the prediction loss.

[0060] During the training of the GUI Agent, only one time-step sample is input at a time—the current user command and the current page screenshot (or interface state). The GUI Agent predicts the next action to be performed and the page element it will affect. This prediction is then compared with the labeled operation (atomic operation and its page element) for that time step, the loss is calculated, and the model is updated. In each subsequent step, the same user command and a new screenshot are input into the GUI Agent. This application does not involve any improvement to the training process of the GUI Agent.

[0061] As explained above regarding training data, actual user clicks on the application are required as training data. Due to privacy concerns, user-side data cannot be directly obtained for training. Therefore, training data is typically obtained through manual annotation. Here, annotation means obtaining training data through manual labeling. Each piece of training data contains user commands and corresponding action sequences. The atomic operations within the action sequences and the page elements they affect constitute the label data during training.

[0062] The annotators will perform actual task clicks on a mobile phone or simulator (i.e., a controlled device) equipped with data acquisition software, according to user instructions. The acquisition software will save information such as page screenshots, touch or mouse events, and the page elements affected during the process. Touch or mouse events refer to atomic operations.

[0063] However, the actual execution of user commands usually requires many steps, which are quite cumbersome; moreover, a large number of user commands also consume a lot of manpower and time.

[0064] Figure 2 This diagram illustrates the principle of how annotators perform annotation operations on three user commands to obtain training data. Multiple page screenshots are shown, with dashed circles representing click operations and dashed lines representing the sequence of operations. User command 1: Order an Americano from YY Store on xxapp. User command 2: Order a latte from YY Store on xxapp. User command 3: Order an oat coffee from YY Store on xxapp. For each user command, the annotator performs the corresponding operation on the controlled device. For example, for user command 1, the annotator clicks the search box on the app's homepage, clicks the search button on the search input page, clicks YY Store on the search results page, clicks the online order button on the YY Store's coffee shop homepage, selects Americano on the YY Store order page, clicks the "Proceed to Checkout" button, and clicks the "Proceed to Payment" button on the shopping cart page. After performing a step, the next page appears. For example, clicking the "Proceed to Checkout" button on the YY Store order page will lead to the shopping cart page.

[0065] Understandable, Figure 2 This is merely an example; some steps may have been omitted. From Figure 2 It is evident that the annotator needs to perform at least 21 steps to execute these three user instructions.

[0066] To improve the efficiency of acquiring training data and reduce labor costs, this embodiment provides a training data generation method. In this method, training data obtained through manual annotation is stored in a reuse library. For subsequent user commands, the action sequence corresponding to the matching sub-command sequence in the reuse library is reused. Annotators only need to execute the annotation operation at the latter part of the user command, without having to execute the entire annotation operation, effectively reducing the labor and time costs during data collection.

[0067] For example, for Figure 2 In this embodiment, the first four pages, operations, and page elements involved in the operations are identical across the three user command action sequences. However, the page elements operated on starting from the YY Store order page are all different. Therefore, in this embodiment, the sub-command sequences corresponding to the action sequences of the first four pages in user command 1 can be extracted using a large model, and these sub-command sequences and action sequences are stored in a reuse library. When user commands 2 and 3 need to be executed, they can be converted into theoretically planned sub-command sequences using the large model. These planned sub-command sequences are then matched with the sub-command sequences in the reuse library, and the resulting action sequences are directly used as the front-end action sequences for user commands 2 and 3. These front-end action sequences do not require manual operation by the annotator. When executing user commands 2 and 3, the annotator can directly start execution from the YY Store order page.

[0068] Figure 3 This is a schematic diagram illustrating the optimized annotation process for three user commands provided in the embodiment. As can be seen, when executing user commands 2 and 3, the annotator begins the annotation operation from the YY store order page. Figure 3 It is evident that the annotation personnel need to perform 13 steps to execute these three user commands, compared to... Figure 2 The annotation method shown reduces the number of clicks by 8.

[0069] Figure 1This is merely one scenario for applying the method provided in this embodiment. In practical applications, the reuse library can be implemented in the annotation device or other devices. In this specification, the controlled device can be a controlled mobile phone, tablet computer, desktop computer, or other simulator. The annotation device can be implemented through any device, equipment, platform, device cluster, etc., with computing and processing capabilities. The controlled device is equipped with data acquisition software that can collect data from the annotation personnel's execution process. The controlled device can be, for example, a mobile phone, tablet, computer, or other device with a screen that can display the app's page and execute operation events on the page. The annotation device can remotely control the controlled device. The annotation device can also call up large models.

[0070] In this specification, "Large Language Model" (LLM) refers to a natural language processing model based on deep learning techniques. Its parameter count typically ranges from billions to hundreds of billions or even higher, possessing powerful language understanding and generation capabilities. LLMs can employ the Transformer architecture or its variants (such as GPT and BERT), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. By pre-training on large-scale corpora, LLMs learn the statistical features and semantic relationships of language, enabling them to generalize effectively. The core capabilities of LLMs include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the LLM to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0071] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0072] It should be noted that the key technologies of large language models can be found in the paper "A Survey of..." The detailed description in LargeLanguage Models (Paper No.: arXiv:2303.18223v16, published on March 11, 2025) will not be repeated here.

[0073] The following is combined Figure 4 The embodiments are described in detail below.

[0074] Figure 4 This is a flowchart illustrating a method for generating training data for training GUI agents, provided as an embodiment. The method can be executed via a labeling device and includes the following steps.

[0075] Step S410: Input the first user instruction U1, which is to be used to generate training data, into the large model LLM. The large model LLM converts the first user instruction U1 into the theoretical planning sub-instruction sequence UC1.

[0076] Here, the first user instruction U1 is any user instruction used to generate training data. This first user instruction U1 can be represented in text form, but is not limited to. For example, the first user instruction U1 could be something like "Open xx app for me" or "Order a latte from yy store for me on xx app." The first user instruction U1 is a user instruction that has not yet been labeled by annotators; that is, its action sequence data does not exist, and therefore there is no corresponding actual sub-instruction sequence. Through a large model, the theoretical text of the sub-instruction sequence of the first user instruction U1 can be obtained, which is called the planned sub-instruction sequence UC1.

[0077] The planned sub-instruction sequence UC1 contains several sub-instructions, which have logical dependencies on each other, thus decomposing the user instruction into smaller instruction units. The planned sub-instruction sequence UC1 is a planned decomposition of the specific execution steps of the user instruction. In other words, the planned sub-instruction sequence contains several sub-instructions executed in a logically dependent order. The sub-instruction sequences mentioned in this embodiment all contain several sub-instructions. "Several" includes one or more, i.e., at least one.

[0078] For example, for the user instruction "Please order me a latte from YY store on xxapp", the planned sub-instruction sequence could be as follows:

[0079] Open xxapp → Open the YY store order page in xxapp → Select latte coffee on the YY store order page → After selecting latte coffee on the YY store order page, click the "Go to Checkout" button → On the shopping cart page where you selected latte coffee, click the "Go to Payment" button (1)

[0080] The planned sub-instruction sequence contains five sub-instructions. It should be noted that the user instructions and the instructions in the sub-instruction sequence involved in this application are not low-level instructions in a computer program. User instructions are high-level intentions and can be represented using natural language. The sub-instructions in the sub-instruction sequence are decomposition instructions of user instructions, i.e., high-level intentions. Each sub-instruction is a semantically complete natural language intention, which is low-level relative to the user instruction.

[0081] When inputting the text of the first user instruction U1 into the large-scale model LLM, the first user instruction U1 can be constructed as a cue word. Inputting the cue word into the large-scale model LLM yields the corresponding planning sub-instruction sequence UC1. The cue word contains the first user instruction U1 and a task instruction. The task instruction is a type of indicative text used to instruct the conversion of the first user instruction U1 into a planning sub-instruction sequence consisting of at least one sub-instruction. The cue word may also contain conversion examples.

[0082] Step S420: Match the planning sub-instruction sequence UC1 with several sub-instruction sequences in the Reuse Library (RL). When the first few sub-instructions in the planning sub-instruction sequence UC1 are successfully matched with the first sub-instruction sequence X1, the corresponding first action sequence S1 and the page address address1 of the first page P1 are obtained.

[0083] Among them, the first sub-instruction sequence X1 is a sub-instruction sequence.

[0084] The reuse library RL stores the correspondence between the first sub-instruction sequence X1, the first action sequence S1, and the page address (address1) of the first page P1. The first action sequence S1 is the sequence of actions performed during the execution of the first sub-instruction sequence X1, and the first page P1 is the page obtained after the execution of the first sub-instruction sequence X1. The reuse library RL contains multiple sub-instruction sequences and their corresponding action sequences and page addresses, including the aforementioned first sub-instruction sequence X1, its corresponding first action sequence S1, and the page address (address1). The first page P1 can be a page in the application, and the page address (address1) can be the page's Uniform Resource Locator (URL) address.

[0085] The reuse library contains multiple such correspondences, which are constructed based on annotations by annotators. In other words, these correspondences are obtained and stored based on manually annotated historical training data. The structure of the sub-instruction sequences in the reuse library is the same as the structure of the planning sub-instruction sequence UC1, both containing several sub-instructions. The number of sub-instructions contained in the multiple sub-instruction sequences in the reuse library varies.

[0086] During matching, for any sub-instruction sequence in the multiplexing library RL (e.g., represented by x), semantic matching of the sub-instruction sequence x with the planned sub-instruction sequence UC1 is performed sequentially, starting from the first sub-instruction. When all sub-instructions in the sub-instruction sequence x match the first few sub-instructions in the planned sub-instruction sequence UC1, it is determined that the sub-instruction sequence x and the planned sub-instruction sequence UC1 are successfully matched. The above matching operation is performed on all sub-instruction sequences in the multiplexing library RL to obtain a number of successfully matched sub-instruction sequences.

[0087] Next, the first sub-instruction sequence X1 can be selected from the successfully matched sub-instruction sequences.

[0088] When there is only one successfully matched sub-instruction sequence, directly use that sub-instruction sequence as the first sub-instruction sequence X1.

[0089] When there are multiple successfully matched sub-instruction sequences, and the sequence lengths of these successful sub-instruction sequences are the same, the sub-instruction sequence with the shortest action sequence is selected as the first sub-instruction sequence X1. When there are multiple successfully matched sub-instruction sequences, and the sequence lengths of these successful sub-instruction sequences are different, the sub-instruction sequence with the longest sequence length is selected as the first sub-instruction sequence X1.

[0090] If the reuse library also stores the confidence score for each sub-instruction sequence, which is extracted from the action sequence using a large model, then the sub-instruction sequence with the highest confidence score can be selected as the first sub-instruction sequence X1 from among the multiple successfully matched sub-instruction sequences.

[0091] When selecting the first sub-instruction sequence X1, the length of the action sequence and the confidence level of the sub-instruction sequence can also be considered.

[0092] When matching the sub-instruction sequence x with the planned sub-instruction sequence UC1, semantic matching can be performed between the first sub-instruction in the planned sub-instruction sequence UC1 and the first sub-instruction in the sub-instruction sequence x to determine whether they match. For example, semantic matching can be performed on the text embedding vectors of the two sub-instructions. If the similarity between the two is greater than a preset threshold, the two sub-instructions are considered to be matched. If the similarity is not greater than the preset threshold, the two sub-instructions are considered not to be matched.

[0093] When the two sub-instructions mentioned above reach a matching state, the next sub-instruction in the planned sub-instruction sequence UC1 is semantically matched with the next sub-instruction sequence in the sub-instruction sequence x to determine whether they reach a matching state. When the two sub-instructions reach a matching state, the above training process continues. This continues until the last sub-instruction is matched. If the two sub-instructions reach a matching state, it is determined that the planned sub-instruction sequence UC1 and the sub-instruction sequence x have successfully matched; if the two sub-instructions do not reach a matching state, it is determined that the planned sub-instruction sequence UC1 and the sub-instruction sequence x have not successfully matched. For the planned sub-instruction sequence UC1 and the sub-instruction sequence x, the lengths of their sub-instruction sequences may be the same or different. That is to say, the number of the first few sub-instructions in the planned sub-instruction sequence UC1 is not fixed, but is related to the matching result.

[0094] The matching process between the planned sub-instruction sequence UC1 and the multiplexing library RL can be implemented using RAG technology. This involves matching the planned sub-instruction sequence UC1 with multiple sub-instruction sequences in the multiplexing library RL to obtain n successfully matched sub-instruction sequences. Then, the planned sub-instruction sequence UC1 and the n successfully matched sub-instruction sequences are input into a large model. The large model then determines the first sub-instruction sequence X1 that successfully matches the planned sub-instruction sequence UC1 from the n sub-instruction sequences.

[0095] Figure 5This is a schematic diagram illustrating the matching process between the planned sub-instruction sequence and the multiplexing library provided in this embodiment. The planned sub-instruction sequence UC1 includes sub-instruction A → sub-instruction B → sub-instruction D. The multiplexing library RL contains sub-instruction sequences 1 to 4. Sub-instruction sequences 1 and 2 contain the same sub-instruction A, but their corresponding action sequences are different, namely sub-instruction sequences 1 and 2 respectively.

[0096] Semantic matching is performed between the first sub-instruction (sub-instruction A) in the planning sub-instruction sequence UC1 and the first sub-instruction of each sub-instruction sequence in the multiplexing library RL. The result shows that sub-instruction A in the planning sub-instruction sequence UC1 matches sub-instruction A in sub-instruction sequences 1 to 4. Since sub-instruction sequences 1 and 2 contain only one sub-instruction (sub-instruction A), it can be determined that sub-instruction sequences 1 and 2 successfully match the planning sub-instruction sequence UC1.

[0097] Continuing with the matching of the second sub-instruction, we perform a semantic match between the second sub-instruction (sub-instruction B) in the planning sub-instruction sequence UC1 and the second sub-instructions in sub-instruction sequences 3 and 4 of the multiplexing library. The result is that sub-instruction B in the planning sub-instruction sequence UC1 matches with sub-instruction B in both sub-instruction sequences 3 and 4. Since the last sub-instruction (sub-instruction B) in sub-instruction sequence 3 has already been matched, we can confirm that sub-instruction sequence 3 and planning sub-instruction sequence UC1 have successfully matched.

[0098] Continuing with the matching of the third sub-instruction, we perform semantic matching between the third sub-instruction D in the planning sub-instruction sequence UC1 and the third sub-instruction C in the multiplexing library RL. The result is that there is no match between the sub-instruction D in the planning sub-instruction sequence UC1 and the sub-instruction C in the sub-instruction sequence 4.

[0099] In other words, if sub-instruction sequences 1-3 in the reuse library RL match successfully, one of the sub-instruction sequences can be selected as the first sub-instruction sequence X1. For example, sub-instruction sequence 3, which has the most matching sub-instructions, can be selected as the first sub-instruction sequence X1. This can minimize manual annotation operations.

[0100] Assuming that only sub-instruction sequences 1 and 2 match successfully, the shortest sub-instruction sequence among sub-instruction sequences 1 and 2 can be taken as the first sub-instruction sequence X1. This minimizes the number of actions required to complete the same sub-instruction.

[0101] If a sub-instruction sequence that matches the planning sub-instruction sequence UC1 can be successfully retrieved from the reuse library RL, it means that the existing action sequence in the reuse library RL can be reused. That is, this part of the action sequence does not need to be annotated by annotators, thereby reducing manual annotation operations.

[0102] When the reuse library RL does not contain a sub-instruction sequence that successfully matches the first few sub-instructions of the planning sub-instruction sequence UC1, the complete action sequence of the first user instruction U1 must still be obtained by having annotators perform annotation operations. Specifically, the first user instruction U1 can be shown to the annotators, who then perform annotation operations on the controlled device. Based on the data collected from this annotation operation, the action sequence of the first user instruction U1 is obtained. The annotation operation is performed by the annotators when they execute the first user instruction U1 on the controlled device. The controlled device can collect data based on the annotation operation and send the collected data to the annotation device, which then processes the collected data to obtain the action sequence of the first user instruction U1.

[0103] Step S430: Control the controlled device to display the first page S1 according to the page address address1 of the first page S1.

[0104] When the labeling device is the application's server, the controlled device can display the first page S1 in the application by redirecting the address in the application to the page address address1. When the labeling device is not the application's server, the labeling device can send the page address address1 to the application's server, causing the server to redirect the address in the application to the page address address1. In practical applications, there are multiple ways to make the controlled device display the first page S1.

[0105] Step S440: Obtain the second action sequence S2 from the controlled device. The second action sequence S2 is collected by the annotator when performing the annotation operation based on the first page P1 to execute the first user instruction U1.

[0106] When the controlled device displays the first page S1 to the annotator, the annotator can begin annotating operations from the first page S1 to complete the first user instruction U1. The controlled device can collect data based on the annotating operations and send the collected data to the annotating device. The annotating device receives the collected data from the controlled device and obtains the second action sequence S2 based on the data.

[0107] The controlled device can display the first user instruction U1 to the annotator, enabling the annotator to perform annotation operations starting from the first page S1. It can also generate new user instructions and display them to the annotator, allowing the annotator to perform annotation operations corresponding to the new user instructions starting from the first page S1.

[0108] After obtaining the first action sequence S1, the annotation device can generate new user instructions based on the remaining sub-instructions in the planned sub-instruction sequence UC1. The remaining sub-instructions are the sub-instructions in the planned sub-instruction sequence UC1 excluding the first few sub-instruction sequences. Specifically, the text corresponding to the planned sub-instruction sequence UC1 and the remaining sub-instructions can be input into the large model, which then generates new user instructions based on the remaining sub-instructions. The new user instructions are semantically identical to the remaining sub-instructions; however, the new user instructions are easier for annotators to understand and more in line with natural language expression.

[0109] For example, for Figure 3 For user instruction 2 in the example, when the action sequence corresponding to the first 4 page screenshots is obtained from the reuse library RL, the YY store order page can be displayed on the controlled device, showing the user instruction "Help me order a latte from YY store on xxapp" to the annotator. This allows the annotator to directly select the latte on the YY store order page and click the checkout button, and then click the payment button in the pop-up shopping cart page.

[0110] Alternatively, a new user instruction can be generated corresponding to the aforementioned user instruction: "Order me a latte from YY Store." The annotator will then know more clearly what actions need to be performed on the YY Store order page to complete the new user instruction. Specifically, the annotator can more quickly select the latte on the YY Store order page, click the "Proceed to Checkout" button, and then click the "Pay" button in the pop-up shopping cart. This new user instruction helps users complete the annotation process quickly, thereby improving the efficiency of obtaining the second action sequence S2.

[0111] Step S450: The first action sequence S1 is used as the preceding action sequence and the second action sequence S2 is used as the following action sequence to splice them together to obtain the complete action sequence of the first user instruction U1.

[0112] In this embodiment, a human-machine hybrid active annotation approach is adopted to reduce repetitive annotation operations, thereby significantly reducing the number of manual executions and time consumption under the same annotation budget, improving unit annotation efficiency, and gradually reducing reliance on manual labor and overall annotation costs.

[0113] When the first action sequence S1 is the shortest action sequence in the matching results in step S420, the complete action sequence of the first user instruction U1 obtained in step S450 can be called the minimum replayable context, i.e., the minimum replayable sequence. In other words, the annotation device can generate a replay sequence based on the sequentially concatenated action sequence and display it to the annotator, allowing the annotator to review the replay sequence and receive the quality score and modification data input by the annotator for the replay sequence. Afterwards, the confidence level of the first sub-instruction sequence X1 can be updated based on the quality score and modification data.

[0114] Manually reviewing the concatenated action sequence can identify discontinuities and errors. If errors are found in the playback sequence, it can be excluded from training data to improve training data quality. Updating the confidence level of sub-instruction sequences in the Reuse Library (RL) also helps improve the quality of generated data.

[0115] The implementation can generate a minimal replayable context, which is then manually reviewed to ensure that the generated training data is both high-value and low-cost decidable samples, thereby accelerating model convergence and improving the return on labeling investment.

[0116] Next, let's combine... Figure 6 The construction process of the Reuse Library (RL) is explained.

[0117] Figure 6 This is a schematic diagram of a process for constructing a reuse library provided in an embodiment. The following description uses the generation process of the correspondence between the first sub-instruction sequence X1, the first action sequence S1, and the page address address1 of the first page P1 in the reuse library RL as an example. This construction process includes the following steps 1 to 3.

[0118] Step 1: Extract the first sub-instruction sequence X1 from the first action sequence S1 of the second user instruction U2 using the large model.

[0119] The first action sequence S1 is obtained based on the annotation operation performed by the annotator. The annotator performs the annotation operation when executing the second user instruction U2 in the controlled device. The controlled device collects data based on the annotation operation and sends the collected data to the annotation device. The annotation device obtains the first action sequence S1 based on the collected data sent by the controlled device.

[0120] Specifically, the first action sequence S1 can be input into the large model, and the first sub-instruction sequence can be extracted from the large model. The first action sequence S1 contains a page screenshot and the corresponding atomic operations and the page elements they affect. Therefore, the large model here can be a multimodal large model, capable of handling data in formats such as images and text. The sub-instructions in the first sub-instruction sequence X1 are semantically related and logically connected.

[0121] Step 2: Determine the page address (address1) of the first page P1 obtained after executing the first action sequence S1.

[0122] The annotation device can extract the page address address1 of the first page P1 from the collected data.

[0123] Step 3: Store the first sub-instruction sequence X1, the first action sequence S1, and the page address address1 of the first page P1 as a correspondence in the reuse library RL.

[0124] In step 1, the large model can also output the confidence level of the first sub-instruction sequence X1. In this case, the first sub-instruction sequence X1, the confidence level, the first action sequence S1, and the page address address1 of the first page P1 can be stored as a correspondence in the reuse library RL.

[0125] To enrich the data in the reuse library RL, a single piece of data can be split into multiple pieces for reuse during the RL construction process. In other words, the construction process can also include steps 4 through 6.

[0126] Step 4: When the first sub-instruction sequence X1 contains multiple sub-instructions, construct a prompt word based on the first sub-instruction sequence X1, the first action sequence S1, and the task instruction. Input the prompt word into the large model LLM to obtain the third sub-instruction sequence X3 and the corresponding third action sequence S3.

[0127] Specifically, the task instruction indicates that, based on semantics, it extracts the first few sub-instructions from the first sub-instruction sequence X1 to obtain the third sub-instruction sequence X3, and extracts the third action sequence S3 corresponding to the third sub-instruction sequence X3 from the first action sequence S1. The first sub-instruction sequence X1 contains more sub-instructions than the third sub-instruction sequence X3.

[0128] For example, if the second user instruction U2 is "Order me a cup of Americano from YY store on xxapp", the corresponding first sub-instruction sequence X1 might look like this:

[0129] Open xxapp → Open the YY store order page in xxapp → Select Americano on the YY store order page → After selecting Americano on the YY store order page, click the "Go to Checkout" button → On the shopping cart page where you selected Americano, click the "Go to Payment" button (2)

[0130] The first sub-instruction sequence X1 contains 5 sub-instructions.

[0131] The large model LLM can extract the following third sub-instruction sequence X3 from the first sub-instruction sequence X1:

[0132] Open xxapp → Open the YY store order page in xxapp (3)

[0133] The third sub-instruction sequence X3 contains two sub-instructions.

[0134] Step 5: Determine the page address (address2) of the second page P2 obtained after executing the third action sequence S3.

[0135] Specifically, the labeling device can obtain the page address address2 from the collected data sent by the controlled device.

[0136] Step 6: Store the third sub-instruction sequence X3, the third action sequence S3, and the page address address2 in the reuse library RL.

[0137] Initially, several user commands and their corresponding action sequences can be obtained through manual annotation. Correspondence relationships are then extracted from these user commands and their corresponding action sequences and stored in the reuse library (RL). When faced with a new user command, if a matching sub-command sequence exists in the RL, the action sequence in the RL can be used as the preceding action sequence, and the following action sequence can be obtained using only manual annotation. When no matching sub-command sequence exists in the RL, the complete action sequence is obtained manually, and the correspondence relationships can be extracted from the user command and action sequence, updating the RL.

[0138] In this embodiment, various methods can be used to construct the corresponding relationship and store it in the reuse library RL, thereby enriching the data in the reuse library RL as much as possible.

[0139] When the labeling device receives the collected data (i.e., session data) sent by the controlled device, it can also use an automatic labeler to generate more corresponding label data based on the user instructions and action sequences in the session data. For example, it can include the higher-order intent of each atomic operation, the step semantics of each step, slot candidates, and information such as success / failure.

[0140] In this specification, the terms "first," "first sub-instruction sequence," "first action sequence," and "first page," as well as the corresponding "second" (if present) in the text, are used merely for ease of distinction and description and do not have any limiting meaning.

[0141] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0142] Figure 7 This is a schematic block diagram of a training data generation device suitable for training GUI agents, provided as an embodiment. This device embodiment is related to... Figure 4 The method embodiment shown corresponds to this. The device 700 includes: a sub-instruction generation module 710, configured to input a first user instruction to be used to generate training data into a large model, and convert the first user instruction into a theoretically planned sub-instruction sequence through the large model, the sub-instruction sequence containing several sub-instructions; a reuse library matching module 720, configured to match the planned sub-instruction sequence with several sub-instruction sequences in a reuse library, and when the first few sub-instructions in the planned sub-instruction sequence successfully match the first sub-instruction sequence, obtain the corresponding first action sequence and the page address of the first page. The reuse library stores the correspondence between the first sub-instruction sequence, the first action sequence during the execution of the first sub-instruction sequence, and the page address of the first page obtained after the execution of the first sub-instruction sequence; a page display module 730, configured to control a controlled device to display the first page based on the page address; and a data acquisition module 740, configured to acquire a second action sequence from the controlled device, the second action sequence being collected by annotators when performing annotation operations on the first page to execute the first user instruction. The sequence splicing module 750 is configured to splice the first action sequence as the preceding action sequence and the second action sequence as the following action sequence to obtain the complete action sequence of the first user instruction.

[0143] In one implementation, the reuse library matching module 720 includes a matching submodule 21 and a selection submodule 22. The matching submodule 21 is configured to perform semantic matching between any given sub-instruction sequence in the reuse library and the planned sub-instruction sequence, starting from the first sub-instruction. When all sub-instructions in the given sub-instruction sequence match the first few sub-instructions in the planned sub-instruction sequence, it is determined that the sub-instruction sequence and the planned sub-instruction sequence are successfully matched. The selection submodule 22 is configured to select the first sub-instruction sequence from the successfully matched sub-instruction sequences.

[0144] In one implementation, submodule 22 is selected and specifically configured such that when there are multiple successfully matched sub-instruction sequences and the sequence lengths of the multiple successfully matched sub-instruction sequences are the same, the sub-instruction sequence with the shortest action sequence is selected from the multiple sub-instruction sequences as the first sub-instruction sequence; when the sequence lengths of the multiple successfully matched sub-instruction sequences are different, the sub-instruction sequence with the longest sequence length is selected from the multiple sub-instruction sequences as the first sub-instruction sequence.

[0145] In one implementation, the reuse library also stores the confidence score of each sub-instruction sequence, which is extracted from the action sequence using a large model. Submodule 22 is selected and specifically configured as follows: when there are multiple successfully matched sub-instruction sequences, the sub-instruction sequence with the highest confidence score is selected as the first sub-instruction sequence from among the multiple successfully matched sub-instruction sequences.

[0146] In one implementation, the device 700 further includes a playback display module, a scoring receiving module, and a confidence update module (not shown in the figure). The playback display module is configured to generate a playback sequence based on the sequentially concatenated action sequence and display it to the annotator. The scoring receiving module is configured to receive the quality score and modification data input by the annotator for the playback sequence. The confidence update module is configured to update the confidence of the first sub-instruction sequence based on the quality score and modification data.

[0147] In one implementation, the device 700 further includes a data acquisition module 760, configured to display a first user instruction to the annotator when there is no sub-instruction sequence in the reuse library that successfully matches the previous few sub-instructions, and to obtain the action sequence of the first user instruction based on the acquired data of the annotation operation. The annotation operation is performed by the annotator when executing the first user instruction in the controlled device.

[0148] In one implementation, the device 700 further includes a reuse library construction module 770, configured to construct a reuse library. The reuse library construction module 770 specifically includes an extraction submodule 71, a first determination submodule 72, and a first storage submodule 73. The extraction submodule 71 is configured to extract a first sub-instruction sequence from the first action sequence of the second user instruction using a large model. The first action sequence is obtained based on annotation operations performed by annotators. The first determination submodule 72 is configured to determine the page address of the first page obtained after executing the first action sequence. The first storage submodule 73 is configured to store the first sub-instruction sequence, the first action sequence, and the page address of the first page as a correspondence in the reuse library.

[0149] In one implementation, the reuse library construction module 770 further includes a construction submodule 74, a splitting submodule 75, a second determining submodule 76, and a second storage submodule 77. The construction submodule 74 is configured to construct a prompt word based on the first sub-instruction sequence, the first action sequence, and the task instruction when the first sub-instruction sequence contains multiple sub-instructions. The splitting submodule 75 is configured to input the prompt word into the large model to obtain a third sub-instruction sequence and a corresponding third action sequence. The task instruction indicates that the first few sub-instructions of the first sub-instruction sequence are extracted semantically to obtain the third sub-instruction sequence, and a third action sequence corresponding to the third sub-instruction sequence is extracted from the first action sequence. The second determining submodule 76 is configured to determine the page address of the second page obtained after executing the third action sequence. The second storage submodule 77 is configured to store the third sub-instruction sequence, the third action sequence, and the page address of the second page in the reuse library.

[0150] In one implementation, device 700 further includes a new instruction generation module and an instruction display module (not shown in the figure). The new instruction generation module is configured to generate a new user instruction based on the remaining sub-instructions in the planned sub-instruction sequence after obtaining the first action sequence. The instruction display module is configured to display the new user instruction to the annotator. The data acquisition module 740 is specifically configured to acquire a second action sequence for the new user instruction from the controlled device. This second action sequence is collected by the annotator when performing annotation operations to execute the new user instruction based on the first page.

[0151] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0152] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform... Figures 1 to 6 Any one of the methods described.

[0153] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement... Figures 1 to 6 Any one of the methods described.

[0154] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0155] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0156] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating training data suitable for training graphical user interface (GUI) agents, comprising: The first user instruction to generate training data is input into the large model, and the large model converts the first user instruction into a sequence of planning sub-instructions, which contains several sub-instructions. The planning sub-instruction sequence is matched with several sub-instruction sequences in the reuse library. When the first few sub-instructions in the planning sub-instruction sequence are successfully matched with the first sub-instruction sequence, the corresponding first action sequence and the page address of the first page are obtained. The reuse library stores the correspondence between the first sub-instruction sequence, the first action sequence during the execution of the first sub-instruction sequence, and the page address of the first page obtained after the execution of the first sub-instruction sequence. The controlled device is controlled to display the first page according to the page address; A second action sequence is obtained from the controlled device. The second action sequence is collected when the annotator performs an annotation operation on the first page to execute the first user instruction. The first action sequence is used as the preceding action sequence, and the second action sequence is used as the following action sequence. These are then concatenated to obtain the complete action sequence of the first user instruction.

2. The method according to claim 1, wherein the step of matching the planning sub-instruction sequence with a plurality of sub-instruction sequences in the reuse library comprises: For any sub-instruction sequence in the reuse library, semantic matching of the sub-instruction sequence with the planning sub-instruction sequence is performed sequentially, starting from the first sub-instruction. When all the sub-instructions in the sub-instruction sequence match the first few sub-instructions in the planning sub-instruction sequence, it is determined that the sub-instruction sequence and the planning sub-instruction sequence are successfully matched. Select the first sub-instruction sequence from the successfully matched sub-instruction sequences.

3. The method according to claim 2, wherein the step of selecting the first sub-instruction sequence from the successfully matched sub-instruction sequences comprises: When there are multiple successfully matched sub-instruction sequences, and the sequence lengths of the multiple successfully matched sub-instruction sequences are the same, the sub-instruction sequence with the shortest action sequence is selected from the multiple sub-instruction sequences as the first sub-instruction sequence. When multiple successfully matched sub-instruction sequences have different sequence lengths, the sub-instruction sequence with the longest sequence length is selected from the multiple sub-instruction sequences as the first sub-instruction sequence.

4. The method according to claim 2, wherein the reuse library further stores the confidence score of each sub-instruction sequence, which is extracted from the action sequence using a large model; the step of selecting the first sub-instruction sequence from the successfully matched sub-instruction sequences includes: When there are multiple successfully matched sub-instruction sequences, the sub-instruction sequence with the highest confidence is selected from the multiple successfully matched sub-instruction sequences as the first sub-instruction sequence.

5. The method according to claim 1, further comprising: A playback sequence is generated based on the sequentially assembled action sequence and displayed to the annotators; Receive quality scores and modification data input by annotation personnel for the playback sequence; The confidence level of the first sub-instruction sequence is updated based on the quality score and the modified data.

6. The method of claim 1, further comprising: When there is no sub-instruction sequence in the reuse library that successfully matches the first few sub-instructions, the first user instruction is shown to the annotator. Based on the collected data of the annotation operation, the action sequence of the first user instruction is obtained. The annotation operation is performed by the annotator when executing the first user instruction in the controlled device.

7. The method according to claim 1, wherein the correspondence is obtained in the following manner: The first sub-instruction sequence is extracted from the first action sequence of the second user instruction using a large model; where... The first action sequence was collected based on the annotation operations of the annotators; Determine the page address of the first page obtained after executing the first action sequence; The first sub-instruction sequence, the first action sequence, and the page address of the first page are stored in the reuse library as the correspondence.

8. The method of claim 7, further comprising: When the first sub-instruction sequence contains multiple sub-instructions, a prompt word is constructed based on the first sub-instruction sequence, the first action sequence, and the task instruction; The prompt words are input into the large model to obtain a third sub-instruction sequence and a corresponding third action sequence; wherein, the task instruction indicates that the first few sub-instructions are extracted from the first sub-instruction sequence according to semantics to obtain the third sub-instruction sequence, and a third action sequence corresponding to the third sub-instruction sequence is extracted from the first action sequence. Determine the page address of the second page obtained after the third action sequence is executed; The third sub-instruction sequence, the third action sequence, and the page address of the second page are stored in the reuse library accordingly.

9. The method according to claim 1, further comprising, after obtaining the first action sequence: Generate new user instructions based on the remaining sub-instructions in the planning sub-instruction sequence; The new user instructions are displayed to the annotation personnel; The step of obtaining the second action sequence from the controlled device includes: A second action sequence for the new user instruction is obtained from the controlled device. The second action sequence is collected by the annotator when performing annotation operations on the first page to execute the new user instruction.

10. The method according to claim 1, wherein the action sequence includes taking a screenshot of a page, performing an atomic operation on the screenshot, and the page element on which the atomic operation acts in the screenshot.

11. A training data generation device suitable for training GUI agents, comprising: The sub-instruction generation module is configured to input the first user instruction to be generated training data into the large model, and convert the first user instruction into a planned sub-instruction sequence through the large model. The sub-instruction sequence contains several sub-instructions. The reuse library matching module is configured to match the planning sub-instruction sequence with several sub-instruction sequences in the reuse library. When the first few sub-instructions in the planning sub-instruction sequence successfully match the first sub-instruction sequence, the corresponding first action sequence and the page address of the first page are obtained. The reuse library stores the correspondence between the first sub-instruction sequence, the first action sequence during the execution of the first sub-instruction sequence, and the page address of the first page obtained after the execution of the first sub-instruction sequence. The page display module is configured to control the controlled device to display the first page based on the page address; The data acquisition module is configured to acquire a second action sequence from the controlled device. The second action sequence is collected when the annotator performs an annotation operation on the first page to execute the first user instruction. The sequence splicing module is configured to splice the first action sequence as the preceding action sequence and the second action sequence as the following action sequence to obtain the complete action sequence of the first user instruction.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-10.

13. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-10.

Images