Training data generation method and apparatus, terminal, and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a training data generation method and device, a terminal and a storage medium. The method comprises the following steps: constructing an operation network graph; wherein the nodes of the operation network graph represent different graphical user interface states, and the directed edges represent operation actions that can be executed under the interface states represented by the corresponding nodes; an instruction-state association relationship between a user natural language instruction and the nodes in the operation network graph is established, and based on the association relationship, a task completion path from a starting interface state node to an interface state node used for completing the user natural language instruction is determined; for each pair of adjacent interface state nodes in the task completion path, decision explanatory text matched with the interface state change represented by the current node pair and the corresponding operation action is generated; and the interface state, the operation action and the decision explanatory text are one-to-one corresponding according to the order of the adjacent interface state nodes in the task completion path to form a track training sample.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training data generation method, apparatus, terminal and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, graphical user interface (GUI) agents need to execute corresponding operations and generate corresponding reasoning text in various applications based on users' natural language instructions to support supervised learning training. Currently, the training data relied upon by such agents is mainly collected manually: annotators execute operations one by one according to specified instructions in real or simulated interface environments, and simultaneously record the interface state, operation, and reasoning text corresponding to each step, ultimately forming a complete trajectory. This method has obvious limitations: First, manual operation is costly, requiring a large amount of manpower to repeatedly execute similar processes to cover different applications, versions, and user paths; second, the efficiency of generating a single trajectory is low, requiring the complete completion of all steps from the start to the goal, making it impossible to reuse intermediate states or cross-path combinations; third, the trajectory coverage is limited, making it difficult to systematically cover sparse scenarios such as interface jump branches, abnormal state entry points, and long path end nodes. Summary of the Invention

[0003] In view of this, embodiments of this application provide a training data generation method, apparatus, terminal, and storage medium, which can effectively solve the problems of high cost, low efficiency, and insufficient coverage of manual trajectory collection in the prior art.

[0004] In a first aspect, embodiments of this application provide a training data generation method, including: Construct an operation network graph; wherein the operation network graph is a directed graph, the nodes of the directed graph represent different graphical user interface states, and the directed edges of the directed graph represent the operation actions that can be performed in the interface state represented by the corresponding node. Establish an instruction-state association between the user's natural language instruction and at least one node in the operation network graph, and based on the instruction-state association, determine the task completion path from the starting interface state node to the interface state node used to complete the user's natural language instruction. For each pair of adjacent interface state nodes in the task completion path, generate decision-explanatory text that matches the interface state change and corresponding operation action represented by the current node pair. The interface state, the operation action, and the decision explanation text are matched one-to-one according to the order of the adjacent interface state nodes in the task completion path to form a trajectory training sample.

[0005] In an optional implementation, the construction of the operation network graph includes: Based on the initial interface state node, the interactive area is identified for the interface state represented by the current node to obtain candidate interactive areas. The interface simulates the execution environment to perform corresponding operations on each of the candidate interactive areas, and the candidate interactive areas whose interface state changes after the operation are consistent with the preset state changes are taken as valid interactive areas. Based on each effective interactive area and its corresponding operation action and post-operation interface state, new directed edges and new nodes are constructed, and the new directed edges and new nodes are added to the operation network graph.

[0006] In an optional implementation, establishing the instruction-state association between the user's natural language instructions and at least one node in the operation network graph includes: Semantically match the user's natural language instructions with at least one interface state node in the operation network graph; Based on the semantic consistency between the task context information carried by the interface state and the task objective represented by the user's natural language instruction, the instruction-state association is determined; wherein, the instruction-state association supports the association of the same user natural language instruction to multiple interface state nodes with different task reachability.

[0007] In an optional implementation, determining the task completion path from the starting interface state node to the interface state node used to complete the user's natural language instruction based on the instruction-state association includes: In response to the user's natural language instruction, at least one interface state node for completing the current user's natural language instruction is located as a target node in the operation network graph. Using the starting interface state node as the root node, perform a graph search in the operation network graph to obtain at least one reachable path from the root node to each of the target nodes; The reachable paths are filtered according to preset filtering rules, and at least one path that satisfies the preset filtering rules is determined as the task completion path; wherein, the preset filtering rules include minimizing the number of interface state nodes contained in the path.

[0008] In an optional implementation, generating decision-explanatory text that matches the interface state change and corresponding operation action represented by the node for each pair of adjacent interface state nodes in the task completion path includes: Based on the user's natural language instructions, the visual content of the interface corresponding to the adjacent interface state nodes, and the operation actions connecting the current node pair, the decision-explanatory text corresponding to the current adjacent interface state node is generated.

[0009] In an optional implementation, it further includes: The effectiveness of all generated trajectory training samples is evaluated to obtain the completion rate of each trajectory training sample, and the trajectory training samples with a completion rate lower than a preset threshold are regarded as difficult sample samples. Extract at least one sub-path from the task completion path corresponding to the difficult example sample, and use the sub-path as a new task completion path to re-execute the step of generating decision explanatory text to obtain the corresponding sub-trajectory training sample; wherein, the sub-path is composed of multiple consecutive interface state nodes in the task completion path.

[0010] In an optional implementation, it further includes: In response to any of the trajectory training samples, based on a preset amplification strategy, the user natural language instructions and / or the decision explanatory text in the trajectory training samples are stylized and rewritten; wherein, the stylization and rewriting maintains the original semantics unchanged, and the generated stylized texts are different from each other in terms of expression dimensions, including speaker attribute dimensions and / or task scenario dimensions. Based on the text obtained after stylization rewriting, at least one stylized trajectory training sample is generated.

[0011] Secondly, embodiments of this application provide a training data generation apparatus, comprising: An operation network graph construction module is used to construct an operation network graph; wherein, the operation network graph is a directed graph, the nodes of the directed graph represent different graphical user interface states, and the directed edges of the directed graph represent the operation actions that can be performed in the interface state represented by the corresponding node. The task completion path generation module is used to establish an instruction-state association relationship between the user's natural language instruction and at least one node in the operation network graph, and based on the instruction-state association relationship, determine the task completion path from the starting interface state node to the interface state node used to complete the user's natural language instruction. The decision explanation text generation module is used to generate decision explanation text that matches the interface state change and corresponding operation action represented by the current node for each pair of adjacent interface state nodes in the task completion path. The trajectory training sample generation module is used to correspond the interface state, the operation action, and the decision explanation text one by one according to the order of the adjacent interface state nodes in the task completion path to form a trajectory training sample.

[0012] Thirdly, embodiments of this application provide a terminal device, the terminal device including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the training data generation method described above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed on a processor, implements the above-described training data generation method.

[0014] The embodiments of this application have the following beneficial effects: This application constructs an operation network graph, establishes the instruction-state association between user natural language commands and interface state nodes, generates decision-explanatory text that matches adjacent interface state node pairs, and maps interface states, operation actions, and decision-explanatory text one-to-one according to the task completion path sequence, thereby achieving the systematic generation of trajectory training samples. This method no longer relies on manual execution and recording of each step; instead, it uses the operation network graph as a unified data base, ensuring that each trajectory originates from a verifiable state transition link, possessing the characteristics of a clear starting point, traceable steps, and reproducible actions. The establishment of the instruction-state association ensures the semantic anchoring of user intent and interface state, avoiding target drift. The node-by-node generation of decision-explanatory text guarantees a strong correspondence between reasoning logic and operation actions. The nodes defined by the task completion path carry the complete task flow structure from start to finish. Overall, without increasing the burden of manual annotation, this method can significantly improve the generation efficiency, structural consistency, and task coverage breadth of trajectory data, thus providing stable and scalable data support for high-quality supervised training of graphical user interface agents. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of an implementation scenario of this application is shown; Figure 2 A first flowchart of the training data generation method according to an embodiment of this application is shown; Figure 3 A schematic diagram of an operational network diagram according to an embodiment of this application is shown; Figure 4 This illustration shows a second flowchart of the training data generation method according to an embodiment of this application; Figure 5A schematic diagram of the third process of the training data generation method according to an embodiment of this application is shown; Figure 6 A schematic diagram of the fourth process of the training data generation method according to an embodiment of this application is shown; Figure 7 The fifth flowchart of the training data generation method according to an embodiment of this application is shown. Detailed Implementation

[0017] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0018] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0020] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0021] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0022] Figure 1This is a schematic diagram of an implementation scenario of a training data generation system disclosed in this application. The system includes an annotation device 100, a controlled device 200, and a database 300. The annotation device 100 includes a training data generation apparatus, configured with multiple functional logic units, including but not limited to an operation network graph construction module 110, a task completion path generation module 120, a decision explanatory text generation module 130, a trajectory training sample generation module 140, a difficult example sample optimization module 150, and a stylization augmentation module 160, etc., used to perform interface state annotation, instruction-state association relationship establishment, and trajectory training sample management, etc. The controlled device 200 is a physical terminal or virtual simulator equipped with an interface simulation execution environment, used to respond to operation instructions and provide feedback on interface state changes. The database 300 is used to store operation network graphs, instruction-state association relationships, task completion paths, decision explanatory text, and various trajectory training samples.

[0023] In some implementations, the annotation device 100 generates trajectory training samples for training graphical user interface agents by receiving operation data from the controlled device 200. Specifically, the annotator performs interface interaction operations on the controlled device 200, and the controlled device 200 collects the interface state, operation actions, and changes in interface state after the operation, and sends the above data to the annotation device 100. The annotation device 100 constructs an operation network graph based on the received data and establishes the instruction-state association relationship between the user's natural language instructions and the interface state nodes in the operation network graph. Subsequently, the annotation device 100 calls the task completion path generation module 120 to determine the task completion path, calls the decision explanatory text generation module 130 to generate decision explanatory text, and finally maps the interface state, operation actions, and decision explanatory text one-to-one according to the order of the task completion path to form trajectory training samples, which are then stored in the database 300.

[0024] In this embodiment, adjacent interface state nodes refer to two consecutive interface state nodes connected in sequence in the task completion path, where the preceding node represents the interface state before the operation and the following node represents the interface state after the corresponding operation is performed.

[0025] A graphical user interface intelligent agent refers to an intelligent system based on a multimodal language model. Its inputs include user natural language commands and the current interface state, and its outputs are operation actions and corresponding decision-making explanatory text. It can be used for automated task execution on mobile, desktop, or web applications. Understandable Figure 1The diagram shown is only one deployment scenario of the method in this application; in other embodiments, the functional modules can be integrated on the same computing device or distributed and deployed on cloud servers and edge terminals; the controlled device 200 can be an Android / iOS mobile terminal, a Windows / macOS desktop system or a Chromium-based web browser; the interface simulation execution environment can be implemented based on Android Debug Bridge, UiAutomator, Appium or Puppeteer; all data interactions between modules are completed through standard interfaces and do not depend on a specific platform.

[0026] The following examples illustrate the training data generation method.

[0027] Figure 2 A schematic flowchart of a training data generation method according to an embodiment of this application is shown. Exemplarily, the training data generation method includes steps S100-S400: Step S100: Construct the operation network diagram.

[0028] The operation network graph refers to a directed graph that represents the states of a graphical user interface (GUI) and its executable operations. Each node represents a different GUI state, and each directed edge represents an executable operation within that state. Each node contains the visual content of the interface corresponding to that state, as well as information on the effective interactive area obtained through manual annotation and interface simulation. Each directed edge contains the operation type, operation parameters, and the verification result of the interface state change after the operation. The operation parameters include spatial positioning information and temporal parameters. Spatial positioning information is used to uniquely determine the location of the operation, and temporal parameters are used to characterize the duration of the operation. For example, the spatial positioning information for a click operation is the screen coordinates (x, y), the spatial positioning information for a swipe operation is the starting coordinates (x1, y1) and the ending coordinates (x2, y2), and the temporal parameter for a long press operation is the duration, etc.

[0029] In this step, when constructing the operation network graph, the starting interface state node can be used as the root node. A breadth-first traversal strategy is used to identify interactive regions of the interface state represented by the current node to obtain candidate interactive regions. Subsequently, the interface simulation execution environment is invoked to perform corresponding operations on each candidate interactive region. Based on whether the interface state changes in accordance with preset criteria after the operation, valid interactive regions are selected. Finally, each valid interactive region, its corresponding operation action, and the interface state after the operation are respectively formed into new directed edges and new nodes, and added to the operation network graph. When a node does not have any verifiable valid interactive regions, the node is terminated as a leaf node.

[0030] Figure 3 A schematic diagram of an operational network diagram is shown, such as... Figure 3 As shown, the operation network diagram includes three nodes: interface A, interface B, and interface C, which respectively represent three different graphical user interface states. Figure 3 There are two directed edges: the first directed edge points from interface A to interface B, indicating that after performing the operation A→B in the state of interface A, the user enters the state of interface B. The operation A→B is a click operation, and its spatial positioning information is coordinates (x1, y1); the second directed edge points from interface A to interface C, indicating that after performing the operation A→C in the state of interface A, the user enters the state of interface C. The operation A→C is a swipe operation, and its spatial positioning information is swiping from coordinates (x2, y2) to coordinates (x3, y3).

[0031] Specifically, such as Figure 4 As shown, the construction of the operation network graph includes steps S110-S130: Step S110: Based on the starting interface state node, identify the interactive area of ​​the interface state represented by the current node to obtain candidate interactive areas.

[0032] In this step, the starting interface state node is the initial entry point for constructing the operation network graph. It represents the general starting state of user tasks in the graphical user interface, such as the Home screen in a mobile operating system, the default desktop in a desktop operating system, or the login page in a web application. The breadth-first traversal is started with the starting interface state node as the root node, and the "current node" processed in the first round is the starting interface state node.

[0033] The annotation device 100 distributes the interface visual content corresponding to the interface state represented by the current node to the annotation personnel's terminal. The annotation personnel annotate all areas of the interface visual content that can be operated on through the graphical annotation interface provided by the annotation device 100. The operable operations include, but are not limited to, clicking, swiping, long pressing, and input. The annotation method for clicking is to select a coordinate point (x, y) on the interface visual content; the annotation method for swiping is to select a starting coordinate point (x1, y1) and an ending coordinate point (x2, y2) on the interface visual content; the annotation method for long pressing is to select a coordinate point (x, y) on the interface visual content and input the duration t; the annotation method for input is to select a text input box area on the interface visual content and input sample text. The annotation device 100 receives the annotation information input by the annotation personnel and generates a structured annotation data. The structured annotation data includes, but is not limited to, operation type, spatial positioning information, time parameters, and sample text. Then, the structured annotation data needs to be associated with the unique identifier of the current node and stored in the database 300.

[0034] The unique identifier is the global index number of the current node in the operation network graph, which can be assigned sequentially in breadth-first traversal order. The global index number of the starting interface state node can be N0. It can be understood that this process in this step can make each candidate interactive area have clear operation semantics, reproducible spatial positioning and verifiable execution parameters, thereby providing complete input for the automated verification of the subsequent interface simulation execution environment.

[0035] Step S120: Perform corresponding operations on each candidate interactive area through the interface simulation execution environment, and take the candidate interactive area whose interface state change after the operation is consistent with the preset state change as the effective interactive area.

[0036] For example, the interface simulation execution environment can be an automated interface operation platform running on a background server, which can be used to send operation instructions to the controlled device 200; the preset state change is a quantifiable difference between the interface state after the operation and the interface state before the operation, and the determination criteria can be at least one of image similarity reduction, interface element structure change, or interface state semantic change.

[0037] Specifically, if image similarity is used as the criterion, the Structural Similarity Index (SSIM) of the visual content of the interface before and after the operation is calculated. If the SSIM value is lower than a preset threshold (e.g., 0.92), the interface state is considered to have undergone a valid change. If the structure of interface elements is used as the criterion, if the type, text content, coordinate position, or visibility attribute of at least one interface element changes after the operation, the interface state is considered to have undergone a valid change. If the semantics of the interface state is used as the criterion, if the task context information carried by the interface state after the operation changes in its relevance to the task objective compared to before the operation, the interface state is considered to have undergone a valid change.

[0038] For any candidate interactive area, if the interface state change caused by its corresponding operation meets any of the above judgment criteria, the candidate interactive area is confirmed as a valid interactive area; otherwise, the candidate interactive area is excluded, and the verification process terminates at the candidate interactive area.

[0039] It is understood that this step, together with step S110, constitutes the first round of verification loop for constructing the operation network graph. Step S110 uses the starting interface state node as the initial processing node and identifies its corresponding candidate interactive regions. Step S120 performs automated verification on each of these candidate interactive regions. The verification result directly determines whether the starting interface state node has the ability to expand outward. That is, the operation network graph can only enter step S130 for graph structure expansion when there is at least one valid interactive region. If all candidate interactive regions fail the verification, the starting interface state node is marked as a leaf node, and the breadth-first traversal process terminates at this node.

[0040] Step S130: Based on each valid interactive area and its corresponding operation action and the interface state after the operation, construct new directed edges and new nodes, and add the new directed edges and new nodes to the operation network graph.

[0041] The new node represents the interface state after the operation, and its components include, but are not limited to, the visual content of the interface corresponding to the interface state and the set of interactive areas initialized to be empty. The new directed edge represents the state transition from the current node to the new node, and its components include, but are not limited to, the operation type, spatial positioning information, time parameters and operation verification result identifier. After the construction is completed, the new node is added to the operation network graph and serves as the node to be processed in the next round of breadth-first traversal. When all nodes in the operation network graph have no valid interactive areas, the breadth-first traversal is completed, and the complete operation network graph is finally obtained.

[0042] Step S200: Establish the instruction-state association between the user's natural language instructions and at least one node in the operation network graph, and based on the instruction-state association, determine the task completion path from the starting interface state node to the interface state node used to complete the user's natural language instructions.

[0043] The task completion path consists of multiple interface state nodes connected in sequence.

[0044] This step uses the operation network diagram constructed in step S130 as input. Its execution process includes two consecutive stages: instruction-state association establishment and task completion path generation. Instruction-state association establishment is as follows: Figure 5 As shown, it specifically includes steps S210 and S220: Step S210: Semantically match the user's natural language command with at least one interface state node in the operation network graph. The system distributes the visual content corresponding to each interface state node in the operation network diagram to the annotator's terminal. The annotator determines whether the interface state can be used to complete a specific user natural language command based on the task context information carried by the interface state. The task context information includes, but is not limited to, the types of controls visible in the interface, text labels, icon semantics, layout structure, and current activation state. The user natural language command is a task objective described in natural language, such as "set mute mode", "login account", "send SMS", etc. The matching process only targets interface state nodes that have the necessary interactive capabilities to complete the task objective. For example, when the user natural language command is "set mute mode", the node is only included in the matching scope if there is an operable mute switch control in the interface state.

[0045] Step S220: Determine the instruction-state association relationship based on the semantic consistency between the task context information carried by the interface state and the task objective represented by the user's natural language instruction.

[0046] Semantic consistency refers to the existence of at least one valid interactive area verified by step S120 in the interface state, and the interface state can enter a state closer to the task goal or directly achieve the task goal after performing the corresponding operation on the area. The instruction-state association relationship supports the association of the same user's natural language instruction to multiple interface state nodes with different task reachability. In addition, the difference in task reachability in this step is reflected in the position attribute of the interface state in the task execution chain. For example, the instruction "login account" can be associated with the login page (requires complete input of credentials), the password retrieval page (requires resetting and redirection to login), and the personal center page in the logged-in state (the task goal has been achieved).

[0047] Furthermore, since leaf nodes represent the endpoint state of a task much more likely than intermediate state nodes, the association establishment in this embodiment can follow a leaf node priority strategy, that is, priority is given to binding instructions to leaf nodes that do not have effective interactive areas in the operation network graph.

[0048] After establishing the instruction-state association, the task completion path is generated, such as... Figure 6 As shown, the task completion path generation includes steps S230-S250: Step S230: In response to the user's natural language command, locate at least one interface state node in the operation network graph that is used to complete the current user's natural language command as the target node.

[0049] This step is performed based on the instruction-state association index table established in step S220 and stored in database 300. The index table uses the user's natural language instruction as the key and the unique identifier of the corresponding interface state node as the value. Each unique identifier of the interface state node is the global index number of the node in the operation network diagram. During the query, the system inputs the user's natural language instruction as the key into the index table and outputs all matching unique identifiers of the interface state nodes. Subsequently, the system extracts the interface visual content and effective interactive area information of the corresponding node from the operation network diagram based on the unique identifier to complete the target node location.

[0050] Step S240: Using the starting interface state node as the root node, perform a graph search in the operation network graph to obtain at least one reachable path from the root node to each target node. In this step, the graph search can use a breadth-first search algorithm. Its traversal process is constrained by the directed edges of the operation network graph, and state transitions are only performed along the directed edges constructed in the effective interactive regions. The obtained reachable path is a directed path that starts from the initial interface state node, passes through a series of interface state nodes, and finally reaches the target node. Each path corresponds to a possible task execution sequence.

[0051] Step S250: Filter reachable paths according to preset filtering rules, and determine at least one path that meets the preset filtering rules as the task completion path.

[0052] The preset filtering rules include, but are not limited to, the following two: The path contains the fewest interface state nodes, that is, the path that requires the fewest interface state nodes from the starting interface state node to the target node is selected; when there are multiple paths with the same number of nodes, all of them are included in the task completion path set.

[0053] The rule minimizes the complexity of the operations contained in the path. This rule can be determined by weighting the complexity of all operations on the path according to their complexity.

[0054] It should be noted that when both filtering rules are enabled at the same time, the rule of minimizing the number of nodes is applied first; if the number of nodes is the same, the rule of minimizing the complexity of operation actions is applied for secondary filtering; this dual filtering mechanism ensures that the determined task completion path achieves the optimal balance between structural simplicity and operational feasibility.

[0055] Step S300: For each pair of adjacent interface state nodes in the task completion path, generate decision-explanatory text that matches the interface state change and corresponding operation action represented by the current node pair.

[0056] It is understandable that in this step, decision-making explanatory text corresponding to the current adjacent interface state node is generated based on the user's natural language instructions, the interface visual content corresponding to the adjacent interface state node, and the operation actions connecting the current node pair.

[0057] Specifically, the decision-explanatory text is generated in response to the combined input of the user's natural language instructions, the visual content of the interface corresponding to the adjacent interface state nodes, and the operation actions connecting the node pairs; the combined input is provided as a whole to the text generation module; and the text generation module generates structured text, which sequentially includes the following four logical steps: Task intent understanding: parsing the final task objective represented by the user's natural language instructions.

[0058] Current interface state recognition: describes the visual content of the interface represented by the previous node among adjacent interface state nodes and the task context information it carries.

[0059] Necessity-based causal reasoning: Based on the task intent and the current interface state, deduce the necessity and rationality of performing the operation action connecting the node pair for achieving the task objective.

[0060] Visual representation of operation actions: Transform the operation actions connecting the node pair into executable atomic-level operation instructions, including operation type, spatial positioning information, and time parameters.

[0061] It can be understood that this structured text, as the decision-explanatory text corresponding to the adjacent interface state node pair, can be used to construct the decision-explanatory text sequence in subsequent trajectory training samples.

[0062] In step S400, the interface state, operation action, and decision explanation text are matched one by one according to the order of adjacent interface state nodes in the task completion path to form a trajectory training sample.

[0063] Here, one-to-one correspondence means that for the i-th pair of adjacent interface state nodes in the task completion path, the interface state represented by the interface state node located ahead of the path in the node pair is taken as the i-th interface state, the operation action connecting the node pair is taken as the i-th operation action, and the corresponding decision explanation text generated in step S300 is taken as the i-th decision explanation text; all i-th elements are combined in ascending order of i to form a trajectory training sample.

[0064] It is understood that for the first pair of adjacent interface state nodes in the task completion path, the interface state represented by the interface state node located ahead of the path in the node pair is taken as the first interface state, the operation action connecting the node pair is taken as the first operation action, and the corresponding decision explanation text generated in step S300 is taken as the first decision explanation text. For each subsequent pair of adjacent interface state nodes, the corresponding interface state, operation action, and decision explanation text are extracted according to the same rules. All extraction results are arranged in the order of appearance of node pairs in the task completion path to form a set of ordered time step records. This set of time step records constitutes a complete trajectory training sample. It is understood that the trajectory training sample is organized into a structured data format, containing three core components: interface state, operation action, and decision explanation text, and is used for supervised learning of the graphical user interface agent.

[0065] In some implementations, after step S400, such as Figure 7 As shown, the method further includes steps S500-S600: Step S500: Evaluate the effectiveness of all generated trajectory training samples, obtain the completion rate of each trajectory training sample, and designate trajectory training samples with a completion rate lower than a preset threshold as difficult sample samples.

[0066] The completion rate can be defined as the probability that a graphical user interface agent can successfully complete a user's natural language instruction based on the training samples of the trajectory under the same initial conditions. The calculation method can be to independently execute multiple inferences in the test environment, count the number of successes and take the percentage (e.g., independently execute 100 inferences in the test environment, count the number of successes and take the percentage).

[0067] The preset threshold can be set based on empirical task completion benchmarks, such as a value range of 0.5-0.8. If the completion rate is lower than this value, it indicates that the trajectory poses a significant learning obstacle to model training.

[0068] Step S600: Extract at least one sub-path from the task completion path corresponding to the difficult example sample, and re-execute the step of generating decision explanatory text as the new task completion path to obtain the corresponding sub-trajectory training sample.

[0069] The sub-path consists of multiple consecutive interface state nodes in the task completion path. Multiple consecutive interface state nodes refer to a group of nodes with consecutive index numbers in the task completion path, and intermediate nodes are not allowed to be skipped. It can be understood that its sub-path is usually a path composed of several consecutive nodes in the task completion path. For example, if the task completion path contains 5 interface state nodes, namely A, B, C, E, F (N=5), then its sub-path can be B→C→E→F, or it can be C→E→F.

[0070] For example, if we start from the second interface state node and extend to the interface state node at index N (i.e., the last interface state node), we form a sub-path; where N is the total number of interface state nodes included in the task completion path; that is, this sub-path includes all interface state nodes with index positions from 2 to N, for a total of N. One interface state node; the sub-path reuses the original user natural language command as input and does not generate new user natural language commands; the directed edges, operation actions and interface states after the operation corresponding to the sub-path all adopt the structure and parameters of the corresponding positions in the original task completion path.

[0071] If the task completion path contains 5 interface state nodes, namely A, B, C, E, and F (N=5), then the interface state node with index 2 is B, and the interface state node with index N is F. The sub-path extracted according to the above rules is B→C→E→F, which contains a total of 4 interface state nodes (N...). (1=4). It can be understood that this processing method in this embodiment, while maintaining the semantic consistency of the user's natural language instructions, enables the graphical user interface agent to preferentially learn the ability to transition from intermediate states to target states during training. By reducing one state jump and the number of operation steps covered by a single sample, it improves the modeling adaptability to long-path tasks.

[0072] In some implementations, the method further includes: responding to any trajectory training sample, stylizing and rewriting the user natural language instructions and / or decision-explanatory text in the trajectory training sample based on a preset amplification strategy, and generating at least one stylized trajectory training sample based on the stylized rewritten text.

[0073] Among them, stylized rewriting preserves the original semantics, and the generated stylized texts are different from each other in terms of expression dimensions, including speaker attribute dimensions and / or task scenario dimensions. It is understood that the stylized rewriting in this embodiment can be performed based on a preset augmentation strategy, and the generated text will reflect the task objective, operation object, and execution intent. Figure Three The text is completely consistent with the original text in all dimensions. The preset amplification strategy includes transforming the expression dimensions of user natural language instructions and / or decision-making explanatory text. The expression dimensions include speaker attribute dimensions and / or task scenario dimensions. Among them, speaker attribute dimensions include, but are not limited to, gender, age, educational background and occupational characteristics; task scenario dimensions include, but are not limited to, formality, urgency and proficiency. Each text obtained after stylization and rewriting corresponds one-to-one with the interface state and operation actions in the original trajectory training sample, together forming a new stylized trajectory training sample.

[0074] In some implementations, the trajectory training samples generated by this method can be used in conjunction with the reinforcement learning training process. The training samples used in reinforcement learning include two types: one type consists of positive samples formed by task completion paths that satisfy the rule of minimizing the number of interface state nodes in the path; the other type consists of comparative samples retrieved from the operation network graph. These comparative samples correspond to other reachable paths under the same user natural language command that do not satisfy the aforementioned rule of minimizing the number of nodes, and their number of interface state nodes is strictly greater than the minimum value. These comparative samples are used to construct a path selection preference signal, assisting the model in tending to output operation sequences with fewer nodes during inference. All comparative samples are obtained by traversing the operation network graph and comparing path lengths, requiring no additional manual annotation.

[0075] This embodiment significantly reduces the cost of acquiring training data for graphical user interface (GUI) agents by constructing a structured operation network graph and automatically generating trajectory training samples based on it. Compared to the traditional method that relies on manual step-by-step operation and annotation, this method only requires one-time annotation and verification of the initial interface and key nodes. It can then derive a large number of diverse trajectories through graph traversal and path combination, thereby reducing repetitive human input while ensuring broad coverage. Simultaneously, the explicit modeling of instruction-state associations and task completion paths gives the generated trajectories a clear task logic structure. The node-by-node generation of decision-explanatory text and sub-path optimization of difficult examples further improve the quality of training samples in state transition modeling and long-path generalization. The stylized rewriting mechanism enhances the adaptability of samples to different user expression habits and task scenarios. Overall, this method, without increasing the annotation burden, balances data scale, structural rationality, and semantic diversity, providing a technical foundation for efficient training of GUI agents.

[0076] The training data generation apparatus provided in the embodiments of this application will be further described below, such as... Figure 1 As shown, the training data generation device includes: Operation network graph construction module 110 is used to construct an operation network graph; wherein, the operation network graph is a directed graph, the nodes of the directed graph represent different graphical user interface states, and the directed edges of the directed graph represent the operation actions that can be performed in the interface state represented by the corresponding node. The task completion path generation module 120 is used to establish the instruction-state association between the user's natural language instructions and at least one node in the operation network graph, and based on the instruction-state association, determine the task completion path from the starting interface state node to the interface state node used to complete the user's natural language instructions. The decision explanation text generation module 130 is used to generate decision explanation text that matches the interface state change and corresponding operation action represented by the current node pair for each pair of adjacent interface state nodes in the task completion path.

[0077] The trajectory training sample generation module 140 is used to correspond the interface state, operation action and decision explanation text one by one according to the order of adjacent interface state nodes in the task completion path to form a trajectory training sample.

[0078] In some implementations, the operation network graph construction module 110 is specifically used to: identify interactive regions based on the interface state represented by the current node according to the starting interface state node, and obtain candidate interactive regions; perform corresponding operations on each candidate interactive region through the interface simulation execution environment, and take the candidate interactive regions whose interface state changes after the operation are consistent with the preset state changes as valid interactive regions; construct new directed edges and new nodes based on each valid interactive region and the corresponding operation action and the interface state after the operation, and add the new directed edges and new nodes to the operation network graph.

[0079] The task completion path generation module 120 is specifically used to: semantically match the user's natural language command with at least one interface state node in the operation network graph; and determine the command-state association relationship based on the semantic consistency between the task context information carried by the interface state and the task objective represented by the user's natural language command; wherein, the command-state association relationship supports the same user natural language command being associated with multiple interface state nodes with different task reachability.

[0080] The task completion path generation module 120 is further configured to: respond to a user's natural language command, locate at least one interface state node in the operation network graph that is used to complete the current user's natural language command as a target node; perform a graph search in the operation network graph with the starting interface state node as the root node to obtain at least one reachable path from the root node to each target node; filter the reachable paths according to a preset filtering rule, and determine at least one path that satisfies the preset filtering rule as the task completion path; wherein, the preset filtering rule includes minimizing the number of interface state nodes contained in the path.

[0081] In some implementations, the decision explanatory text generation module 130 is specifically used to: generate decision explanatory text corresponding to the current adjacent interface state node based on the user's natural language instructions, the interface visual content corresponding to the adjacent interface state node, and the operation actions connecting the current node pair.

[0082] The trajectory training sample generation module 140 is used to respond to any trajectory training sample, and based on a preset amplification strategy, to stylize and rewrite the user's natural language instructions and / or decision-making explanatory text in the trajectory training sample, and to generate at least one stylized trajectory training sample based on the stylized rewritten text; wherein, the stylized rewriting maintains the original semantics unchanged, and the generated stylized texts are different from each other in terms of expression dimensions, including speaker attribute dimensions and / or task scenario dimensions.

[0083] The difficult example sample optimization module 150 is used to evaluate the effect of all generated trajectory training samples, obtain the completion rate of each trajectory training sample, and designate trajectory training samples with completion rates lower than a preset threshold as difficult example samples; extract at least one sub-path from the task completion path corresponding to the difficult example sample, and re-execute the step of generating decision explanatory text as a new task completion path to obtain the corresponding sub-trajectory training sample; wherein, the sub-path consists of multiple consecutive interface state nodes in the task completion path.

[0084] It is understood that the apparatus in this embodiment corresponds to the training data generation method in the above embodiments, and the options in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0085] This application also provides a terminal device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the above-described training data generation method or training data generation apparatus.

[0086] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0087] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0088] This application also provides a computer-readable storage medium for storing the computer program used in the aforementioned terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0090] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0091] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0092] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for generating training data, characterized in that, include: Construct an operation network graph; wherein the operation network graph is a directed graph, the nodes of the directed graph represent different graphical user interface states, and the directed edges of the directed graph represent the operation actions that can be performed in the interface state represented by the corresponding node. Establish an instruction-state association between the user's natural language instruction and at least one node in the operation network graph, and based on the instruction-state association, determine the task completion path from the starting interface state node to the interface state node used to complete the user's natural language instruction. For each pair of adjacent interface state nodes in the task completion path, generate decision-explanatory text that matches the interface state change and corresponding operation action represented by the current node pair. The interface state, the operation action, and the decision explanation text are matched one-to-one according to the order of the adjacent interface state nodes in the task completion path to form a trajectory training sample.

2. The training data generation method according to claim 1, characterized in that, The construction of the operation network graph includes: Based on the initial interface state node, the interactive area is identified for the interface state represented by the current node to obtain candidate interactive areas. The interface simulates the execution environment to perform corresponding operations on each of the candidate interactive areas, and the candidate interactive areas whose interface state changes after the operation are consistent with the preset state changes are taken as valid interactive areas. Based on each effective interactive area and its corresponding operation action and post-operation interface state, new directed edges and new nodes are constructed, and the new directed edges and new nodes are added to the operation network graph.

3. The training data generation method according to claim 1, characterized in that, The establishment of the instruction-state association between the user's natural language instructions and at least one node in the operation network graph includes: Semantically match the user's natural language instructions with at least one interface state node in the operation network graph; Based on the semantic consistency between the task context information carried by the interface state and the task objective represented by the user's natural language instruction, the instruction-state association is determined; wherein, the instruction-state association supports the association of the same user natural language instruction to multiple interface state nodes with different task reachability.

4. The training data generation method according to claim 1, characterized in that, The step of determining the task completion path from the starting interface state node to the interface state node used to complete the user's natural language command based on the instruction-state association includes: In response to the user's natural language instruction, at least one interface state node for completing the current user's natural language instruction is located as a target node in the operation network graph. Using the starting interface state node as the root node, perform a graph search in the operation network graph to obtain at least one reachable path from the root node to each of the target nodes; The reachable paths are filtered according to preset filtering rules, and at least one path that satisfies the preset filtering rules is determined as the task completion path; wherein, the preset filtering rules include minimizing the number of interface state nodes contained in the path.

5. The training data generation method according to claim 1, characterized in that, For each pair of adjacent interface state nodes in the task completion path, generating decision-explanatory text that matches the interface state change and corresponding operation action represented by that node includes: Based on the user's natural language instructions, the visual content of the interface corresponding to the adjacent interface state nodes, and the operation actions connecting the current node pair, the decision-explanatory text corresponding to the current adjacent interface state node is generated.

6. The training data generation method according to claim 1, characterized in that, Also includes: The effectiveness of all generated trajectory training samples is evaluated to obtain the completion rate of each trajectory training sample, and the trajectory training samples with a completion rate lower than a preset threshold are regarded as difficult sample samples. Extract at least one sub-path from the task completion path corresponding to the difficult example sample, and use the sub-path as a new task completion path to re-execute the step of generating decision explanatory text to obtain the corresponding sub-trajectory training sample; wherein, the sub-path is composed of multiple consecutive interface state nodes in the task completion path.

7. The training data generation method according to claim 1, characterized in that, Also includes: In response to any of the trajectory training samples, based on a preset amplification strategy, the user natural language instructions and / or the decision explanatory text in the trajectory training samples are stylized and rewritten; wherein, the stylization and rewriting maintains the original semantics unchanged, and the generated stylized texts are different from each other in terms of expression dimensions, including speaker attribute dimensions and / or task scenario dimensions. Based on the text obtained after stylization rewriting, at least one stylized trajectory training sample is generated.

8. A training data generation device, characterized in that, include: An operation network graph construction module is used to construct an operation network graph; wherein, the operation network graph is a directed graph, the nodes of the directed graph represent different graphical user interface states, and the directed edges of the directed graph represent the operation actions that can be performed in the interface state represented by the corresponding node. The task completion path generation module is used to establish an instruction-state association relationship between the user's natural language instruction and at least one node in the operation network graph, and based on the instruction-state association relationship, determine the task completion path from the starting interface state node to the interface state node used to complete the user's natural language instruction. The decision explanation text generation module is used to generate decision explanation text that matches the interface state change and corresponding operation action represented by the current node for each pair of adjacent interface state nodes in the task completion path. The trajectory training sample generation module is used to correspond the interface state, the operation action, and the decision explanation text one by one according to the order of the adjacent interface state nodes in the task completion path to form a trajectory training sample.

9. A terminal device, characterized in that, The terminal device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the training data generation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements the training data generation method according to any one of claims 1-7.

Images