Training data generation method and device, electronic equipment and storage medium

Abstract

The invention discloses a training data generation method and device, electronic equipment and a storage medium, and relates to the field of computers, in particular to the technical field of artificial intelligence such as large models, deep learning and agents. According to the specific implementation scheme, firstly, interaction data with an application is obtained, a state transition diagram is determined according to the interaction data, then multiple pieces of trajectory data are generated based on the state transition diagram, then the trajectory data are input into a multi-modal model, and reasoning reference information corresponding to operation data in the trajectory data output by the multi-modal model is obtained; and finally, based on the trajectory data and the reasoning reference information corresponding to the operation data in the trajectory data, generating training data for training the interactive agent model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as large models, deep learning, and intelligent agents, specifically to training data generation methods, devices, electronic devices, and storage media. Background Technology

[0002] Currently, there are generally two methods to obtain data for training surrogate models: one is through manual annotation, and the other is through a tool generated by a large model based on manually set trajectories. However, both methods rely on manual labor, are costly, and inefficient. Summary of the Invention

[0003] This disclosure provides a training data generation method, apparatus, electronic device, and storage medium. The specific solution is as follows:

[0004] According to one aspect of this disclosure, a method for generating training data is provided, comprising:

[0005] Acquire interaction data with the application, wherein the interaction data includes multiple operation data, a user interface diagram before the operation corresponding to each operation data, and a user interface diagram after the operation;

[0006] Based on the interaction data, a state transition diagram is determined, wherein the nodes in the state transition diagram are user interface diagrams, and the connecting edges between every two user interface diagrams are operation data associated with the two user interface diagrams;

[0007] Based on the state transition diagram, multiple trajectory data are generated, wherein each trajectory data is used to indicate the trajectory diagram and operation data sequence corresponding to a task;

[0008] The trajectory data is input into a multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model, wherein the inference reference information is used to describe the inference process of the operational data;

[0009] Based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data, training data for training the interactive agent model is generated.

[0010] According to another aspect of this disclosure, a training data generation apparatus is provided, comprising:

[0011] The acquisition module is used to acquire interaction data with the application, wherein the interaction data includes multiple operation data, a user interface diagram before the operation corresponding to each operation data, and a user interface diagram after the operation.

[0012] The determination module is used to determine a state transition diagram based on the interaction data, wherein the nodes in the state transition diagram are user interface diagrams, and the connecting edges between every two user interface diagrams are operation data associated with the two user interface diagrams;

[0013] The first generation module is used to generate multiple trajectory data based on the state transition diagram, wherein each trajectory data is used to indicate a trajectory diagram and operation data sequence corresponding to a task;

[0014] An input module is used to input the trajectory data into a multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model. The inference reference information is used to describe the inference process of the operational data.

[0015] The second generation module is used to generate training data for training the interactive agent model based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in the above embodiments.

[0020] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in the above embodiments.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the above embodiments.

[0022] The training data generation method, apparatus, electronic device, and storage medium disclosed herein offer the following advantages: First, interaction data with the application is acquired, and a state transition diagram is determined based on the interaction data. Then, multiple trajectory data are generated based on the state transition diagram. Next, the trajectory data is input into a multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model. Finally, training data for training the interactive agent model is generated based on the trajectory data and the inference reference information corresponding to the operational data within the trajectory data. Thus, by parsing the collected interaction data between the interactive agent model and the application, trajectory data corresponding to different tasks is generated, thereby achieving automatic trajectory data generation. This provides a basis for reducing the cost and time consumption of training data generation. Furthermore, the multimodal model is used to generate inference reference information for the operational data within the trajectory data. Based on the trajectory data and the inference reference information for the operational data within it, training data is automatically generated, improving the efficiency of training data generation.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0024] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0025] Figure 1 A flowchart illustrating a training data generation method provided in an embodiment of this disclosure;

[0026] Figure 2 A flowchart illustrating a training data generation method provided in another embodiment of this disclosure;

[0027] Figure 3 A schematic flowchart illustrating the generation of a trajectory map in a training data generation method provided in another embodiment of this disclosure;

[0028] Figure 4 This is an example diagram of the trajectory map generated in the training data generation method proposed in this embodiment of the disclosure;

[0029] Figure 5 A flowchart illustrating a training data generation method provided in another embodiment of this disclosure;

[0030] Figure 6 A flowchart illustrating a training data generation method provided in another embodiment of this disclosure;

[0031] Figure 7 This is a schematic diagram of the structure of a training data generation apparatus provided in an embodiment of the present disclosure;

[0032] Figure 8 This is a block diagram of an electronic device used to implement the training data generation method of the embodiments of this disclosure. Detailed Implementation

[0033] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] This disclosure relates to the fields of artificial intelligence technologies such as large models, deep learning, and intelligent agents.

[0035] Artificial Intelligence (AI) is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence.

[0036] Large models, also known as Foundation Models, are models that extract knowledge from hundreds of millions of corpora or images, learn, and then produce large models with hundreds of millions of parameters.

[0037] Deep learning (DL) learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound.

[0038] An intelligent agent, also known as a smart agent, is an intelligent entity that autonomously perceives its environment, makes decisions based on goals, and interacts with the environment to achieve specific functions. Intelligent agents perceive changes in the environment (e.g., through sensors or data input), make judgments and decisions based on their learned knowledge and algorithms, and then execute actions to influence the environment or achieve predetermined goals.

[0039] It should be noted that intelligent agents can include mobile agents and graphical user interface (GUI) agents.

[0040] The Mobile Agent is an intelligent agent based on a multimodal large language model. By recognizing and locating visual and textual information in the application interface, it can efficiently plan and execute complex tasks in mobile applications, supporting cross-application operations and purely visual solutions. That is, it does not rely on the system's User Interface (UI) files, but understands and operates mobile devices by analyzing images.

[0041] A graphical user interface (GUI) agent is an intelligent agent driven by a multimodal visual model that can automatically infer and execute UI interactions, simulating human user actions such as clicking, inputting, dragging, and reading interface information to complete tasks required by humans.

[0042] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0043] The following description, with reference to the accompanying drawings, outlines a training data generation method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure.

[0044] Figure 1 This is a flowchart illustrating a training data generation method provided in an embodiment of the present disclosure.

[0045] like Figure 1 As shown, the training data generation method includes:

[0046] Step 101: Obtain interaction data with the application.

[0047] The interactive data includes multiple operation data, a user interface diagram before the operation for each operation data, and a user interface diagram after the operation.

[0048] Interaction data can be the interaction data between the interaction agent model and the application.

[0049] The interactive agent model refers to an intelligent agent possessing perception, decision-making, and interaction capabilities. It can automatically complete human-computer interaction tasks through image perception, semantic understanding, and reasoning abilities. Its specific type and structure can be configured as needed. For example, it can be a reinforcement learning agent model, a large language model-based agent model, a multimodal agent model, etc., and this disclosure does not limit its scope.

[0050] The interactive agent model can be applied to scenarios such as automated operation of smart terminals, accessibility assistance, intelligent driving, and smart homes, and this disclosure does not limit it.

[0051] The applications that interact with the interaction agent model can be configured as needed. For example, they can be commonly used applications in daily life, such as e-commerce applications, meeting applications, communication applications, calendar applications, alarm clock applications, etc. This disclosure does not limit them.

[0052] The operation data can be the data generated during interaction with the application. For example, it can be operations such as clicking a button or link, sliding a slider, or entering text, etc. This disclosure does not limit the scope of the data.

[0053] In some possible implementations, when acquiring interaction data with an application, one can first obtain any user interface diagram of the application, then identify the set of operable elements within that diagram, and finally perform operations on each operable element to obtain the corresponding operation data and the resulting user interface diagram. By performing operations on each operable element in the user interface diagram, all executable operations within the current diagram can be fully explored, thus constructing diverse and comprehensive interaction data. Therefore, when constructing training data for an interaction agent model based on this interaction data, it is possible to incorporate more interaction scenarios and task objectives into the training of the interaction agent model. This allows the trained interaction agent model to accurately respond to user commands and automatically complete tasks in more scenarios during practical applications, improving the efficiency and experience of human-computer interaction and enhancing user convenience.

[0054] The set of operable elements can include all operable controls in the user interface diagram. For example, it can include any controls in the user interface diagram such as buttons, links, cards, text input boxes, selection boxes, menus, and sliders, etc., and this disclosure does not limit this.

[0055] Step 102: Determine the state transition diagram based on the interaction data.

[0056] In the state transition diagram, the nodes are user interface diagrams, and the connecting edges between any two user interface diagrams are the operation data associated with those two user interface diagrams.

[0057] In this disclosure, after obtaining the interaction data with the application, the user interface graph in the interaction data can be used as a node, and the operation data associated with the two user interface graphs can be used as a connection edge to construct a state transition graph, thereby providing a data foundation for the subsequent efficient generation of high-quality training data.

[0058] Step 103: Generate multiple trajectory data based on the state transition diagram.

[0059] Each trajectory data point is used to indicate the trajectory diagram and operation data sequence corresponding to a task.

[0060] In this context, "task" can refer to the tasks supported by the application and their associated tasks. For example, when the application is an e-commerce application, the tasks could be product search, order inquiry, logistics, etc., and this disclosure does not impose any limitations on this.

[0061] The trajectory diagram can include a user interface diagram showing all the steps to complete the task.

[0062] The operation data sequence is a sequence composed of each operation data that completes the task, arranged in the order of execution.

[0063] In this disclosure, multiple trajectory data are generated based on a state trajectory graph, thereby enabling the rapid and efficient generation of trajectory data based on a graph structure.

[0064] In some possible implementations, after generating multiple trajectory data sets, the similarity between each trajectory data set can be determined based on the trajectory graphs within those sets. If the similarity between any two trajectory data sets exceeds a similarity threshold, they can be considered highly similar, meaning they are duplicate trajectory data. In this case, one of the duplicate trajectory data sets can be deleted from the multiple trajectory data sets. This removes duplicate trajectories, reduces redundant data, and improves the quality of the trajectory data.

[0065] The similarity threshold is a critical value used to determine whether any two trajectory data are duplicate trajectory data. It can be preset as needed, and this disclosure does not limit it.

[0066] Step 104: Input the trajectory data into the multimodal model to obtain the inference reference information corresponding to the operational data in the trajectory data output by the multimodal model.

[0067] The specific type and architecture of the multimodal model can be preset as needed. For example, the multimodal model can be a two-stream architecture model, an encoder-decoder architecture model, a generative multimodal model, etc., and this disclosure does not limit it.

[0068] The reasoning reference information can be textual information, which may include analysis of the user interface diagram before the operation data is executed, the thought process before selecting the operation data, the instructions for selecting the operation data, and so on. For example, the reasoning reference information may be a reasoning thought chain, which is not limited in this disclosure.

[0069] In this disclosure, by inputting trajectory data into a multimodal model, the multimodal model can generate user interface diagram analysis information before the execution of the operation, thinking information before the execution of the operation, instruction information for the execution of the operation, etc., based on the changes in the user interface diagram before and after the execution of each operation data. Then, this information is integrated into the reasoning reference information corresponding to the operation data.

[0070] In some possible implementations, when generating inference reference information corresponding to the operation data, the task target corresponding to the trajectory data can also be generated based on the trajectory diagram and operation data sequence in the trajectory data. This disclosure does not limit this.

[0071] Step 105: Based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data, generate training data for training the interactive agent model.

[0072] In this disclosure, when generating training data for training the interactive agent model based on trajectory data and inference reference information corresponding to the operational data in the trajectory data, the format of the training data can be: user interface diagram → inference reference information corresponding to the operational data + operational data → user interface diagram. For example, the format of the training data can be user interface diagram #1 → inference reference information corresponding to operational data #1 + operational data #1 → user interface diagram #2 → inference reference information corresponding to operational data #2 + operational data #2 → user interface diagram #3 → inference reference information corresponding to operational data #3 + operational data #3 → user interface diagram #4... This disclosure does not limit this.

[0073] The training data obtained through the training data generation method proposed in this disclosure is generated based on the interaction data of a fully mined interaction agent model and the application. This means the interaction data includes a large number of interaction scenarios and tasks. Therefore, the interaction agent model trained based on this training data can be applied to more interaction scenarios, improving the diversity and comprehensiveness of the human-computer interaction scenarios of the interaction agent model and enhancing the user experience.

[0074] In this embodiment, interaction data with the application is first acquired, and a state transition diagram is determined based on the interaction data. Then, multiple trajectory data are generated based on the state transition diagram. The trajectory data is then input into a multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model. Finally, training data for training the interaction agent model is generated based on the trajectory data and the inference reference information corresponding to the operational data within the trajectory data. Thus, by parsing the collected interaction data between the interaction agent model and the application, trajectory data corresponding to different tasks is generated, thereby achieving automatic generation of trajectory data. This provides a basis for reducing the cost and time consumption of training data generation. Furthermore, the multimodal model is used to generate inference reference information for the operational data within the trajectory data. Based on the trajectory data and the inference reference information for the operational data within it, training data is automatically generated, improving the efficiency of training data generation.

[0075] Figure 2 This is a schematic flowchart of a training data generation method provided in another embodiment of the present disclosure.

[0076] like Figure 2 As shown, the training data generation method includes:

[0077] Step 201: Obtain interaction data with the application.

[0078] The interactive data includes multiple operation data, a user interface diagram before the operation for each operation data, and a user interface diagram after the operation.

[0079] Step 202: Determine the state transition diagram based on the interaction data.

[0080] In the state transition diagram, the nodes are user interface diagrams, and the connecting edges between any two user interface diagrams are the operation data associated with those two user interface diagrams.

[0081] The specific implementation of steps 201 to 205 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0082] Step 203: If the state transition graph includes cyclic paths, determine the target task corresponding to the cyclic path.

[0083] In the state transition diagram, a cyclic path is one where the first and last nodes are the same. In other words, the starting and ending user interface diagrams of a cyclic path are identical.

[0084] The target task can be determined based on the user interface diagram and operation data in the loop path. For example, if the loop path is "System Desktop → Click Calendar Application Icon → Calendar Application Homepage → Click Create Item Button → Item Creation Page → Click Complete Button → Item Creation Complete Page → Click Back Button → System Desktop", then the target task of this loop path can be determined to be creating a calendar item. This disclosure does not limit this.

[0085] In this disclosure, after determining the state transition diagram, since the state transition diagram is generated based on all interaction data, and the interaction data includes multiple operation data, as well as the user interface diagram before and after each operation data, there may be a terminating user interface diagram in a certain path that serves as the starting user interface diagram for that path. That is, the state transition diagram may contain cyclic paths. In this case, the target task corresponding to the cyclic path can be determined based on the user interface diagrams and operation data included in the cyclic path.

[0086] Step 204: Based on the target task, determine the starting user interface diagram and the ending user interface diagram within the loop path.

[0087] In this disclosure, after determining the target task of the cyclic path, the starting and ending user interface diagrams within the cyclic path can be determined based on the target task. For example, taking the above example of a cyclic path, if the target task is to create a calendar event, the first operation should be opening the calendar application from the system desktop, and the last operation should be completing the event creation. In this case, the starting user interface diagram of the cyclic path would be the interface before clicking the calendar application icon, i.e., the system desktop, and the ending user interface diagram would be the interface after clicking the "Done" button, i.e., the event creation completion page. This disclosure does not limit this aspect.

[0088] Step 205: Disconnect the edges between the starting user interface diagram and the ending user interface diagram to obtain the acyclic state transition diagram corresponding to the target task.

[0089] In some possible implementations, this disclosure can use a pre-set depth-finite search algorithm with cyclic pruning to determine whether there is a cyclic path in the state transition graph by judging whether the first and last nodes of the path are repeated, and prune the cyclic path. This disclosure does not limit this.

[0090] In this disclosure, after determining the starting user interface diagram and the ending user interface diagram in a cyclic path, the acyclic state transition diagram corresponding to the target task is obtained by disconnecting the connecting edges between the starting user interface diagram and the ending user interface diagram. For example, taking the above example of a cyclic path, the starting user interface diagram is the system desktop, and the ending user interface diagram is the task creation completion page. Therefore, the connecting edge between the task creation completion page and the system desktop interface can be disconnected, that is, the "click the back button" operation data can be removed. This disclosure does not limit this.

[0091] By pruning cyclic paths in the state transition graph, acyclic state transition graphs are obtained, reducing redundant data and improving the quality of the state transition graphs. This provides a data foundation for obtaining high-quality training data. Furthermore, the interaction agent model can be trained based on this high-quality training data, improving the performance of the trained interaction agent model. This allows the trained interaction agent model to accurately respond to user commands and complete tasks, thus enhancing the user experience.

[0092] Step 206: Generate multiple trajectory data based on the state transition diagram.

[0093] Each trajectory data point is used to indicate the trajectory diagram and operation data sequence corresponding to a task.

[0094] In this disclosure, after pruning the looped paths in the state transition graph to obtain an acyclic state transition graph, multiple trajectory data can be generated based on the acyclic state transition graphs corresponding to all tasks in the state transition graph.

[0095] Step 207: Input the trajectory data into the multimodal model to obtain the inference reference information corresponding to the operational data in the trajectory data output by the multimodal model.

[0096] Step 208: Based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data, generate training data for training the interactive agent model.

[0097] The specific implementation of steps 206 to 208 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0098] In this embodiment, interaction data with the application is first acquired, and a state transition graph is determined based on the interaction data. Then, if the state transition graph includes cyclic paths, the target task corresponding to the cyclic path is determined, and the starting user interface graph and the ending user interface graph within the cyclic path are determined based on the target task. The connection edges between the starting and ending user interface graphs are then disconnected to obtain an acyclic state transition graph corresponding to the target task. Multiple trajectory data are generated based on the state transition graph. Finally, the trajectory data is input into a multimodal model to obtain inference reference information corresponding to the operation data in the trajectory data output by the multimodal model. Based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data, training data for training the interaction agent model is generated. Thus, after generating the state transition graph based on the interaction data between the interaction agent model and the application, the cyclic paths in the state transition graph are pruned to obtain an acyclic state transition graph, thereby reducing redundant data in the state transition graph, improving the quality of the state transition graph, and providing a data foundation for subsequently generating high-quality training data. Next, the state transition diagram is used to generate trajectory data for different tasks, reducing noise in the trajectory data. Then, a multimodal model is used to generate inference reference information for the operational data in the trajectory data. Based on the trajectory data and the inference reference information for the operational data, training data is automatically generated, thereby improving the quality and efficiency of the generated training data.

[0099] Figure 3 This is a schematic flowchart illustrating the process of generating a trajectory map in a training data generation method provided in another embodiment of this disclosure.

[0100] like Figure 3 As shown, the process of generating the trajectory map in this training data generation method includes:

[0101] Step 301: Determine the operation object corresponding to the operation data based on the operation data corresponding to each connection edge in the state transition diagram.

[0102] The operation object is the corresponding operable element in the user interface diagram, i.e., the operable control.

[0103] In this disclosure, after acquiring the interaction data with the application and determining the state transition diagram based on the interaction data, when generating the trajectory diagram in the trajectory data based on the state transition diagram, the operation object corresponding to the operation data can first be determined according to the operation data corresponding to each connecting edge in the state transition diagram. For example, taking the above example, if the operation data is "click the calendar application icon", then the operation object corresponding to this operation data is the calendar application icon on the system desktop; if the operation data is "click the create item button", then the corresponding operation object is the create item button on the calendar application home page. This disclosure does not limit this.

[0104] Step 302: Label the operation objects in the user interface diagram before the operation corresponding to the operation data to obtain the labeled user interface diagram.

[0105] In this disclosure, after obtaining the operation object in the user interface before determining the operation corresponding to the operation data, it can be labeled using a preset method to obtain a labeled user interface diagram, thereby enabling the subsequent multimodal model to quickly identify the operation object. For example, the operation object can be labeled using a solid line bounding box, etc., and this disclosure does not limit it in this way.

[0106] Step 303: Following the order of operations from front to back, stitch together all the labeled user interface diagrams corresponding to each task to obtain the trajectory diagram corresponding to the task.

[0107] In this disclosure, after determining the labeled user interface diagram for each task, all labeled user interface diagrams corresponding to each task can be sequentially stitched together according to the operation sequence from front to back to obtain the trajectory diagram corresponding to the task. For example, the generated trajectory diagram can be as follows: Figure 4 As shown, Figure 4 This is an example diagram of the trajectory map generated in the training data generation method proposed in the embodiments of this disclosure. Figure 4 Taking the creation of a calendar event as an example, and using solid-line bounding boxes to annotate the operation object, this task includes four annotated user interface images. Following the order of operations from front to back, the system desktop image before clicking the calendar application icon, the calendar application homepage before clicking the "Create Event" button, the event creation page before clicking the "Done" button, and the final user interface image (the event creation completion page) are sequentially stitched together to obtain the corresponding trajectory image. Figure 3 Each interface image in this document is for illustrative purposes only and is not intended to be restrictive.

[0108] In this embodiment, firstly, based on the operation data corresponding to each connection edge in the state transition graph, the operation object corresponding to the operation data is determined. Then, the operation object in the user interface graph before the operation corresponding to the operation data is labeled to obtain a labeled user interface graph. Finally, in order of operation sequence from front to back, all labeled user interface graphs corresponding to each task are sequentially stitched together to obtain the trajectory graph corresponding to the task. Thus, by labeling the operation objects of the operation data in the user interface graph of each task and stitching the labeled user interface graphs together in order of operation, the trajectory graph corresponding to the task is obtained. This allows the multimodal model to quickly locate the operation object, providing a foundation for the subsequent efficient and accurate generation of inference reference information for the operation data by the multimodal model, thereby improving the efficiency of training data generation.

[0109] Figure 5This is a schematic flowchart of a training data generation method provided in another embodiment of the present disclosure.

[0110] like Figure 5 As shown, the training data generation method includes:

[0111] Step 501: Obtain interaction data with the application.

[0112] The interactive data includes multiple operation data, a user interface diagram before the operation for each operation data, and a user interface diagram after the operation.

[0113] Step 502: Determine the state transition diagram based on the interaction data.

[0114] In the state transition diagram, the nodes are user interface diagrams, and the connecting edges between any two user interface diagrams are the operation data associated with those two user interface diagrams.

[0115] The specific implementation of steps 501 to 502 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0116] Step 503: If the number of nodes in any branch of the state transition graph is greater than the threshold, the task corresponding to any branch is decomposed into multiple subtasks.

[0117] The threshold can be used to determine the number of nodes required to determine whether the task corresponding to a branch is a complex task. It can be preset as needed, and this disclosure does not limit it.

[0118] In this disclosure, after determining the state transition diagram based on the interaction data, since there may be a large number of branches with long trajectories and complex tasks in the state transition diagram, in order to reduce the complexity of the state transition diagram and ensure that the logic of the state transition diagram is clearer, the tasks corresponding to such branches can be decomposed.

[0119] Therefore, if the number of nodes in any branch of the state transition graph is greater than the threshold, the task corresponding to that branch can be determined to be a complex task. In this case, the task corresponding to that branch can be decomposed into multiple subtasks.

[0120] For example, if an alarm is triggered when creating a new calendar item, then the task includes "creating a new item in the calendar + setting an alarm for a specific day or time". In this case, the task can be split into two sub-tasks: one is creating the item, and the other is setting the alarm. This disclosure does not limit this.

[0121] In some possible implementations, when the task corresponding to any branch is decomposed into multiple sub-tasks, the trajectory diagram and operation data sequence corresponding to any branch can be input into the multimodal model to obtain the multiple sub-tasks corresponding to any branch, as well as the starting user interface diagram and ending user interface diagram corresponding to each sub-task output by the multimodal model.

[0122] In some possible implementations, when decomposing the task corresponding to any branch into multiple subtasks, the number of subtasks included in the task corresponding to any branch can be determined based on the number of entities and / or operation data included in the task corresponding to any branch.

[0123] It should be noted that when determining the multiple subtasks included in the task corresponding to any branch based on the number of entities and / or operation data included in the task corresponding to any branch, a language model can be used to parse the task, and then the task can be decomposed according to the number of entities and / or operation data included in the task. This disclosure does not limit this.

[0124] In this disclosure, when decomposing a task into multiple corresponding sub-tasks, multimodal models can be used to identify trajectory maps and operation data sequences for decomposition. Alternatively, language models can be directly used to parse the task for decomposition, such as based on the number of entities and / or operation data in the task. Alternatively, task decomposition can be performed simultaneously using multimodal models and based on the number of entities and / or operation data included in the task. Thus, tasks can be decomposed in multiple ways, and the specific decomposition method can be determined as needed, improving the diversity and flexibility of task decomposition and ensuring the accuracy and reliability of the obtained sub-tasks.

[0125] Step 504: Based on multiple subtasks, decompose the state transition diagram corresponding to any branch into multiple sub-state transition diagram branches.

[0126] In this context, each sub-state transition graph branch corresponds to a sub-task.

[0127] In this disclosure, after decomposing the task corresponding to any branch into multiple subtasks, the state transition diagram corresponding to any branch can be decomposed into multiple sub-state transition diagram branches based on these multiple subtasks. This improves the readability and logical clarity of the state transition diagram by decomposing branches with longer trajectories and more complex tasks into multiple sub-state transition branches.

[0128] For example, taking the above example, based on the two sub-tasks of creating an item and setting an alarm, the state transition diagram of the corresponding branch is decomposed into the sub-state transition diagram branches corresponding to these two sub-tasks. This disclosure does not limit this.

[0129] Step 505: Generate multiple trajectory data based on the state transition diagram.

[0130] Each trajectory data point is used to indicate the trajectory diagram and operation data sequence corresponding to a task.

[0131] In this disclosure, after decomposing the state transition diagram corresponding to a branch with a long trajectory and complex task into multiple sub-state transition diagram branches, multiple trajectory data can be generated based on all branches contained in the state transition diagram.

[0132] Step 506: Input the trajectory data into the multimodal model to obtain the inference reference information corresponding to the operational data in the trajectory data output by the multimodal model.

[0133] Step 507: Based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data, generate training data for training the interactive agent model.

[0134] The specific implementation of steps 505 to 507 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0135] In this embodiment, interaction data with the application is first acquired, and a state transition graph is determined based on the interaction data. Then, if the number of nodes in any branch of the state transition graph exceeds a threshold, the task corresponding to that branch is decomposed into multiple sub-tasks. Based on these sub-tasks, the state transition graph corresponding to that branch is further decomposed into multiple sub-state transition graph branches. Next, multiple trajectory data are generated based on the state transition graph, and the trajectory data is input into a multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model. Finally, training data for training the interaction agent model is generated based on the trajectory data and the inference reference information corresponding to the operational data in the trajectory data. Thus, after determining the state transition graph based on the interaction data between the interaction agent model and the application, the readability and logical clarity of the state transition graph are improved by decomposing branches with longer trajectories and more complex tasks into multiple sub-branches. This allows for the rapid and accurate generation of trajectory data for different tasks based on the state transition graph, as well as the determination of inference reference information for the operational data in the trajectory data. Training data is then generated based on the trajectory data and inference reference information, achieving automated generation of training data for training the interaction agent model and improving the efficiency of training data generation.

[0136] Figure 6 This is a schematic flowchart of a training data generation method provided in another embodiment of the present disclosure.

[0137] like Figure 6 As shown, the training data generation method includes:

[0138] Step 601: Obtain interaction data with the application.

[0139] The interactive data includes multiple operation data, a user interface diagram before the operation for each operation data, and a user interface diagram after the operation.

[0140] Step 602: Determine the state transition diagram based on the interaction data.

[0141] In the state transition diagram, the nodes are user interface diagrams, and the connecting edges between any two user interface diagrams are the operation data associated with those two user interface diagrams.

[0142] Step 603: If the number of nodes in any branch of the state transition graph is greater than the threshold, the task corresponding to any branch is decomposed into multiple subtasks.

[0143] The specific implementation of steps 601 to 603 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0144] Step 604: Determine the reference user interface diagram for each subtask.

[0145] The reference user interface diagram for each subtask is pre-set according to the specific subtask, and this disclosure does not limit it.

[0146] In this disclosure, after the task corresponding to the branch is broken down into multiple subtasks, some operations and user interface diagrams in the subtasks may be missing when the task is broken down into subtasks, resulting in incomplete subtasks obtained from the decomposition.

[0147] For example, taking the task decomposition example above, the two sub-tasks obtained from the decomposition are the task of creating an event and the task of setting an alarm. However, the starting user interface diagram of the decomposed task of setting an alarm may be directly the time setting interface, and the preceding operations and user interface diagrams are missing.

[0148] Therefore, in order to determine whether the subtasks obtained from the decomposition are complete, and to accurately and reliably complete the incomplete subtasks, we can first determine the reference user interface diagram for each subtask.

[0149] Step 605: If the starting user interface diagram corresponding to any subtask does not match the reference user interface diagram, determine the corresponding ending user interface diagram in the state transition diagram that matches the starting user interface diagram of any subtask and includes the subtask with the fewest nodes.

[0150] In this disclosure, if the starting user interface diagram corresponding to any subtask does not match the reference user interface diagram, it can be determined that the subtask is incomplete, lacking some operations and user interface diagrams. In this case, the corresponding ending user interface diagram in the state transition diagram can be determined as the subtask with the fewest nodes compared to the starting user interface diagram of that subtask. This ensures that when completing the subtask, redundant operations and user interface diagrams are greatly reduced, improving quality.

[0151] Step 606: Based on the state transition diagram corresponding to the subtask, the state transition diagram corresponding to any subtask is spliced ​​and completed to obtain the complete state transition diagram of any subtask.

[0152] In this disclosure, the state transition diagram corresponding to any subtask is stitched together and completed by matching the initial user interface diagram of any subtask with the fewest nodes, so as to obtain the complete state transition diagram of any subtask. This effectively reduces redundant trajectories in the completed subtask, ensures the logical clarity and readability of the completed subtask, and improves data quality.

[0153] Step 607: Generate multiple trajectory data based on the state transition diagram.

[0154] Each trajectory data point is used to indicate the trajectory diagram and operation data sequence corresponding to a task.

[0155] In this disclosure, after completing the state transition diagram corresponding to the subtask in the state transition diagram, multiple trajectory data can be generated based on the state transition diagrams corresponding to all tasks contained in the state transition diagram.

[0156] Step 608: Input the trajectory data into the multimodal model to obtain the inference reference information corresponding to the operational data in the trajectory data output by the multimodal model.

[0157] Step 609: Based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data, generate training data for training the interaction agent model.

[0158] The specific implementation of steps 607 to 609 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0159] In this embodiment, interaction data with the application is first acquired, and a state transition diagram is determined based on the interaction data. If the number of nodes in any branch of the state transition diagram is greater than a threshold, the task corresponding to any branch is decomposed into multiple subtasks. Then, a reference user interface diagram for each subtask is determined. If the starting user interface diagram corresponding to any subtask does not match the reference user interface diagram, the corresponding ending user interface diagram in the state transition diagram is determined. The subtask that matches the starting user interface diagram of any subtask and includes the fewest nodes is selected. Based on the state transition diagram corresponding to this subtask, the state transition diagram corresponding to this subtask is stitched together and completed to obtain the complete state transition diagram of this subtask. Then, based on the state transition diagram, multiple trajectory data are generated, and the trajectory data is input into a multimodal model to obtain inference reference information corresponding to the operation data in the trajectory data output by the multimodal model. Finally, based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data, training data for training the interaction agent model is generated. Therefore, based on the interaction data between the interactive agent model and the application, a state transition diagram is generated. After decomposing the tasks corresponding to the longer branches in the state transition diagram into multiple subtasks, if the starting interface diagram of a subtask does not match its corresponding reference user interface diagram, it can be determined that the subtask is incomplete. By matching the corresponding ending user interface diagram in the state transition diagram with the starting user interface diagram of the incomplete subtask, and including the subtask with the fewest nodes, the incomplete subtask is completed. This effectively reduces the amount of redundant data in the completed subtask and improves the quality of the state transition diagram. Next, based on the state transition diagram, high-quality trajectory data can be obtained. Then, using a multimodal model to process the high-quality trajectory data, high-quality inference reference information can be obtained. Based on the high-quality trajectory data and inference reference information, high-quality training data can be automatically generated, improving the efficiency and accuracy of training data generation. This allows for the training of a high-performance interactive agent model to enhance the user's human-computer interaction experience.

[0160] To implement the above embodiments, this disclosure also proposes a training data generation device.

[0161] Figure 7 This is a schematic diagram of the structure of a training data generation device provided in an embodiment of the present disclosure.

[0162] like Figure 7 As shown, the training data generation device 700 includes: an acquisition module 701, a determination module 702, a first generation module 703, an input module 704, and a second generation module 705.

[0163] The acquisition module 701 is used to acquire interaction data with the application, wherein the interaction data includes multiple operation data, a user interface diagram before the operation corresponding to each operation data, and a user interface diagram after the operation.

[0164] The determination module 702 is used to determine the state transition diagram based on the interaction data, wherein the nodes in the state transition diagram are user interface diagrams, and the connecting edges between every two user interface diagrams are operation data associated with the two user interface diagrams.

[0165] The first generation module 703 is used to generate multiple trajectory data based on the state transition diagram, wherein each trajectory data is used to indicate the trajectory diagram and operation data sequence corresponding to a task;

[0166] The input module 704 is used to input trajectory data into the multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model. The inference reference information is used to describe the inference process of the operational data.

[0167] The second generation module 705 is used to generate training data for training the interactive agent model based on trajectory data and inference reference information corresponding to the operation data in the trajectory data.

[0168] Optionally, the aforementioned acquisition module 701 is specifically used for:

[0169] Get any user interface image of the application;

[0170] Identify any user interface diagram and determine the set of operable elements in any user interface diagram;

[0171] Perform an operation on each operable element in the operable element set to obtain the corresponding operation data and the user interface diagram after the operation.

[0172] Optionally, the determining module 702 described above is further configured to:

[0173] In the case of a loop path in the state transition graph, determine the target task corresponding to the loop path;

[0174] Based on the target task, determine the starting and ending user interface diagrams within the loop path;

[0175] Disconnect the edges between the starting user interface diagram and the ending user interface diagram to obtain the acyclic state transition diagram corresponding to the target task.

[0176] Optionally, the first generation module 703 described above is further configured to:

[0177] Based on the operation data corresponding to each connection edge in the state transition diagram, determine the operation object corresponding to the operation data;

[0178] Label the operation objects in the user interface diagram before the operation corresponding to the operation data to obtain the labeled user interface diagram;

[0179] Following the order of operations from front to back, all the labeled user interface diagrams corresponding to each task are stitched together sequentially to obtain the trajectory diagram corresponding to the task.

[0180] Optionally, the determining module 702 described above is further configured to:

[0181] If the number of nodes in any branch of the state transition graph is greater than the threshold, the task corresponding to any branch will be decomposed into multiple subtasks.

[0182] Based on multiple subtasks, the state transition diagram corresponding to any branch is decomposed into multiple sub-state transition diagram branches, where each sub-state transition diagram branch corresponds to one subtask.

[0183] Optionally, the determining module 702 described above is further configured to:

[0184] Determine the reference user interface diagram for each subtask;

[0185] If the starting user interface diagram for any subtask does not match the reference user interface diagram, determine the corresponding ending user interface diagram in the state transition diagram that matches the starting user interface diagram of any subtask and includes the subtask with the fewest nodes.

[0186] Based on the state transition diagram corresponding to a subtask, the state transition diagram corresponding to any subtask is pieced together and completed to obtain the complete state transition diagram of any subtask.

[0187] Optionally, the determining module 702 described above is further configured to perform at least one of the following:

[0188] Input the trajectory diagram and operation data sequence corresponding to any branch into the multimodal model to obtain multiple sub-tasks corresponding to any branch, as well as the starting user interface diagram and ending user interface diagram corresponding to each sub-task output by the multimodal model.

[0189] Based on the number of entities and / or operation data included in the task corresponding to any branch, determine the multiple subtasks included in the task corresponding to any branch.

[0190] Optionally, the first generation module 703 described above is further configured to:

[0191] Based on the trajectory diagrams in each trajectory data, determine the similarity between the trajectory data;

[0192] If the similarity between any two trajectory data is greater than the similarity threshold, one of the two trajectory data will be deleted from the multiple trajectory data.

[0193] It should be noted that the explanation of the aforementioned training data generation method embodiment also applies to the training data generation device of this embodiment, and therefore will not be repeated here.

[0194] In this embodiment, interaction data with the application is first acquired, and a state transition diagram is determined based on the interaction data. Then, multiple trajectory data are generated based on the state transition diagram. The trajectory data is then input into a multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model. Finally, training data for training the interaction agent model is generated based on the trajectory data and the inference reference information corresponding to the operational data within the trajectory data. Thus, by parsing the collected interaction data between the interaction agent model and the application, trajectory data corresponding to different tasks is generated, thereby achieving automatic generation of trajectory data. This provides a basis for reducing the cost and time consumption of training data generation. Furthermore, the multimodal model is used to generate inference reference information for the operational data within the trajectory data. Based on the trajectory data and the inference reference information for the operational data within it, training data is automatically generated, improving the efficiency of training data generation.

[0195] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0196] Figure 8 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0197] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 802 or a computer program loaded from storage unit 808 into RAM (Random Access Memory) 803. RAM 803 can also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. I / O (Input / Output) interface 805 is also connected to bus 804.

[0198] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0199] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as training data generation methods. For example, in some embodiments, the training data generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the training data generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a training data generation method by any other suitable means (e.g., by means of firmware).

[0200] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0201] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0202] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0204] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0205] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0206] According to embodiments of this disclosure, this disclosure also provides a computer program product that, when executed by an instruction processor, performs the training data generation method proposed in the above embodiments of this disclosure.

[0207] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0208] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating training data, characterized in that, include: Acquire interaction data with the application, wherein the interaction data includes multiple operation data, a user interface diagram before the operation corresponding to each operation data, and a user interface diagram after the operation; Based on the interaction data, a state transition diagram is determined, wherein the nodes in the state transition diagram are user interface diagrams, and the connecting edges between every two user interface diagrams are operation data associated with the two user interface diagrams; Based on the state transition diagram, multiple trajectory data are generated, wherein each trajectory data is used to indicate the trajectory diagram and operation data sequence corresponding to a task; The trajectory data is input into a multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model, wherein the inference reference information is used to describe the inference process of the operational data; Based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data, training data for training the interactive agent model is generated.

2. The method as described in claim 1, characterized in that, The acquisition of interaction data with the application includes: Obtain any user interface image of the application; Identify any of the user interface diagrams to determine the set of operable elements in any of the user interface diagrams; Perform an operation on each operable element in the set of operable elements to obtain the corresponding operation data and the user interface diagram after the operation.

3. The method as described in claim 1, characterized in that, Following the determination of the state transition diagram, the following is also included: In the case where the state transition diagram includes a cyclic path, the target task corresponding to the cyclic path is determined; Based on the target task, determine the starting user interface diagram and the ending user interface diagram within the loop path; Disconnect the connection edges between the starting user interface diagram and the ending user interface diagram to obtain the acyclic state transition diagram corresponding to the target task.

4. The method as described in claim 1, characterized in that, The process of generating the trajectory map includes: Based on the operation data corresponding to each connection edge in the state transition diagram, determine the operation object corresponding to the operation data; The operation objects in the user interface diagram before the operation corresponding to the operation data are labeled to obtain the labeled user interface diagram. Following the order of operations from front to back, all the labeled user interface images corresponding to each task are stitched together sequentially to obtain the trajectory image corresponding to the task.

5. The method according to any one of claims 1-4, characterized in that, Following the determination of the state transition diagram, the following is also included: If the number of nodes in any branch of the state transition graph is greater than a threshold, the task corresponding to any branch is decomposed into multiple subtasks. Based on the multiple subtasks, the state transition diagram corresponding to any branch is decomposed into multiple sub-state transition diagram branches, wherein one sub-state transition diagram branch corresponds to one subtask.

6. The method as described in claim 5, characterized in that, After decomposing any one branch into multiple sub-branches, the method further includes: Determine the reference user interface diagram for each subtask; If the starting user interface diagram corresponding to any subtask does not match the reference user interface diagram, determine the corresponding ending user interface diagram in the state transition diagram, which matches the starting user interface diagram of any subtask and includes the subtask with the fewest nodes. Based on the state transition diagram corresponding to the subtask, the state transition diagram corresponding to any subtask is pieced together and completed to obtain the complete state transition diagram of any subtask.

7. The method as described in claim 5, characterized in that, The step of breaking down the task corresponding to any one branch into multiple sub-tasks includes at least one of the following: Input the trajectory diagram and operation data sequence corresponding to any branch into the multimodal model to obtain multiple sub-tasks corresponding to any branch, and the starting user interface diagram and ending user interface diagram corresponding to each sub-task output by the multimodal model. Based on the number of entities and / or operation data included in the task corresponding to any one branch, determine the multiple sub-tasks included in the task corresponding to any one branch.

8. The method according to any one of claims 1-4, characterized in that, After generating multiple trajectory data, the process also includes: Based on the trajectory diagrams in each trajectory data, determine the similarity between the trajectory data; If the similarity between any two trajectory data is greater than a similarity threshold, one of the two trajectory data will be deleted from the plurality of trajectory data.

9. A training data generation device, characterized in that, include: The acquisition module is used to acquire interaction data with the application, wherein the interaction data includes multiple operation data, a user interface diagram before the operation corresponding to each operation data, and a user interface diagram after the operation. The determination module is used to determine a state transition diagram based on the interaction data, wherein the nodes in the state transition diagram are user interface diagrams, and the connecting edges between every two user interface diagrams are operation data associated with the two user interface diagrams; The first generation module is used to generate multiple trajectory data based on the state transition diagram, wherein each trajectory data is used to indicate a trajectory diagram and operation data sequence corresponding to a task; An input module is used to input the trajectory data into a multimodal model to obtain inference reference information corresponding to the operational data in the trajectory data output by the multimodal model, wherein the inference reference information is used to describe the inference process of the operational data; The second generation module is used to generate training data for training the interactive agent model based on the trajectory data and the inference reference information corresponding to the operation data in the trajectory data.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; characterized in that, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

Images