Interaction data construction method, interaction prediction model training method, and related devices

Abstract

Embodiments of the present application disclose an interactive data construction method, a training method of an interactive prediction model, and related devices. The interactive data construction method comprises: obtaining a first interface sequence collected by a sample application when the sample application executes a sample interactive task; performing field masking based on the correlation between each data field in the interface structure data of the sample interactive interface and the sample interactive interface to obtain the masked structure data of the sample interactive interface; selecting a first interactive interface to perform a deduplication operation based on the first similarity between the data feature representations extracted from the masked structure data of each sample interactive interface, to obtain a new first interface sequence; dividing based on the new first interface sequence to obtain a second interface sequence, and obtaining first sample interactive data of the interactive prediction model. The above scheme can improve the construction quality of the interactive data and help improve the training effect of the interactive prediction model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an interactive data construction method, an interactive prediction model training method, and related apparatus. Background Technology

[0002] With the development of artificial intelligence technology, AI models can be used to build instruction-driven interactive intelligent agents, enabling them to perform multi-step tasks in external environments, such as software interface operation, information retrieval, and shopping order placement.

[0003] In existing technologies, training artificial intelligence models typically relies on manually collecting a large number of interaction trajectories. However, manual collection is not only inefficient, but the quality of the collected training data also varies, impacting model training effectiveness and resulting in poor model performance. Therefore, improving the quality of constructed interaction data to enhance the training performance of interaction prediction models has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a method for constructing interactive data, a method for training interactive prediction models, and related apparatus, which can improve the quality of interactive data construction and help improve the training effect of interactive prediction models.

[0005] To address the aforementioned technical problems, the first aspect of this application provides an interactive data construction method, comprising: acquiring a first interface sequence collected by a sample application when performing a sample interaction task; wherein the first interface sequence includes several sample interaction interfaces, and adjacent sample interaction interfaces are switched by sample interaction actions; performing field masking based on the correlation between each data field in the interface structure data of the sample interaction interfaces and the sample interaction interfaces to obtain mask structure data of the sample interaction interfaces; selecting a pair of first interaction interfaces based on a first similarity between the data feature representations extracted from the mask structure data of each sample interaction interface to perform a deduplication operation to obtain a new first interface sequence; wherein, during the deduplication operation, one sample interaction interface in the first interaction interface pair is replaced by another sample interaction interface; dividing the new first interface sequence to obtain a second interface sequence; wherein the second interface sequence includes at least one pair of adjacent sample interaction interfaces and the sample interaction actions between them; and obtaining first sample interaction data of an interaction prediction model based on the second interface sequence and the sample interaction task to which the second interface sequence originates; wherein the interaction prediction model is used to predict interaction actions based on the task to be interacted with.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a training method for an interactive prediction model, comprising: acquiring first sample interaction data of the interactive prediction model; wherein the first sample interaction data includes: adjacent sample interaction interfaces and sample interaction actions between them when performing a sample interaction task, and the first sample interaction data is obtained by the interaction data construction method in the first aspect described above; and training the interactive prediction model based on the first sample interaction data.

[0007] To address the aforementioned issues, a third aspect of this application provides a task execution method, comprising: responding to an interactive task of a target application, obtaining the current interactive interface of the target application; predicting the interactive task and the current interactive interface based on an interaction prediction model to obtain a predicted interactive action; wherein the interaction prediction model is trained based on the training method of the interaction prediction model in the second aspect described above; invoking a target agent to execute the predicted interactive action in the target application to obtain a new current interactive interface of the target application; returning to the step of predicting the interactive task and the current interactive interface based on the interaction prediction model to obtain the predicted interactive action, until the interactive task is completed.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides an interactive data construction apparatus, comprising: a sequence acquisition module, a field masking module, a deduplication module, a sequence partitioning module, and a sample construction module. The sequence acquisition module acquires a first interface sequence collected by the sample application during the execution of a sample interaction task. The first interface sequence includes several sample interaction interfaces, and adjacent sample interaction interfaces are transitioned between each other by a sample interaction action. The field masking module performs field masking based on the correlation between each data field in the interface structure data of the sample interaction interface and the sample interaction interface, obtaining mask structure data of the sample interaction interface. The deduplication module performs deduplication based on the mask structure of each sample interaction interface. The data extracts feature representations of the data and perform a deduplication operation on the first interactive interface pair to obtain a new first interface sequence. During the deduplication operation, one sample interactive interface in the first interactive interface pair is replaced by another sample interactive interface. The sequence segmentation module is used to segment the data based on the new first interface sequence to obtain a second interface sequence. The second interface sequence contains at least one pair of adjacent sample interactive interfaces and the sample interactive actions between them. The sample construction module is used to obtain the first sample interactive data of the interaction prediction model based on the second interface sequence and the sample interactive task from which the second interface sequence originates. The interaction prediction model is used to predict the interactive actions based on the task to be interacted with.

[0009] To address the aforementioned technical problems, a fifth aspect of this application provides a training apparatus for an interactive model, comprising: a sample acquisition module and a model training module. The sample acquisition module is used to acquire first sample interaction data of the interactive prediction model. The first sample interaction data includes: adjacent sample interaction interfaces and sample interaction actions between them when performing a sample interaction task, and the first sample interaction data is obtained by the interaction data construction apparatus in the fourth aspect described above. The model training module is used to train the interactive prediction model based on the first sample interaction data.

[0010] To address the aforementioned technical problems, a sixth aspect of this application provides a task execution apparatus, comprising an interface acquisition module, an action prediction module, an action execution module, and an iterative execution module. The interface acquisition module is used to acquire the current interactive interface of the target application in response to an interactive task to be performed by the target application. The action prediction module is used to predict the interactive action based on the interactive task to be performed and the current interactive interface using an interaction prediction model. The interaction prediction model is trained using the training device of the interaction prediction model in the fifth aspect described above. The action execution module is used to invoke the target agent to perform the predicted interactive action in the target application, thereby obtaining a new current interactive interface of the target application. The iterative execution module is used to return to the step of predicting the interactive action based on the interactive prediction model and the current interactive interface, until the interactive task to be performed is completed.

[0011] To address the aforementioned technical problems, the seventh aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other. The memory stores at least program instructions, and the processor executes the program instructions to implement the interactive data construction method in the first aspect, or the interactive model training method in the second aspect, or the task execution method in the third aspect.

[0012] To address the aforementioned technical problems, the eighth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor. These program instructions are used to implement the interactive data construction method in the first aspect, the interactive model training method in the second aspect, or the task execution method in the third aspect.

[0013] The above scheme, in the interaction data construction stage, on the one hand, performs field masking based on the correlation between each data field in the interface structure data of the sample interaction interface and the sample interaction interface, and extracts the first similarity between the data feature representations based on the mask structure data of each sample interaction interface for deduplication. This can reduce the essentially the same but redundant collected data in the first interface sequence while retaining key samples as much as possible, based on the essential content of the sample interaction interface represented by the mask structure data feature representation, thus improving the construction quality of interaction data. On the other hand, by extracting the sequence combination of adjacent interaction interfaces and their triggering actions, the resulting second interface sequence, as well as the sample interaction task to which the second interface sequence originates from the first interface sequence, ensures the correlation of training samples in task processing logic and covers different interaction scenarios as much as possible, thus helping to improve the training effect of the interaction prediction model. In the interaction prediction model training stage, the interaction prediction model is trained based on the first sample interaction data, including the sample interaction actions of adjacent sample interaction interfaces and between them when executing sample interaction tasks. This can improve the training effect of the interaction prediction model based on the first similarity between the first sample interaction data and the data feature representations of each sample interaction interface. The task processing logic in this interactive data enables the interaction prediction model to understand the position and role of interactive actions in the task flow, thereby improving the model's prediction accuracy. Furthermore, since the first sample interaction data has undergone deduplication, removing redundant information and retaining key samples, it helps the interaction prediction model learn interaction features more efficiently during training, avoiding reduced training efficiency due to redundant data interference, thus improving the overall training effect of the interaction prediction model. During the task execution phase, based on the trained interaction prediction model, it can progressively predict the interactive actions to be executed according to the target application's interactive task and the current interactive interface, and call the target agent to progressively execute the predicted interactive action sequence in the target application, achieving automated execution of the interactive task. This not only improves task execution efficiency but also reduces errors and deviations that may be caused by manual operation. Because the interaction prediction model is trained using pre-constructed first sample interaction data, its accuracy and robustness in predicting interactive actions are improved, enabling it to cope with various complex situations that may occur during actual interaction, and improving the smoothness and accuracy of task execution. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating an embodiment of the interactive data construction method of this application; Figure 2 This is a schematic diagram of the framework of another embodiment of the interactive data construction method of this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the interactive data construction method of this application for constructing second sample interactive data; Figure 4 This is a flowchart illustrating an embodiment of the training method for the interactive prediction model of this application; Figure 5 This is a schematic diagram of the framework of an embodiment of the interactive prediction model pre-training in the training method of the interactive prediction model of this application; Figure 6 This is a schematic diagram of the framework of an embodiment of supervised training of the interactive prediction model in the training method of the interactive prediction model of this application; Figure 7 This is a schematic diagram of the framework of an embodiment of online reflective fine-tuning of the interactive prediction model in the training method of the interactive prediction model of this application; Figure 8 This is a flowchart illustrating an embodiment of the task execution method of this application; Figure 9 This is a schematic diagram of the structure of an embodiment of the task execution method of this application for deploying an interactive prediction model; Figure 10 This is a schematic diagram of the framework of an embodiment of the interactive data construction apparatus of this application; Figure 11 This is a schematic diagram of the framework of an embodiment of the training device for the interactive model of this application; Figure 12 This is a schematic diagram of the framework of an embodiment of the task execution device of this application; Figure 13 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 14 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0015] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0016] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0017] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0018] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0019] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the interactive data construction method of this application. Specifically, it may include the following steps: Step S11: Obtain the first interface sequence of the sample application when the sample application performs the sample interaction task.

[0020] In this embodiment of the disclosure, the first interface sequence includes several sample interaction interfaces, and adjacent sample interaction interfaces are switched by sample interaction actions.

[0021] In an implementation scenario, a sample interaction task refers to the target description of the task to be completed, which can be a natural language instruction, a structured target, or a combination of both. Specifically, sample interaction tasks can cover a variety of types, such as document editing in office software, web browsing in a browser, and performing specific operations in a game.

[0022] In a specific implementation scenario, an interactive action refers to an interactive operation performed on the interactive interface at a time step. Specifically, an interactive action includes the action type and its parameters, such as click coordinates, scroll direction and distance, input target and text, etc.

[0023] In a specific implementation scenario, each specific operation moment during the interaction process can be regarded as a time step. Each time step corresponds to a specific interactive action and the subsequent change in the interface state. Specifically, the introduction of time steps allows the interaction process to be subdivided into a series of discrete, ordered operation units. Each unit includes the interactive action executed at that moment and the immediate impact of that action on the interactive interface.

[0024] In one implementation scenario, the sample interactive interface is specifically a graphical user interface (GUI), which refers to a software interface composed of visual elements that can be interacted with by the user through clicking, inputting, swiping, scrolling, etc.

[0025] In a specific implementation scenario, the collected sample interaction interface includes screenshots of the corresponding graphical user interface, interface structure data, and metadata. The screenshots represent the visual presentation of the sample interaction interface. Each sample interaction interface in the first interface sequence collected specifically includes interface information. Specifically, the interface information includes screenshots of the sample interaction interface, interface structure information, and metadata. The metadata includes entry point, exploration depth, timestamp, page path summary, risk markers, etc.

[0026] In a specific implementation scenario, interface structure data represents the structured representation of the interactive interface, such as an interface hierarchy tree, control attribute set, accessibility tree, or XML. This data is used to assist in locating interactive elements and determining the feasibility of actions. Interactive elements refer to the set of elements that can be triggered by clicks, input, scrolling, etc., in the interactive interface state. Each element can contain information such as its bounding box, type, interactive attributes, and text.

[0027] In one implementation scenario, the sample interaction interface of the sample application at the start of the sample interaction task is obtained as the first sample interaction interface in the first interface sequence. Based on the interface structure data of the latest sample interaction interface in the first interface sequence, the set of operable elements of the latest sample interaction interface in the first interface sequence is parsed to obtain the set of operable elements of the latest sample interaction interface in the first interface sequence. The operable element related to the sample interaction task in the set of operable elements is selected as the target operation element. The interaction action performed on the target operation element is taken as the sample interaction action. Based on the latest sample interaction interface in the first interface sequence, the latest sample interaction action is executed to obtain the new jump interaction interface of the sample application as the new sample interaction interface in the first interface sequence. The process of parsing the interface structure data of the latest sample interaction interface in the first interface sequence to obtain the set of operable elements of the latest sample interaction interface in the first interface sequence is repeated until the sample interaction task is completed. The above scheme uses the initial sample interaction interface of the sample application when performing the sample interaction task as the starting point of the first interface sequence, and continuously analyzes the interface structure data of the latest sample interaction interface to locate the operable elements related to the task, thereby triggering the corresponding sample interaction action and updating the interaction interface until the task is completed. It can automatically construct a first interface sequence containing several interaction scenarios and task logic, which not only provides the richest possible basic data for subsequent data deduplication and feature extraction, but also ensures the continuity and diversity of training samples in the task processing flow, thereby further improving the quality and efficiency of interaction data construction.

[0028] Please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of the framework of another embodiment of the interactive data construction method of this application. For example... Figure 2 As shown, in a specific implementation scenario, starting from the first sample interaction interface, it is added to the exploration queue (BFS) or the exploration stack (DFS), i.e., the first interface sequence. The visited set and the exploration set are maintained, and the exploration process of "retrieving the sample interaction interface in the exploration state - generating an action - executing the action - recording the result - updating the set" is executed in a loop. For each sample interaction interface in the exploration state, its corresponding interface screenshot and interface structure information are obtained. Based on the interface structure data of the latest sample interaction interface in the first interface sequence, the set of operable elements of the latest sample interaction interface in the first interface sequence is parsed to obtain the set of operable elements of the latest sample interaction interface in the first interface sequence. Specifically, the parsing rules include: obtaining click / long press candidates for clickable / long pressable elements, obtaining input candidates (including input box positioning and input content generation strategy) for editable text boxes, and obtaining scrolling candidates (including direction, distance or start and end points) for scrollable containers, etc.

[0029] In a specific implementation scenario, for each sample interaction interface in the state to be explored, its corresponding interface screenshot and interface structure information are obtained. Based on the interface structure data of the latest sample interaction interface in the first interface sequence, the operable elements of the latest sample interaction interface in the first interface sequence are parsed. Then, a set of candidate actions is generated based on the operable elements, and the target operation element associated with the sample interaction task is selected as the target operation element. The action type includes at least click, input, swipe / scroll, and return. Specifically, the upper limit of candidate actions for each state can be set, such as trying a maximum of K click points or trying a maximum of M distances for each scrolling container. After the action is executed, the type and parameters of the executed target operation element, the interface information of the sample interaction interface before execution, and the interface information of the new sample interaction interface after execution are recorded. Specifically, the interface information includes the interaction interface screenshot and interface structure data.

[0030] In a specific implementation scenario, during the process of acquiring the first interface sequence of the sample application while performing sample interaction tasks, a rollback strategy is also set up. This strategy allows users to revert from deeper sample interaction interfaces to higher-level ones by returning, closing pop-ups, or returning to the homepage, enabling them to traverse other branches in different sample interaction tasks. Specifically, the rollback strategy can be flexibly set according to the interaction scenario and task requirements. For example, it can be triggered when encountering error messages, being unable to continue an action, or reaching a preset exploration depth. This ensures that the construction process of the first interface sequence covers more interaction paths and scenarios, improving the comprehensiveness and robustness of data construction.

[0031] In a specific implementation scenario, during the process of acquiring the first interface sequence of the sample application when it performs sample interaction tasks, security control strategies are also set, including: sensitive page identification and skipping, high-risk action blacklist, high-risk action whitelist, maximum exploration depth, maximum number of steps, timeout termination, and abnormal recovery (returning to the entry state after crash restart).

[0032] Step S12: Based on the correlation between each data field in the interface structure data of the sample interactive interface and the sample interactive interface, perform field masking to obtain the mask structure data of the sample interactive interface.

[0033] In one implementation scenario, the correlation between various data fields in the interface structure data of a sample interactive interface and the sample interactive interface itself is obtained. Specifically, the correlation can be comprehensively evaluated by calculating multiple dimensions such as the frequency of occurrence of data fields in the interface structure data, their positional importance, and their relevance to the interaction task. For example, for a node in the interface hierarchy tree, its correlation may be related to factors such as the node's depth in the tree, whether it contains interactive elements, and the frequency of the element's use in the interaction task. Specifically, dynamic fields, random IDs, and ad slots in the interface structure data have low correlation with the sample interactive interface, while operable elements, page titles, and key navigation links have high correlation with the sample interactive interface. Based on these correlation evaluation results, field masking is performed on each data field, that is, selectively retaining fields according to their importance, thereby obtaining the masked structure data of the sample interactive interface. The masked structure data can highlight the core features of the sample interactive interface, reduce the interference of redundant information on subsequent processing, and improve the efficiency and accuracy of data processing.

[0034] In a specific implementation scenario, the field masking process can employ different masking strategies depending on actual needs. For example, for data fields with extremely low relevance, direct masking can be performed, replacing their values ​​with specific placeholders or deleting them; for data fields with high relevance, their original values ​​are retained. Through field masking, the resulting masked structure data not only retains the key information of the sample interactive interface but also removes redundant and irrelevant information, providing a more refined and effective data foundation for subsequent data deduplication, feature extraction, and training of interactive prediction models. Furthermore, the field masking strategy can be dynamically adjusted according to different interactive scenarios and task requirements to adapt to data processing needs in different scenarios, further enhancing the flexibility and practicality of interactive data construction.

[0035] In a specific implementation scenario, data fields in the interface structure data of the sample interactive interface that meet a preset condition regarding their relevance to the sample interactive interface are masked. Specifically, the preset condition is that the relevance is below a preset threshold, which can be flexibly set according to the actual application scenario and task requirements. For example, in a specific interactive task, if most data fields in the interface structure data have a low relevance to the interactive task, the preset threshold can be set to a relatively low value to retain more data fields that may affect the interactive task; conversely, if most data fields in the interface structure data have a high relevance to the interactive task, the preset threshold can be set to a relatively high value to remove more redundant data fields. This approach can better adapt to different interactive scenarios and task requirements, improving the targeting and effectiveness of interactive data construction.

[0036] Step S13: Based on the first similarity between the data feature representations extracted from the mask structure data of each sample interaction interface, select the first interaction interface pair to perform deduplication operation to obtain a new first interface sequence.

[0037] In this embodiment of the disclosure, during the deduplication operation, one sample interactive interface in the first interactive interface pair is replaced by the other sample interactive interface. Specifically, the first interactive interface pair includes two sample interactive interfaces. A first similarity is calculated between the data feature representations extracted from the mask structure data of these two sample interactive interfaces to determine whether they are similar or duplicated. If the first similarity exceeds a preset threshold, the two sample interactive interfaces are considered similar or duplicated. In this case, one sample interactive interface can be retained, while the other sample interactive interface is removed from the first interface sequence, and the retained sample interactive interface replaces the removed sample interactive interface.

[0038] In one implementation scenario, data feature representations are extracted based on the mask structure data of each sample interaction interface. Specifically, the extraction of data feature representations can be carried out in various ways, such as encoding the mask structure data through a deep learning model to obtain a fixed-dimensional feature vector. This feature vector can characterize the key information and structural features of the sample interaction interface.

[0039] In one implementation scenario, the system checks whether the first similarity between different sample interactive interfaces meets the filtering criteria. If the first similarity does not meet the filtering criteria, the system returns to the previous step of checking whether the similarity between different sample interactive interfaces meets the filtering criteria, until the similarity between any two sample interactive interfaces has been detected. If the first similarity meets the filtering criteria, the corresponding sample interactive interface is selected as the first interactive interface pair, and one of the sample interactive interfaces in the first interactive interface pair is replaced with the other sample interactive interface in the first interactive interface pair. In this scheme, when the first similarity between different sample interfaces does not meet the filtering criteria, it indicates that the two sample interfaces being compared are essentially different, so there is no need to perform deduplication. The system continues to detect similarity between other sample interactive interfaces until all sample interactive interfaces have been compared. When the first similarity between different sample interactive interfaces meets the filtering criteria, it means that the two sample interactive interfaces have a high degree of similarity in their essential content. In this case, selecting one sample interactive interface to replace the other can effectively reduce redundant data while retaining key sample information, thereby improving data quality while ensuring data diversity.

[0040] In a specific implementation scenario, the screening conditions include: the first similarity is not less than the first preset threshold. Specifically, the first preset threshold can be flexibly set according to the actual interaction scenario and task requirements, such as 0.8, 0.85, etc. When the first similarity calculated between two sample interaction interfaces reaches or exceeds the threshold, it is determined that the two meet the screening conditions and the deduplication operation can be performed.

[0041] In a specific implementation scenario, after extracting the data feature representation, the first similarity between the feature vectors of the two sample interaction interfaces in the first interaction interface pair is calculated. The similarity calculation method can adopt similarity measurement methods such as cosine similarity and Euclidean distance.

[0042] In one implementation scenario, before dividing the data based on the new first interface sequence to obtain the second interface sequence, feature extraction is performed on each sample interaction interface to obtain image feature representations of each sample interaction interface. Based on the second similarity between the image feature representations of each sample interaction interface, a deduplication operation is performed on the selected second interaction interface pair to obtain a new first interface sequence. During the deduplication operation, one sample interaction interface in the second interaction interface pair is replaced by another sample interaction interface. This scheme, by selecting the second interaction interface pair for deduplication based on the second similarity between the image feature representations of each sample interaction interface, can reduce essentially identical but redundant collected data in the first interface sequence in the image visual dimension. Therefore, it can further improve the quality of interaction data construction and help reduce the interference of redundant data on model training.

[0043] In a specific implementation scenario, feature extraction is performed on screenshots of each sample interaction interface to obtain image feature representations of each interface. The screenshots represent the visual presentation of the sample interaction interface, including visual information such as layout, color, and element shape. Using a deep learning model, such as a convolutional neural network (CNN), feature extraction is performed on these screenshots to obtain fixed-dimensional image feature vectors, which represent the visual features of the sample interaction interface. After obtaining the image feature representations of each sample interaction interface, a second similarity is calculated between the image feature vectors of two sample interaction interfaces in a second interaction interface pair. Specifically, the similarity calculation method can also employ commonly used similarity measures such as cosine similarity and Euclidean distance.

[0044] In a specific implementation scenario, if the second similarity between the image feature representations of two sample interactive interfaces in a second interactive interface pair exceeds a preset similarity threshold, then the two sample interactive interfaces are considered visually similar or repetitive. In this case, one sample interactive interface can be retained, while the other sample interactive interface is removed from the first interface sequence. The retained sample interactive interface replaces the removed sample interactive interface, thereby achieving deduplication at the image dimension. Through this processing, it can be ensured that the new first interface sequence does not contain too many visually repetitive sample interactive interfaces, thus improving the diversity and quality of the dataset.

[0045] In a specific implementation scenario, the similarity threshold can be flexibly adjusted according to actual application needs and scenario characteristics. For example, for interactive tasks with high requirements for visual details, the similarity threshold can be set relatively low to retain more sample interactive interfaces that are visually slightly different but may contain important information; while for interactive tasks with relatively low requirements for visual details, the similarity threshold can be set higher to more effectively remove visually highly repetitive sample interactive interfaces and reduce data redundancy.

[0046] In a specific implementation scenario, when performing the dual deduplication operation, a preliminary screening is first performed based on the feature similarity of the mask structure data to remove samples that are structurally highly similar. Then, a second screening is performed based on the similarity of the image feature representations to further eliminate visually duplicated samples. Specifically, the correlation between each data field in the interface structure data of the sample interaction interface and the sample interaction interface is masked to obtain the mask structure data of the sample interaction interface. Based on the first similarity between the data feature representations extracted from the mask structure data of each sample interaction interface, the first interaction interface pair is selected to perform the deduplication operation to obtain a new first interface sequence. Then, feature extraction is performed on each sample interaction interface in the new first interface sequence to obtain the image feature representation of each sample interaction interface. Based on the second similarity between the image feature representations of each sample interaction interface, the second interaction interface pair is selected to perform the deduplication operation to obtain a new first interface sequence. The above scheme achieves multi-dimensional deduplication from structure to vision by combining masking of interface structure data with similarity calculation of image feature representation. It can eliminate redundant information in the data and ensure that the retained sample interaction interface is representative at both the structural and visual levels. This not only ensures the efficiency of data processing but also improves the accuracy of deduplication results.

[0047] In one implementation scenario, after determining the first similarity between the data feature representations extracted from the mask structure data of each sample interaction interface, performing deduplication on the first interaction interface pair to obtain a new first interface sequence, and before dividing based on the new first interface sequence to obtain a second interface sequence, a target index score for the new first interface sequence is obtained. The target index score represents the first index score of the data coverage of the new first interface sequence and the second index score of the data deduplication effect of the new first interface sequence. In response to the target index score not meeting the preset conditions, the steps of determining the first similarity between the data feature representations extracted from the mask structure data of each sample interaction interface, performing deduplication on the first interaction interface pair, and obtaining a new first interface sequence are re-executed. The preset conditions include: the first index score is not greater than a second preset threshold and the second index score is not greater than a third preset threshold. The above scheme obtains the target index score of the new first interface sequence. This score includes the first index score of data coverage and the second index score of data deduplication effect. It can comprehensively evaluate the impact of deduplication operation on data quality. If the target index score does not meet the preset conditions, that is, the data coverage or deduplication effect does not meet the predetermined threshold, the deduplication operation is triggered to be re-executed. This ensures that the deduplication process can not only effectively reduce redundant data, but also maintain the broad coverage of data. Thus, while improving the quality of interactive data construction, it also ensures the comprehensiveness and diversity of data.

[0048] In a specific implementation scenario, the first indicator score representing the coverage of the new first interface sequence data specifically includes the first sub-score of the number of unique states after deduplication, the second sub-score of page / route coverage statistics (grouped by entry or page type), and the third sub-score of element coverage statistics (distribution of button, input box, list, pop-up, scroll container, etc. types). The second indicator score representing the deduplication effect of the new first interface sequence data specifically includes the fourth sub-score of the comparison of the number of samples before and after deduplication, the fifth sub-score of the interface structure repetition rate, and the sixth sub-score of the visual near repetition rate (cluster size distribution after clustering, or near repetition ratio under similarity threshold).

[0049] In a specific implementation scenario, the first indicator score is obtained by fusing the first sub-score, the second sub-score, and the third sub-score; the second indicator score is obtained by fusing the fourth sub-score, the fifth sub-score, and the sixth sub-score; and the first indicator score and the second indicator score are used to determine whether the preset conditions are met.

[0050] In another specific implementation scenario, the scores of the first, second, and third sub-scores are compared with the preset score thresholds of the corresponding indicators. If the scores of the first, second, and third sub-scores all satisfy the relationship between their magnitudes and the preset score thresholds of the corresponding indicators, then the score of the first indicator satisfies the preset condition. Similarly, the scores of the fourth, fifth, and sixth sub-scores are compared with the preset score thresholds of the corresponding indicators. If the scores of the fourth, fifth, and sixth sub-scores all satisfy the relationship between their magnitudes and the preset score thresholds of the corresponding indicators, then the score of the second indicator satisfies the preset condition.

[0051] In a specific implementation scenario, the target index score also includes a third index score on the effectiveness and stability of the transfer of new first interface sequence data. Specifically, the third index score includes the seventh sub-score of effective transfer rate (the proportion of effective interface changes after an action), the eighth sub-score of rollback success rate (the proportion of the interface that can be restored to the previous sample interaction interface after rollback), and the ninth sub-score of abnormal recovery rate (the proportion of the interface that can be restarted and returned to the initial sample interaction interface after a crash / freeze).

[0052] Step S14: Divide the new first interface sequence to obtain the second interface sequence.

[0053] In this embodiment of the disclosure, the second interface sequence includes at least one pair of adjacent sample interaction interfaces and the sample interaction actions between them.

[0054] In one implementation scenario, a second interface sequence is obtained based on partitioning rules. Specifically, the partitioning rules can be dynamically set according to the needs of the actual interaction task. For example, partitioning can be based on the logical relationship between sample interaction interfaces. If there is a clear operation flow order between two sample interaction interfaces, such as clicking the login button to enter the login interface, then entering the information to complete the login and entering the main interface, then the interface where the login button is located, the login information input interface, and the main interface can be divided into a group of adjacent sample interaction interfaces and their interaction actions according to this operation order, and included in the second interface sequence. Alternatively, partitioning can be based on time order. If the sample interaction interfaces are collected in chronological order, then sample interaction interfaces collected at adjacent time points and their interaction actions can be grouped together.

[0055] In a specific implementation scenario, for each sample interaction interface, adjacent sample interaction interfaces that conform to the partitioning rules are found. These adjacent interfaces and their interactions are combined to form independent units, which together constitute the second interface sequence. This approach makes the second interface sequence more consistent with the flow and logic of the actual interaction task, thus improving the quality of the training data construction.

[0056] In a specific implementation scenario, the new first interface sequence can be represented as follows: ,in, The sample interaction interface at the corresponding time step is characterized, specifically including screenshots of the sample interaction interface, interface structure data, etc. This represents the sample interaction actions between adjacent sample interfaces, which is understandable. In response to Execution of the interactive interface Then, based on the aforementioned new first interface sequence, a second interface sequence is obtained, for example, This serves as the second interface sequence obtained through partitioning.

[0057] Step S15: Based on the second interface sequence and the sample interaction task to which the first interface sequence from which the second interface sequence originates, obtain the first sample interaction data of the interaction prediction model.

[0058] In this embodiment of the disclosure, the interaction prediction model is used to predict interaction actions based on the task to be interacted with.

[0059] In one implementation scenario, after acquiring the second interface sequence, the system further integrates the specific information of the sample interaction task from which the second interface sequence originates, such as task type, task objective, and task execution environment. This information is then correlated and integrated with the sample interaction interfaces and their interactions within the second interface sequence to form the first sample interaction data of a structurally complete and information-rich interaction prediction model. Specifically, the first sample interaction data not only includes the dynamic interaction process between sample interaction interfaces but also incorporates task-related contextual information. This helps the interaction prediction model more accurately understand and learn patterns and rules in actual interaction tasks, thereby improving the accuracy and reliability of the model's prediction of interaction actions when facing interaction tasks.

[0060] In one implementation scenario, sample input data is obtained based on the sample interaction task, a third interface sequence, sample memory data at the target time step of executing the sample interaction task, and historical interaction actions executed at each historical time step before the target time step. The sample memory data represents reusable data across steps. The third interface sequence includes historical interaction interfaces triggered at several historical time steps before the target time step. Prediction is performed based on the sample input data to obtain sample output data. The sample output data includes the sample interaction actions at the target time step, the sample thought process of the sample interaction actions, and the sample action description of the sample interaction actions. The sample input data serves as the input data for the interaction prediction model training phase, and the sample output data serves as the training objective for the interaction prediction model training phase. Based on the sample input data and sample output data, the second sample interaction data for the interaction prediction model is constructed. For example, the sample interaction task is... The third interface sequence is The sample memory data at the target time step of the sample interaction task is The historical interaction actions executed at each historical time step before the target time step are: The sample interaction action at the target time step is The thought process of the sample interaction action is The sample action description of the sample interaction action The second sample interaction data is then represented as The above scheme predicts based on sample input data constructed from sample interaction tasks, third interface sequences, sample memory data at the target time step of executing the sample interaction task, and historical interaction actions executed at each historical time step before the target time step. This yields sample output data containing sample interaction actions at the target time step, sample thought processes for the sample interaction actions, and sample action descriptions for the sample interaction actions. This allows the construction of second sample interaction data for the interaction prediction model based on the sample input and sample output data. This ensures that the training data not only includes the interaction actions themselves but also the thought processes and action descriptions, thus more comprehensively reflecting the information during the interaction process. This helps the interaction prediction model learn richer interaction features during training, further improving the model's training effect and generalization ability.

[0061] In a specific implementation scenario, the historical interaction interface records the state changes of the interaction task within a specific time period. The third interface sequence includes: historical interaction interfaces triggered by several historical time steps before the target time step. Specifically, the number of historical time steps can be dynamically set according to actual needs. For example, the historical interaction interfaces triggered by the three historical time steps before the target time step can be selected to form the third interface sequence.

[0062] In a specific implementation scenario, the third interface sequence includes screenshots of historical interactive interfaces triggered several historical time steps before the target time step.

[0063] In a specific implementation scenario, the sample memory data at the target time step of the sample interaction task represents reusable data across time steps. Specifically, the sample memory data retains relevant information about the sample interaction task across time steps and can be used to store intermediate results, key fields, or user preferences, etc. The sample memory data can be updated with each time step. For example, if the sample interaction task is to "compare the price of item A in APP1 and APP2 and determine which APP has a cheaper price for item A", the sample memory data can record the price of item A in APP1 and the price of item A in APP2 at a specific time step. This information will be reused in subsequent time steps to support the completion of price comparison and judgment tasks. If the memory is not used, the sample memory data can be set to empty.

[0064] In a specific implementation scenario, the historical interaction actions executed at each historical time step prior to the target time step record the operation instructions and behavioral patterns of the interaction task at different stages. These historical interaction actions reflect the decision-making logic in past time steps, including the critical path and potential patterns during the execution of the interaction task. Specifically, historical interaction actions can be refined into the operation types of interface elements, such as clicks, swipes, and inputs, as well as the specific interface elements targeted by these operations, such as buttons, text boxes, and list items.

[0065] In a specific implementation scenario, interactive actions include a first type of interactive action that does not affect the interactive interface and a second type of interactive action that does affect the interactive interface. Specifically, the first type of interactive action is used for information output and internal state management (answering questions, writing to memory, etc.), which does not directly change the interface, but affects task completion and subsequent decisions.

[0066] In a specific implementation scenario, the first type of interactive action includes answer actions and memory actions. Specifically, answer actions include: answer content (which can be natural language text, a set of structured fields, or a combination of both), answer format description (e.g., plain text / JSON key-value pairs / tables, etc.), and a summary of the basis supporting the answer (e.g., key information from current / historical screenshots, referenced field names, etc.). Memory actions include write actions, update actions, delete actions, and read actions. Write actions include: memory key, memory value (which can be text or a structured object), and metadata (timestamp, source page, confidence level, TTL, task ID, etc.). Update actions, delete actions, and read actions are used to perform corresponding operations on write actions.

[0067] In a specific implementation scenario, the second type of interactive actions includes click actions, long press actions, input actions, scrolling / swiping actions, system navigation actions, etc. Click actions include screen coordinate click point, target element identifier, target element bounding box, and confidence level. Long press actions include click point and press duration. Input actions include input target (which can be coordinate point / element identifier / element bounding box), input string, and input mode (append / replace / clear before input, etc.). Scrolling / swiping actions include starting point... ,end The scroll container identifier and the container element bounding box are used to constrain the area where scrolling occurs. System navigation actions include returning, returning to the application home page, and opening a specified application.

[0068] In one implementation scenario, after obtaining the new first interface sequence, a validator is defined. Used to determine whether the sample interaction task is successfully completed. Specifically, the verifier outputs a binary judgment result to determine whether the sample interaction task is successfully completed, that is, a first result representing successful execution or a second result representing unsuccessful execution. It can also output a judgment result in the score range of [0, 1], which is not limited in this application.

[0069] In a specific implementation scenario, when it is determined that the sample interaction task has not been successfully executed, the action step that first caused the subsequent sample interaction task to fail or significantly deviate from the task objective in the first interface sequence of the sample interaction task is identified. The sample interaction action under the obtained action step is corrected, the correct sample interaction action that should be executed is provided, and the training target for identifying and reflecting on errors and correcting actions is provided for the sample interaction action in the next time step of the action step.

[0070] In a specific implementation scenario, based on a multimodal model, such as a large language model, prediction is performed on the sample input data to obtain the sample output data.

[0071] In a specific implementation scenario, prediction is performed based on sample input data to obtain sample interaction actions at the target time step. Specifically, a candidate set is obtained by sampling multiple candidates based on the operable elements of the sample interaction interface. This is used to subsequently filter out the most consistent and executable interactive actions with the sample interaction task, or to constrain the interactive action prediction process to only take values ​​on the element bounding box / candidate point of the operable element, that is, to programmatically map the action parameters to a certain element, predict the sample interactive action at the target time step, and based on the sample input data and the sample interactive action at the target time step, predict the sample thought process of the sample interactive action, the sample action description of the sample interactive action, and obtain the sample output data.

[0072] In another specific implementation scenario, the sample input data is processed based on a multimodal model, and the sample interaction actions, sample thought processes of the sample interaction actions, and sample action descriptions of the sample interaction actions at the target time step are directly output as sample output data.

[0073] In a specific implementation scenario, after obtaining the sample interaction actions, sample thought processes, and sample action descriptions for the target time step, the sample output data undergoes consistency screening and quality control. Specifically, it determines whether the sample interaction actions, sample thought processes, and sample action descriptions for the target time step are consistent, such as whether "the target click is the button / field described" or "the scrolling direction matches the intent." It also checks whether the sample interaction actions hit the interactive area of ​​the sample interaction interface, whether the expected state change occurs, and whether abnormal pop-ups appear.

[0074] In a specific implementation scenario, if the sample interaction actions at the target time step do not affect the interactive interface, after predicting based on the sample input data and obtaining the sample output data, an extended interaction action, different from the sample interaction action but not affecting the interactive interface, is constructed as a new sample interaction action based on historical interaction actions and the target interactive interface at the target time step. Prediction is then performed based on the target interactive interface, historical interaction actions, and the new sample interaction action to obtain an extended sample task as a new sample interaction task. Based on the new sample interaction task, the third interface sequence, the sample memory data at the target time step for executing the new sample interaction task, and historical interaction actions, new sample input data is obtained. Prediction is then performed based on the new sample input data to obtain new output data. The new output data includes the new sample interaction action, the sample thought process of the new sample interaction action, and the sample action description of the new sample interaction action. Based on the new sample input data and the new sample output data, a new second sample interaction data is constructed for the interaction prediction model. The above scheme constructs extended interactive actions, which are different from the sample interactive actions but do not affect the interactive interface, as new sample interactive actions based on historical interactive actions and the target interactive interface at the target time step. Based on the target interactive interface, historical interactive actions, and new sample interactive actions, prediction is performed to obtain extended sample tasks as new sample interactive tasks. New sample input data and new output data can be further constructed based on the new sample interactive tasks, thereby obtaining new second sample interactive data for the interactive prediction model. This can be used to expand the sample interactive data belonging to the interactive action type that does not affect the interactive interface, thereby improving the richness of the second sample interactive data. Especially when sample interactive data related to interactive actions that do not affect the interactive interface are difficult to obtain, it helps to improve the quality of interactive data construction.

[0075] In a specific implementation scenario, after constructing new second sample interaction data for the interaction prediction model based on new sample input data and new sample output data, in order to avoid noisy interaction samples such as those based on memorization or unfounded answers, a reasonableness judgment is made based on the constructed new second sample interaction data. For example, it is judged whether the sample thought process of the new sample interaction action in the new second sample interaction data, whether the sample action description of the new sample interaction action is consistent with the new sample interaction action, and whether there is obvious fabrication. Or, as mentioned above, when the sample interaction action at the target time step is a response action, it is judged whether the new sample interaction action can be supported by screenshots or historical context, or at least does not contradict them. When the sample interaction action at the target time step is a memory action, it is judged whether the key / value in the new sample interaction action is related to the progress of the task and whether it has reusable meaning.

[0076] In a specific implementation scenario, after constructing the second sample interaction data, the consistency pass rate of the second sample interaction data is obtained. For the second type of interaction action, sampling or full execution verification is performed to count the proportion of actions that are executable, hit the interactive area, and do not trigger anomalies. For the first type of interaction action, the rationality judgment pass rate is counted, and the passed samples are manually verified by sampling to calibrate the threshold.

[0077] In one implementation scenario, second sample interaction data is acquired for the interaction prediction model. This second sample interaction data includes sample input data and sample output data. The sample input data includes the sample interaction task, a third interface sequence, sample memory data at the target time step of executing the sample interaction task, and historical interaction actions executed at each historical time step before the target time step. The sample output data includes the sample interaction actions at the target time step, the sample thought process of the sample interaction actions, and the sample action description of the sample interaction actions. Error injection is performed based on the second sample interaction data to obtain third sample interaction data. Error injection includes replacing the sample interaction actions in the sample output data of the second sample interaction data with incorrect sample interaction actions. This approach, by injecting errors based on the second sample interaction data and replacing correct sample interaction actions with incorrect ones, can simulate possible errors that may occur during actual interaction, thereby constructing third sample interaction data containing erroneous samples. This not only enriches the diversity of training data but also enables the interaction prediction model to learn how to identify and correct erroneous interaction actions during training, helping to improve the robustness and accuracy of the interaction prediction model.

[0078] In a specific implementation scenario, error injection includes invalid action injection, which replaces the originally correct sample interaction action with an invalid action, such as clicking a non-interactive area, clicking an adjacent but unrelated element, or performing scrolling in a non-scrollable area. Specifically, it can be used to cover failure signals such as "no change in interface".

[0079] In a specific implementation scenario, error injection includes error target injection, which replaces the action target with a similar but incorrect element, such as clicking an adjacent button, selecting an error list item, or entering an error page. Specifically, it can be used to cover failure signals of the type "interface changes but deviates from the target".

[0080] In a specific implementation scenario, actions that trigger pop-ups or interface occlusion, or simulated return or jumps causing state shifts, are inserted into the second interface sequence in the first sample interaction data to cover failure signals of the "interrupted and rollback recovery" type.

[0081] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the interactive data construction method for the second sample in this application. Figure 3 As shown, the sample interaction task is The third interface sequence is The sample memory data at the target time step of the sample interaction task is The historical interaction actions executed at each historical time step before the target time step are: Optional inputs include the interface structure data X of the sample interactive interface and the set of operable elements of the sample interactive interface. The thought process of generating sample interactive actions Description of sample actions for interacting with samples Based on consistency detection with K operable actions from multiple candidate samples, specifically by checking whether the screenshots of the sample interaction interface match the interface structure data and whether the interaction action hits the interactive area, the second sample interaction data is obtained. For sample interaction actions that do not affect the user interface, based on the third interface sequence In addition to answering and remembering actions, extended interactive actions are generated as new sample interactive actions. The sample interactive task, the sample thought process of the sample interactive action, and the sample action description of the sample interactive action are then constructed in reverse for reasonableness assessment. The assessment result can be supported by evidence and is based on its relevance to the progress of the sample interactive task. This data is then sent to a quality control assessment device for quality inspection to obtain new second sample interactive data. .

[0082] The above scheme, in the interactive data construction stage, on the one hand, performs field masking based on the correlation between each data field in the interface structure data of the sample interactive interface and the sample interactive interface, and extracts the first similarity between the data feature representations based on the mask structure data of each sample interactive interface for deduplication. This can reduce the essentially the same but redundant collected data in the first interface sequence while retaining as many key samples as possible, based on the essential content of the sample interactive interface represented by the mask structure data feature representation, thus improving the quality of interactive data construction. On the other hand, by extracting the sequence combination of adjacent interactive interfaces and their triggering actions, the resulting second interface sequence, as well as the sample interactive task to which the second interface sequence originates from the first interface sequence, ensures the correlation of training samples in task processing logic and covers different interactive scenarios as much as possible, thus helping to improve the training effect of the interactive prediction model. Therefore, it can improve the quality of interactive data construction and help improve the training effect of the interactive prediction model.

[0083] Please see Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of the training method for the interactive prediction model of this application. Specifically, it may include the following steps: Step S21: Obtain the first sample interaction data of the interactive prediction model.

[0084] In this embodiment of the disclosure, the first sample interaction data includes: when performing a sample interaction task, the sample interaction interfaces of adjacent samples and the sample interaction actions between them, and the first sample interaction data is obtained by the interaction data construction method in any of the above embodiments.

[0085] It should be noted that the process of constructing the first sample interaction data can be referred to any of the aforementioned interaction data construction method embodiments, and will not be repeated here for the sake of brevity.

[0086] Step S22: Train the interaction prediction model based on the first sample interaction data.

[0087] In one implementation scenario, based on the first sample interaction data, pre-training sample data for the interaction prediction model is extracted. During the pre-training process: any sample interaction interface is used as input data, and the interface description text of the sample interaction interface is used as the training target; and / or, the previous sample interaction interface and sample interaction action in adjacent sample interaction interfaces are used as input data, and the interface description text of the next sample interaction interface in adjacent sample interaction interfaces is used as the training target. Based on the pre-training sample data, the interaction prediction model is pre-trained. Based on the first sample interaction data, the interaction prediction model is subjected to supervised training. During the supervised training process: the sample interaction task and the previous sample interaction interface in adjacent sample interaction interfaces are used as input data, and the sample interaction action is used as the training target. The above scheme, based on a two-stage training strategy of pre-training and supervised fine-tuning, can fully utilize the information in the first sample interaction data. In the pre-training stage, by using the sample interaction interface or its combination with previous interaction actions as input and the interface description text or subsequent interaction interface description as the target, the model is guided to learn the semantic representation of the interaction interface and the action association rules. In the supervised training stage, by using the sample interaction task and previous interface as input and the actual interaction action as the target, the model can further master the interaction decision-making ability under task guidance, which can improve the training accuracy of the interaction prediction model and help improve the prediction accuracy of the model in complex interaction tasks.

[0088] In a specific implementation scenario, during the pre-training process of the interaction prediction model, several final first interface sequences and several second interface sequences are extracted based on the first sample interaction data. The several final first interface sequences based on the sample interaction tasks are specifically represented as full-scene state data and are used to train the general knowledge ability of the interaction prediction model. The several second interface sequences are specifically represented as action transition data and are used to train the action prediction ability of the interaction prediction model.

[0089] In a specific implementation scenario, a training set for pre-training the interaction prediction model is composed of several final first interface sequences collected based on sample interaction tasks at a first preset ratio and several second interface sequences at a second preset ratio. Specifically, the sum of the first and second preset ratios is 1, and the first and second preset ratios are dynamically adjusted according to actual needs to balance the general knowledge and action prediction capabilities of the interaction prediction model. For example, when stronger general knowledge is required, the first preset ratio can be appropriately increased; when more accurate action prediction capabilities are required, the second preset ratio can be appropriately increased.

[0090] In a specific implementation scenario, during the pre-training process, the interaction prediction model, based on the sample interaction interface and operable elements given by the first sample interaction data, predicts the changes after executing the sample action corresponding to any operable element, such as page jump, pop-up appearance, entering input state, list scrolling, etc., or predicts the summary representation of the next sample interaction interface. It also determines whether the sample action falls within the interactive area and whether it is effective by giving the sample action parameters and the element information of the current sample interaction interface.

[0091] In a specific implementation scenario, based on the first sample interaction data, sample data is constructed in which the sample interaction interface does not change or changes in a way that does not meet expectations after the sample action is performed. The interaction prediction model is trained to determine that the action is invalid and outputs an explanation of the reason that is consistent with the evidence of the sample interaction interface, such as clicking a non-interactive element, scrolling to the bottom, the control being obscured, or not entering the input state. Based on the identification of invalid actions, the interaction prediction model is trained to further classify the error into several categories and output attribution text.

[0092] In a specific implementation scenario, given the first sample interaction data, the sample task, the historical interaction interface triggered by several historical time steps, and the historical interaction actions executed by each historical time step, when the previous action is invalid or causes deviation from the target of the sample task, the interaction prediction model is trained to generate reflection text and correction strategy description, and to give correction intentions, such as going back, closing the pop-up window, changing the click on another element, scrolling to the visible area and then clicking, etc. When the correction strategy includes a rollback operation, the interaction prediction model is further trained to summarize the previous error after rollback and propose alternative attempts to reduce repeated mistakes.

[0093] In a specific implementation scenario, after the interaction prediction model is pre-trained, indicator data representing the invalid action recognition capability, reflection text consistency, and correction strategy rationality are obtained respectively. The invalid action recognition capability is used to evaluate the accuracy of the interaction prediction model in distinguishing between invalid and valid actions on the invalid action sample set, so as to verify whether the interaction prediction model has a stable failure signal recognition capability. The reflection text consistency is used to evaluate whether the reflection text generated by the interaction prediction model is consistent with the interface evidence of the sample interaction interface, such as whether it can support the claimed buttons, pop-ups, scrolling to the bottom, etc. from the interface screenshots or interface structure data. The correction strategy rationality is used to evaluate whether the correction strategy generated by the interaction prediction model is reasonable and whether it can avoid repeating errors. When all the above indicator data meet the pre-training convergence requirements, the interaction prediction model after pre-training convergence is subjected to supervised training.

[0094] Please see Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of the interactive prediction model pre-training method in the training method of the interactive prediction model of this application. For example... Figure 5 As shown, based on the first sample interaction data, the full-scene state data and before-and-after transition data are extracted. The action result prediction and executability determination of the interaction prediction model are performed through the basic causal modeling task. The invalid action identification, error attribution, reflection and correction test strategy generation of the interaction prediction model are performed through the reflection and correction task. The training is rolled back and then tried again in the reflection and correction task, so that the interaction prediction model can identify failure signals, generate reflection and correction intentions after pre-training, and provide a better initialization interaction prediction model for subsequent supervised training and online reflection fine-tuning training.

[0095] In one implementation scenario, second and third sample interaction data are acquired. The second sample interaction data includes sample input data and sample output data. The sample input data includes the sample interaction task, the third interface sequence, sample memory data at the target time step of executing the sample interaction task, and the historical interaction actions executed at each historical time step before the target time step. The sample output data includes the sample interaction actions at the target time step, the sample thought process of the sample interaction actions, and the sample action description of the sample interaction actions. The third interface sequence includes the historical interaction interfaces triggered at several historical time steps before the target time step. The third sample interaction data is obtained by error injection based on the second sample interaction data. Error injection includes replacing the sample interaction actions in the sample output data of the second sample interaction data with incorrect sample interaction actions. The interaction prediction model is trained in a supervised manner based on the second sample interaction data and the interaction prediction model is trained in a supervised manner based on the third sample interaction data. In the supervised training process: the sample input data of the second sample interaction data is used as the input data, and the sample output data of the second sample interaction data is used as the training target. The third sample interaction data is used as the input data, and the sample interaction actions of the third sample data before error injection are used as the training target. The above approach combines second-sample interaction data containing correct interactive actions and their thought processes and action descriptions with third-sample interaction data containing incorrect interactive actions to conduct supervised training of the interaction prediction model. This enables the model to distinguish between correct and incorrect interactive actions while understanding the logic and intent behind the interactive actions. This enhances the accuracy of the interaction prediction model in predicting interactive actions and improves its ability to identify and correct potential errors during the interaction process. As a result, the robustness and generalization ability of the interaction prediction model are improved, especially when facing complex interaction scenarios.

[0096] It should be noted that the acquisition of the second and third sample interaction data can be referred to in the embodiments of any of the aforementioned interaction data construction methods, and will not be repeated here for the sake of brevity.

[0097] In a specific implementation scenario, the second and third sample interaction data are mixed and trained in a certain proportion. Specifically, this proportion is an adjustable hyperparameter. When the goal of the interaction prediction model emphasizes error correction and stability, the proportion of the third sample interaction data should be increased. When the goal of the interaction prediction model emphasizes the rapid acquisition of basic capabilities, the proportion of the second sample interaction data can be increased.

[0098] In a specific implementation scenario, when supervising the training of the interaction prediction model based on the second sample interaction data, the sample interaction task, the third interface sequence, the sample memory data at the target time step, and the historical interaction actions are used as input data. The interaction prediction model outputs the predicted interaction action at the target time step, the predicted thought process of the predicted interaction action, and the predicted action description of the predicted interaction action. The supervising training of the interaction prediction model is achieved based on the first loss between the sample interaction action and the predicted interaction action, the second loss between the predicted thought process and the sample thought process, and the third loss between the predicted action description and the sample action description.

[0099] In a specific implementation scenario, during supervised training of an interaction prediction model based on third-sample interaction data, the third-sample interaction data is labeled with the specific content of the error injection and the correct sample interaction data before the error injection. This third-sample interaction data is used as input data for the interaction prediction model. The interaction prediction model outputs the error cause of the third-sample interaction data, the corrective interaction action at the target time step, and the thought process behind the corrective interaction action, thus achieving reflective supervision. After an error occurs, the interaction prediction model is required to output reflective content consistent with the error injection content, explaining the error type and cause, such as "the wrong button was clicked," "the target control was obscured by a pop-up," or "the scrolling direction was incorrect or the scrolling reached the bottom." Based on the reflection, the interaction prediction model is required to output a correction strategy and the intention of the corrective action, such as closing the pop-up, going back one step, clicking the correct element, or scrolling to the visible area before clicking. This solution, by introducing a reflective supervision mechanism into the supervised training process, enables the interaction prediction model not only to identify errors but also to deeply understand the causes of errors and generate reasonable correction strategies and intentions accordingly, thereby improving the model's self-correction ability and decision-making level in complex interaction scenarios.

[0100] In a specific implementation scenario, after the interactive prediction model undergoes supervised training, it acquires metrics representing offline trajectory alignment, error correction capability, end-to-end success rate, and recovery rate. The offline trajectory alignment metric evaluates the degree to which the interactive prediction model matches the correct actions on the sample validation set, as well as the consistency pass rate of the sample thought process and sample action description, which is used to verify the end-to-end output alignment quality. The error correction capability metric evaluates whether the interactive prediction model can identify error types and provide reasonable correction strategies on the sample validation set containing error injection. Metrics such as reflective text consistency pass rate and correction strategy rationality pass rate can be used. The end-to-end success rate and recovery rate evaluate the task success rate in an offline playback or simulated execution environment, as well as the recovery rate of whether the task can be completed through correction after an error occurs.

[0101] Please see Figure 6 , Figure 6This is a schematic diagram illustrating the framework of an embodiment of supervised training of the interactive prediction model in the training method of the interactive prediction model of this application. For example... Figure 6 As shown, a successful trajectory library is obtained based on the second sample interaction data. Based on the segment selector, key trajectory segments are selected from the successful trajectories. Through the error injector, invalid actions, erroneous targets, and state disturbances are injected into the key trajectory segments. Through reflection and correction supervision, corrected interaction actions, the corrected thought process of corrected interaction actions, and the corrected action description of corrected interaction actions are generated. Based on consistency checks and feasibility verification, filtering is implemented. A sample database is constructed with the successful trajectories extracted based on the second sample interaction data. Through the training set mixer, the successful trajectories and the reflected trajectories after error injection are mixed in proportion to achieve supervised training of the interaction prediction model.

[0102] In a specific implementation scenario, after supervised training of the interaction prediction model, online reflection and fine-tuning are performed on the interaction prediction model to iteratively improve its reflection and error correction capabilities. In this case, the transfer contribution of the interaction prediction model to online training after supervised training can also be obtained. Under fixed online training settings, the efficiency, early success rate and stability differences of using the third sample interaction data in this stage and not using online samples are compared.

[0103] In a specific implementation scenario, during the online reflection and fine-tuning phase, an interactive prediction model trained using a system simulator or a cluster of real machines can be deployed. To improve concurrent data acquisition efficiency, a master-slave distributed architecture can be adopted, with slave machines running simulators to execute actions, and the master machine responsible for model inference and training computation. Online tasks can be instantiated from task templates and dynamic parameters. Specifically, online tasks can be categorized by complexity to facilitate the adoption of a course-based learning strategy.

[0104] In a specific implementation scenario, during the online reflection and fine-tuning phase, trajectory data generated by the interaction between the current interaction prediction model and the environment is collected online. Specifically, each trajectory data record at least the online task, the interaction interface and its structural information at the target time step, historical actions, and a triple consisting of the current output target interaction action, the target thought process of the target interaction action, and the target action description of the target interaction action. In addition, screenshots and state differences after the target interaction action is executed can also be obtained. The collected trajectory data is evaluated by a validator and divided into successful trajectories and failed trajectories. For successful trajectories, even if the trajectory is ultimately successful, there may be redundant steps or steps that are accidentally corrected after local errors. To reduce noise, the successive correctness checks are performed on successful trajectories, and only the steps judged as correct are retained for training. For failed trajectories, it is necessary to locate the first time step that leads to deviation from the task target or irreversible error. For the erroneous interaction action of the first erroneous step, the correct interaction action that should be executed under the same input conditions is constructed as the supervision target. For the next step after the first erroneous step, a correction sample containing reflection content is constructed. The interaction prediction model is required to clearly identify the previous error, explain the cause, and output the correction strategy and correction interaction action. When the error correction interaction action is a rollback operation, further rollback retry samples can be constructed to train the interaction prediction model to summarize errors and select alternative actions in the rollback state, thereby reducing repeated mistakes.

[0105] In a specific implementation scenario, after constructing sample data for the online reflection and fine-tuning phase, the consistency between the reflection text, action description, and grounding actions is checked, along with their alignment with observational evidence. The interactive actions are also examined to verify their execution within the environment or their adherence to rules to ensure they hit interactive areas and produce reasonable changes. Successfully filtered samples and error-corrected samples that pass quality checks are merged to form an online training set. This set is then used to fine-tune the current interaction prediction model, resulting in the next round of model development. Specifically, the number of iterations and the sample size per round are adjustable parameters that can be determined based on resources and objectives.

[0106] In a specific implementation scenario, to improve sample efficiency and stability, task sampling weight adjustment and course learning strategies can be adopted in the online reflection and fine-tuning stage. The sampling probability is adjusted according to the task success rate, so that difficult tasks can obtain a higher sampling ratio in subsequent iterations, thereby improving weak capabilities more quickly. Low-difficulty tasks are used for several rounds of iteration to stabilize the basic strategy, and then high-difficulty tasks are gradually introduced or only low-difficulty tasks with success rates below the threshold are retained for mixed training with high-difficulty tasks to continuously challenge the capability boundaries of the interactive prediction model.

[0107] In a specific implementation scenario, during the online reflection and fine-tuning phase, the success rate of various tasks after each iteration is statistically analyzed and trends are displayed according to difficulty levels. The proportion of interactive prediction models that can complete the task through reflection and correction after a first-time error is statistically analyzed to measure the ability to recover from errors. The proportion of successful location of the first-time error in the failure trajectory, the proportion of error-corrected samples that pass quality inspection, and the proportion of executable samples are statistically analyzed to measure the reliability of online data. The success rate improvement brought about by consuming a certain number of online interactive steps or a certain number of samples is evaluated to measure the efficiency of online training.

[0108] Please see Figure 7 , Figure 7 This is a schematic diagram of the framework of an embodiment of the online reflective fine-tuning of the interactive prediction model in the training method of the interactive prediction model of this application. Figure 7 As shown, the current interactive prediction model is deployed in an online execution environment such as a simulator / real device. Through task sampling and course learning, it executes predicted interactive actions. Based on the trajectory recorder's data, the model records sample interactive tasks, third-interface sequences, sample memory data at the target time step of the sample interactive task, historical interactive actions executed at each historical time step before the target time step, sample interactive actions, the thought process behind the sample interactive actions, and sample action descriptions. The system determines whether a task is successful through a validator, or by progressively determining whether each action in the task deviates from the task objective. A success trajectory filter removes noise from successful trajectories and stores them in a success sample library. For failure trajectories, it identifies the first time step that leads to deviation from the task objective or is irreversible. For the erroneous interaction action of the first erroneous step, an error correction sample constructor constructs the correct interaction action to be executed under the same input conditions as a supervision target. For the next step after the first erroneous step, it constructs a correction sample containing reflection content. The interaction prediction model is required to clearly identify the previous error, explain the cause, and output a correction strategy and corrective interaction action. Quality control filtering is implemented based on consistency checks and executability verification. An online training set is built for online fine-tuning and updating of the interaction prediction model.

[0109] The above scheme, during the training phase of the interaction prediction model, trains the interaction prediction model based on first sample interaction data, including the interaction interfaces of adjacent samples and the interaction actions between samples when performing sample interaction tasks. This allows the interaction prediction model to understand the position and role of interaction actions in the task flow based on the task processing logic in the first sample interaction data, thereby improving the model's prediction accuracy of interaction actions. Furthermore, since the first sample interaction data has undergone deduplication processing, removing redundant information and retaining key samples, it helps the interaction prediction model learn interaction features more efficiently during training, avoiding reduced training efficiency due to redundant data interference, thus improving the overall training effect of the interaction prediction model.

[0110] Please see Figure 8 , Figure 8 This is a flowchart illustrating an embodiment of the task execution method of this application. Specifically, it may include the following steps: Step S31: In response to the interactive task of the target application, obtain the current interactive interface of the target application.

[0111] In one implementation scenario, information about the current interactive interface can be captured by calling the interface provided by the target application or by using screenshot technology. This information includes, but is not limited to, the interface layout, control positions, and text content.

[0112] In a specific implementation scenario, the target application represents various applications that require human-computer interaction, such as office software, game applications, and online shopping platforms. The task to be interacted with represents the specific operation or goal expected to be completed in the target application, such as editing a document in office software, completing a level challenge in a game application, or searching for and purchasing specific products in an online shopping platform.

[0113] In a specific implementation scenario, the task to be interacted with needs to navigate between different target applications. For example, after completing document editing in office software, one might navigate to an email application to send the document as an attachment; or after searching for products on an online shopping platform, one might navigate to a payment application to complete the purchase process. To achieve this cross-application interaction, the task execution device needs to have cross-application interaction capabilities, able to identify the interaction logic and data flow requirements between different applications. Specifically, the device can automatically generate the sequence of interactive actions required to navigate between different applications by parsing key information in the task to be interacted with, such as the target product name and document title, combined with predefined cross-application interaction rules. In the current case, the current interactive interface of the target application may be an interface containing thumbnails of various target applications.

[0114] Step S32: Based on the interaction prediction model, predict the interaction task and the current interaction interface to obtain the predicted interaction action.

[0115] In this embodiment of the disclosure, the interactive prediction model is trained based on the embodiments of the training methods of any of the aforementioned interactive prediction models, and for the sake of brevity, it will not be described in detail here.

[0116] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an embodiment of the task execution method of this application, which deploys an interactive prediction model. Figure 9As shown, the first sample interaction data generated in the interaction data construction phase includes a full-scene state data set and a state transition data set, as well as a set of operable elements, structural information of each sample interaction interface, and screenshots of the sample interaction interfaces. Based on the full-scene state data set, second sample interaction data and the reverse construction of memory actions and response actions are generated. Quality checks are performed using consistency and executableness checks. Based on the filtered first and second sample interaction data, a three-stage training of the interaction prediction model is implemented. The pre-training stage injects reflective capabilities, the supervised training stage injects reflection and correction, and the online reflective fine-tuning stage achieves closed-loop iterative updates of the prediction interaction model. Finally, the trained end-to-end output interaction prediction model is deployed; specifically, the interaction prediction model is deployed in the target intelligent agent. The above-described interaction data construction method and interaction prediction model training method can be found in the detailed description of the foregoing embodiments, and will not be repeated here for brevity.

[0117] Step S33: Invoke the target agent to perform the predicted interaction action in the target application to obtain the new current interaction interface of the target application.

[0118] In one implementation scenario, the target intelligent agent is an entity capable of performing interactive actions, such as an automated script, a robot, or an intelligent software agent. After receiving the predicted interactive action, the target intelligent agent will interact with the target application based on the predicted interactive action, triggering a change in the application state and generating a new current interactive interface.

[0119] In one implementation scenario, the target agent inputs the task to be interacted with, a screenshot of the current interface, the interface structure data of the current interface, and the current memory data of the target agent executing the task to be interacted with at the current time step into the interaction prediction model as target input data. The target output data of the interaction prediction model is obtained. The output data includes the predicted interaction action executed by the target agent at the current time step of the target application, the predicted thought process of the predicted interaction action, and the description of the predicted interaction action.

[0120] In a specific implementation scenario, after invoking the target agent to perform a predicted interactive action in the target application and obtaining the new current interactive interface of the target application: the target input data and target output data of the interaction prediction model during this task execution are stored. The stored data is used to update the training dataset used to train the interaction prediction model, and / or, to allow the target object to recall the predicted thought process and interactive action description of the predicted interactive action during this task execution. The above scheme, by inputting the task to be interacted with, the current interface screenshot, interface structure data, and current memory data as target input data into the interaction prediction model, and obtaining target output data containing the predicted interactive action, predicted thought process, and interactive action description, stores the input and output data during the task execution process after executing the predicted interactive action. This can be used to update the training dataset to continuously improve model performance, or to provide the target object with the thought process and action description of the predicted interactive action, thereby enhancing the transparency and interpretability of task execution, helping to understand the model's decision-making basis and optimize subsequent interaction strategies.

[0121] Step S34: Return to the step of predicting the interactive task and the current interactive interface based on the interactive prediction model to obtain the predicted interactive action, until the interactive task is completed.

[0122] In one implementation scenario, during the return to the predicted execution step, the target AI continuously monitors the execution status of the task to be interacted with. Each time a predicted interaction action is completed and a new current interaction interface is obtained, the remaining portion of the task to be interacted with is reassessed to determine whether the prediction strategy or the sequence of interaction actions needs adjustment. For example, if unexpected interface changes or interaction obstacles are encountered during execution, the interaction prediction model will regenerate adaptive predicted interaction actions based on the new current interaction interface and task context.

[0123] In a specific implementation scenario, the target agent also supports dynamic task updates, allowing the target or parameters of the task to be interacted with to be modified during task execution based on user input or changes in the external environment, ensuring the flexibility and robustness of the interaction process. The above solution can efficiently and accurately complete complex interaction tasks, while adapting to diverse application scenarios and user needs.

[0124] The above-described scheme, during the task execution phase, leverages a trained interaction prediction model to progressively predict the interactive actions to be performed based on the target application's interactive task and the current interface. It then invokes the target agent to execute the predicted sequence of interactive actions step-by-step within the target application, automating the execution of the interactive task. This not only improves task execution efficiency but also reduces errors and biases that may arise from manual operation. Furthermore, because the interaction prediction model is trained using pre-constructed first-sample interaction data, its accuracy and robustness in predicting interactive actions are enhanced, enabling it to handle various complex situations that may arise during actual interaction and improving the smoothness and accuracy of task execution.

[0125] Please see Figure 10 , Figure 10 This is a schematic diagram of a framework of an embodiment of the interactive data construction device of this application. The interactive data construction device 100 includes: a sequence acquisition module 101, a field masking module 102, a deduplication module 103, a sequence segmentation module 104, and a sample construction module 105. The sequence acquisition module 101 is used to acquire a first interface sequence collected by the sample application when the sample application performs a sample interaction task; wherein, the first interface sequence includes several sample interaction interfaces, and adjacent sample interaction interfaces are triggered to jump by sample interaction actions; the field masking module 102 is used to perform field masking based on the correlation between each data field in the interface structure data of the sample interaction interface and the sample interaction interface, to obtain the mask structure data of the sample interaction interface; the deduplication module 103 is used to perform deduplication based on the mask structure data of each sample interaction interface. The first similarity between the extracted data features is used to select a first interactive interface pair and perform a deduplication operation to obtain a new first interface sequence. During the deduplication operation, one sample interactive interface in the first interactive interface pair is replaced by another sample interactive interface. The sequence segmentation module 104 is used to segment based on the new first interface sequence to obtain a second interface sequence. The second interface sequence contains at least one pair of adjacent sample interactive interfaces and the sample interactive actions between them. The sample construction module 105 is used to obtain the first sample interactive data of the interaction prediction model based on the second interface sequence and the sample interactive task to which the second interface sequence originates. The interaction prediction model is used to predict interactive actions based on the task to be interacted with.

[0126] The above scheme, in the interactive data construction stage, on the one hand, performs field masking based on the correlation between each data field in the interface structure data of the sample interactive interface and the sample interactive interface, and extracts the first similarity between the data feature representations based on the mask structure data of each sample interactive interface for deduplication. This can reduce the essentially the same but redundant collected data in the first interface sequence while retaining as many key samples as possible, based on the essential content of the sample interactive interface represented by the mask structure data feature representation, thus improving the quality of interactive data construction. On the other hand, by extracting the sequence combination of adjacent interactive interfaces and their triggering actions, the resulting second interface sequence, as well as the sample interactive task to which the second interface sequence originates from the first interface sequence, ensures the correlation of training samples in task processing logic and covers different interactive scenarios as much as possible, thus helping to improve the training effect of the interactive prediction model. Therefore, it can improve the quality of interactive data construction and help improve the training effect of the interactive prediction model.

[0127] In some disclosed embodiments, the deduplication module 103 further includes a first similarity detection module (not shown), used to detect whether the first similarity between different sample interaction interfaces meets the screening conditions; wherein, the screening conditions include: the first similarity is not less than a first preset threshold; the deduplication module 103 further includes a first response module (not shown), used to respond to the first similarity between different sample interaction interfaces not meeting the screening conditions, return to the step of detecting whether the similarity between different sample interaction interfaces meets the screening conditions, until the similarity between any two sample interaction interfaces has been detected; the deduplication module 103 further includes a second response module (not shown), used to respond to the first similarity between different sample interaction interfaces meeting the screening conditions, select the corresponding sample interaction interface as the first interaction interface pair, and replace one of the sample interaction interfaces in the first interaction interface pair with the other sample interaction interface in the first interaction interface pair.

[0128] Therefore, if the first similarity between different sample interfaces does not meet the screening criteria, it indicates that the two sample interfaces being compared are essentially different. In this case, there is no need to perform deduplication, and the similarity detection of other sample interfaces continues until the comparison of all sample interfaces is completed. However, when the first similarity between different sample interfaces meets the screening criteria, it means that the two sample interfaces have a high degree of similarity in their essential content. At this time, selecting one sample interface to replace the other can effectively reduce redundant data while retaining key sample information, thereby improving data quality while ensuring data diversity.

[0129] In some disclosed embodiments, before dividing based on the new first interface sequence to obtain the second interface sequence, the interactive data construction device 100 further includes an image feature extraction module (not shown), used to extract features based on each sample interactive interface to obtain image feature representations of each sample interactive interface; the interactive data construction device 100 also includes a second similarity detection module (not shown), used to select a second interactive interface pair based on the second similarity between the image feature representations of each sample interactive interface to perform a deduplication operation to obtain a new first interface sequence; wherein, during the deduplication operation, one sample interactive interface in the second interactive interface pair is replaced by another sample interactive interface.

[0130] Therefore, based on the second similarity between the image feature representations of each sample interaction interface, the second interaction interface is selected to perform deduplication. This can reduce the essentially identical but redundant collected data in the first interface sequence in the image visual dimension, thereby further improving the quality of the interaction data construction and helping to reduce the interference of redundant data on model training.

[0131] In some disclosed embodiments, the interactive data construction device 100 further includes a sample input data acquisition module (not shown), used to obtain sample input data based on the sample interaction task, the third interface sequence, sample memory data at the target time step of executing the sample interaction task, and the historical interaction actions executed at each historical time step before the target time step; wherein, the sample memory data represents cross-step reusable data, and the third interface sequence includes: historical interaction interfaces triggered at several historical time steps before the target time step; the interactive data construction device 100 further includes a sample output data prediction module (not shown), used to predict based on the sample input data to obtain sample output data; wherein, the sample output data includes: sample interaction actions at the target time step, sample thought processes of sample interaction actions, and sample action descriptions of sample interaction actions, the sample input data serves as input data for the interactive prediction model training stage, and the sample output data serves as the training target for the interactive prediction model training stage; the interactive data construction device 100 further includes a second sample construction module (not shown), used to construct the second sample interaction data of the interactive prediction model based on the sample input data and sample output data.

[0132] Therefore, by predicting the sample input data constructed from the sample interaction task, the third interface sequence, the sample memory data at the target time step of the sample interaction task, and the historical interaction actions executed at each historical time step before the target time step, we can obtain sample output data containing the sample interaction action at the target time step, the sample thought process of the sample interaction action, and the sample action description of the sample interaction action. This allows us to construct the second sample interaction data for the interaction prediction model based on the sample input data and sample output data. This ensures that the training data not only includes the interaction action itself but also covers the thought process and action description of the interaction action, thus reflecting the information in the interaction process more comprehensively. This helps the interaction prediction model learn richer interaction features during training, further improving the model's training effect and generalization ability.

[0133] In some disclosed embodiments, when the sample interaction action at the target time step does not affect the interactive interface, after predicting based on the sample input data to obtain sample output data, the interaction data construction device 100 further includes an interaction action construction module (not shown), used to construct an extended interaction action that is different from the sample interaction action but does not affect the interactive interface, based on historical interaction actions and the target interactive interface at the target time step, as a new sample interaction action; the interaction data construction device 100 also includes an update extension (not shown), used to predict based on the target interactive interface, historical interaction actions, and the new sample interaction action to obtain an extended sample task as a new sample interaction task; the interaction data construction device 100 also includes an update acquisition module. The interactive data construction device 100 includes a block (not shown) for obtaining new sample input data based on the new sample interaction task, the third interface sequence, sample memory data at the target time step of executing the new sample interaction task, and historical interaction actions; the interactive data construction device 100 also includes a sample update prediction module (not shown) for making predictions based on the new sample input data to obtain new output data; wherein, the new output data includes: new sample interaction actions, sample thinking process of new sample interaction actions, and sample action description of new sample interaction actions; the interactive data construction device 100 also includes an update construction module (not shown) for constructing new second sample interaction data for the interactive prediction model based on the new sample input data and the new sample output data.

[0134] Therefore, based on historical interaction actions and the target interaction interface at the target time step, extended interaction actions that differ from sample interaction actions but do not affect the interaction interface are constructed as new sample interaction actions. Based on the target interaction interface, historical interaction actions, and new sample interaction actions, prediction is performed to obtain extended sample tasks as new sample interaction tasks. New sample input data and new output data can be further constructed based on the new sample interaction tasks, thereby obtaining new second sample interaction data for the interaction prediction model. This can be used to expand the sample interaction data belonging to the interaction action type that does not affect the interaction interface, thereby improving the richness of the second sample interaction data. Especially when sample interaction data related to interaction actions that do not affect the interaction interface are difficult to obtain, it helps to improve the construction quality of interaction data.

[0135] In some disclosed embodiments, the sequence acquisition module 101 further includes an execution interface acquisition module (not shown), used to acquire the sample interaction interface of the sample application when it starts executing the sample interaction task, as the first sample interaction interface in the first interface sequence; the sequence acquisition module 101 further includes an element parsing module (not shown), used to parse the interface structure data of the latest sample interaction interface in the first interface sequence to obtain a set of operable elements of the latest sample interaction interface in the first interface sequence; the sequence acquisition module 101 further includes an element selection module (not shown), used to select operable elements related to the sample interaction task from the set of operable elements as target operable elements; wherein, the interaction action performed on the target operable element is a sample interaction action; the sequence acquisition module 101 further includes an interface jump module (not shown), used to execute the latest sample interaction action based on the latest sample interaction interface in the first interface sequence to obtain a new jump interaction interface of the sample application as a new sample interaction interface in the first interface sequence, and return the step of parsing the interface structure data of the latest sample interaction interface in the first interface sequence to obtain a set of operable elements of the latest sample interaction interface in the first interface sequence, until the sample interaction task is completed.

[0136] Therefore, by obtaining the initial sample interaction interface of the sample application when performing the sample interaction task as the starting point of the first interface sequence, and continuously parsing the interface structure data of the latest sample interaction interface to locate the operable elements related to the task, thereby triggering the corresponding sample interaction action and updating the interaction interface until the task is completed, a first interface sequence containing several interaction scenarios and task logic can be automatically constructed. This not only provides the richest possible basic data for subsequent data deduplication and feature extraction, but also ensures the continuity and diversity of training samples in the task processing flow, thereby further improving the quality and efficiency of interaction data construction.

[0137] In some disclosed embodiments, the interactive data construction device 100 further includes a sample interactive data acquisition module (not shown), used to acquire second sample interactive data of the interactive prediction model; wherein, the second sample interactive data includes sample input data and sample output data, the sample input data includes sample interactive tasks, a third interface sequence, sample memory data at the target time step of executing the sample interactive tasks, and historical interactive actions executed at each historical time step before the target time step, and the sample output data includes sample interactive actions at the target time step, sample thought processes of sample interactive actions, and sample action descriptions of sample interactive actions; the interactive data construction device 100 further includes an error injection module (not shown), used to perform error injection based on the second sample interactive data to obtain third sample interactive data; wherein, error injection includes: replacing sample interactive actions in the sample output data of the second sample interactive data with erroneous sample interactive actions.

[0138] Therefore, by injecting errors based on the second sample interaction data and replacing the correct sample interaction actions with incorrect sample interaction actions, it is possible to simulate the error situations that may occur in the actual interaction process, thereby constructing the third sample interaction data containing error samples. This not only enriches the diversity of training data, but also enables the interaction prediction model to learn how to identify and correct erroneous interaction actions during the training process, which helps to improve the robustness and accuracy of the interaction prediction model.

[0139] In some disclosed embodiments, after selecting a first interactive interface pair based on the first similarity between the data feature representations extracted from the mask structure data of each sample interactive interface to perform deduplication and obtain a new first interface sequence, and before dividing based on the new first interface sequence to obtain a second interface sequence, the interactive data construction device 100 further includes an index score acquisition module (not shown), used to acquire the target index score of the new first interface sequence; wherein, the target index score represents the first index score of the data coverage of the new first interface sequence and the second index score of the data deduplication effect of the new first interface sequence; the interactive data construction device 100 further includes a score judgment module (not shown), used to re-execute the step of selecting a first interactive interface pair based on the first similarity between the data feature representations extracted from the mask structure data of each sample interactive interface to perform deduplication and obtain a new first interface sequence in response to the target index score not meeting the preset conditions; wherein, the preset conditions include: the first index score is not greater than a second preset threshold and the second index score is not greater than a third preset threshold.

[0140] Therefore, by obtaining the target index score of the new first interface sequence, which includes the first index score of data coverage and the second index score of data deduplication effect, the impact of deduplication operation on data quality can be comprehensively evaluated. If the target index score does not meet the preset conditions, that is, the data coverage or deduplication effect does not meet the predetermined threshold, the process of re-executing the deduplication operation is triggered, ensuring that the deduplication process can not only effectively reduce redundant data, but also maintain the broad coverage of data, thereby improving the quality of interactive data construction while ensuring the comprehensiveness and diversity of data.

[0141] Please see Figure 11 , Figure 11 This is a schematic diagram of the framework of an embodiment of the training device for the interaction model of this application. The training device 110 for the interaction model includes: a sample acquisition module 111 and a model training module 112. The sample acquisition module 111 is used to acquire first sample interaction data of the interaction prediction model; wherein, the first sample interaction data includes: adjacent sample interaction interfaces and sample interaction actions between them when performing sample interaction tasks, and the first sample interaction data is obtained by the interaction data construction device 100 in any of the above embodiments; the model training module 112 is used to train the interaction prediction model based on the first sample interaction data.

[0142] The above scheme, during the training phase of the interaction prediction model, trains the interaction prediction model based on first sample interaction data, including the interaction interfaces of adjacent samples and the interaction actions between samples when performing sample interaction tasks. This allows the interaction prediction model to understand the position and role of interaction actions in the task flow based on the task processing logic in the first sample interaction data, thereby improving the model's prediction accuracy of interaction actions. Furthermore, since the first sample interaction data has undergone deduplication processing, removing redundant information and retaining key samples, it helps the interaction prediction model learn interaction features more efficiently during training, avoiding reduced training efficiency due to redundant data interference, thus improving the overall training effect of the interaction prediction model.

[0143] In some disclosed embodiments, the model training module 112 further includes a sample extraction module (not shown), used to extract pre-training sample data for the interaction prediction model based on the first sample interaction data; wherein, during the pre-training process: any sample interaction interface is used as input data, and the interface description text of the sample interaction interface is used as the training target; and / or, the previous sample interaction interface and sample interaction action in adjacent sample interaction interfaces are used as input data, and the interface description text of the next sample interaction interface in adjacent sample interaction interfaces is used as the training target; the model training module 112 further includes a pre-training module (not shown), used to pre-train the interaction prediction model based on the pre-training sample data; the model training module 112 further includes a supervised training module (not shown), used to perform supervised training on the interaction prediction model based on the first sample interaction data; wherein, during the supervised training process: the sample interaction task and the previous sample interaction interface in adjacent sample interaction interfaces are used as input data, and the sample interaction action is used as the training target.

[0144] Therefore, the two-stage training strategy based on pre-training and supervised fine-tuning can fully utilize the information in the first sample interaction data. In the pre-training stage, by using the sample interaction interface or its combination with the preceding interaction action as input and the interface description text or subsequent interaction interface description as the target, the model is guided to learn the semantic representation of the interaction interface and the action association rules. In the supervised training stage, by using the sample interaction task and the preceding interface as input and the actual interaction action as the target, the model can further master the interaction decision-making ability under task guidance, which can improve the training accuracy of the interaction prediction model and help improve the prediction accuracy of the model in complex interaction tasks.

[0145] In some disclosed embodiments, the training device 110 of the interaction model further includes a sample interaction data acquisition module (not shown), used to acquire second sample interaction data and third sample interaction data; wherein, the second sample interaction data includes: sample input data and sample output data, the sample input data includes sample interaction tasks, a third interface sequence, sample memory data at the target time step of executing the sample interaction task, and historical interaction actions executed at each historical time step before the target time step, the sample output data includes sample interaction actions at the target time step, sample thought processes of sample interaction actions, and sample action descriptions of sample interaction actions, the third interface sequence includes historical interaction interfaces triggered at several historical time steps before the target time step, and the third sample interaction... The data is obtained by injecting errors into the second sample interaction data. Error injection includes replacing the sample interaction actions in the sample output data of the second sample interaction data with incorrect sample interaction actions. The training device 110 of the interaction model also includes a supervised training submodule (not shown) for supervised training of the interaction prediction model based on the second sample interaction data and supervised training of the interaction prediction model based on the third sample interaction data. In the supervised training process: the sample input data of the second sample interaction data is used as the input data and the sample output data of the second sample interaction data is used as the training target. The third sample interaction data is used as the input data and the sample interaction actions of the third sample data before error injection are used as the training target.

[0146] Therefore, by combining second-sample interaction data containing correct interactive actions and their thought processes and action descriptions with third-sample interaction data containing incorrect interactive actions, supervised training of the interaction prediction model enables the model to distinguish between correct and incorrect interactive actions while understanding the logic and intent behind the interactive actions. This enhances the accuracy of the interaction prediction model in predicting interactive actions and improves its ability to identify and correct potential errors during the interaction process, thereby improving the robustness and generalization ability of the interaction prediction model, especially when facing complex interaction scenarios.

[0147] Please see Figure 12 , Figure 12This is a schematic diagram of the framework of an embodiment of the task execution device of this application. The task execution device 120 includes: an interface acquisition module 121, an action prediction module 122, an action execution module 123, and an iterative execution module 124. The interface acquisition module 121 is used to acquire the current interactive interface of the target application in response to the interactive task to be interacted with by the target application. The action prediction module 122 is used to predict the interactive action based on the interactive prediction model and the current interactive interface. The interactive prediction model is trained based on the training device 110 of the interactive prediction model in any of the above embodiments. The action execution module 123 is used to call the target agent to execute the predicted interactive action in the target application to obtain a new current interactive interface of the target application. The iterative execution module 124 is used to return to the step of predicting the interactive action based on the interactive prediction model and the current interactive interface until the interactive task to be interacted with is completed.

[0148] The above-described scheme, during the task execution phase, leverages a trained interaction prediction model to progressively predict the interactive actions to be performed based on the target application's interactive task and the current interface. It then invokes the target agent to execute the predicted sequence of interactive actions step-by-step within the target application, automating the execution of the interactive task. This not only improves task execution efficiency but also reduces errors and biases that may arise from manual operation. Furthermore, because the interaction prediction model is trained using pre-constructed first-sample interaction data, its accuracy and robustness in predicting interactive actions are enhanced, enabling it to handle various complex situations that may arise during actual interaction and improving the smoothness and accuracy of task execution.

[0149] In some disclosed embodiments, the action prediction module 122 further includes a data input module (not shown), used by the target agent to input the task to be interacted with, a screenshot of the current interface of the current interactive interface, the interface structure data of the current interface, and the current memory data of the target agent executing the task to be interacted with at the current time step as target input data to the interaction prediction model; the action prediction module 122 further includes a data output module (not shown), used to obtain the target output data of the interaction prediction model; wherein, the output data includes the predicted interactive action executed by the target agent at the current time step of the target application, the predicted thought process of the predicted interactive action, and the interactive action description of the predicted interactive action; after calling the target agent to execute the predicted interactive action in the target application to obtain the new current interactive interface of the target application, the task execution device 120 further includes a data storage module, used to store the target input data and target output data of the interaction prediction model during the current task execution process; wherein, the stored data is used to: update the training dataset used to train the interaction prediction model, and / or, so that the target object can call the predicted thought process of the predicted interactive action and the interactive action description of the predicted interactive action during the current task execution process.

[0150] Therefore, by inputting the task to be interacted with, the current interface screenshot, interface structure data, and current memory data into the interaction prediction model as target input data, and obtaining target output data containing predicted interaction actions, predicted thought processes, and descriptions of interaction actions, the input and output data during the task execution process can be stored after the predicted interaction actions are executed. This data can be used to update the training dataset to continuously improve model performance, or to provide the target object with the thought process and action description of the predicted interaction actions, thereby enhancing the transparency and interpretability of task execution, helping to understand the model's decision-making basis and optimize subsequent interaction strategies.

[0151] Please see Figure 13 , Figure 13 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 130 includes at least a memory 131 and a processor 132 coupled to each other. The memory 131 stores at least program instructions, and the processor 132 is used to execute the program instructions to implement the steps in any of the above-described method embodiments for acquiring drawing data, or to implement the steps in any of the above-described data conversion method embodiments. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 130 may include, but is not limited to, devices such as laptops and personal computers; the specific type of the electronic device 130 is not limited here.

[0152] Specifically, processor 132 controls itself and memory 131 to implement the steps in any of the above-described method embodiments for acquiring drawing data, or to implement the steps in any of the above-described data conversion method embodiments. Processor 132 may also be referred to as a CPU (Central Processing Unit). Processor 132 may be an integrated circuit chip with signal processing capabilities. Processor 132 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 132 may be implemented using integrated circuit chips.

[0153] In the above scheme, during the interactive data construction phase, the electronic device 130 performs field masking based on the correlation between each data field in the interface structure data of the sample interactive interface and the sample interactive interface. It then extracts the first similarity between the data feature representations of each sample interactive interface based on the mask structure data for deduplication. This allows for the reduction of essentially identical but redundant collected data in the first interface sequence while preserving key samples as much as possible, based on the essential content of the sample interactive interface represented by the mask structure data feature representation, thus improving the quality of interactive data construction. On the other hand, by extracting the sequence combination of adjacent interactive interfaces and their triggering actions, a second interface sequence is obtained, along with the sample interactive task from which the second interface sequence originates. This ensures the correlation of training samples in task processing logic and covers different interactive scenarios as much as possible, thus helping to improve the training effect of the interactive prediction model. During the interactive prediction model training phase, the interactive prediction model is trained based on the first sample interactive data, including the interaction actions between adjacent sample interactive interfaces when executing sample interactive tasks. This allows for the training of the interactive prediction model based on the first sample interactive data, including the interaction actions between adjacent sample interactive interfaces when executing sample interactive tasks. The task processing logic in the first sample interaction data enables the interaction prediction model to understand the position and role of interactive actions in the task flow, thereby improving the model's prediction accuracy of interactive actions. Furthermore, since the first sample interaction data has undergone deduplication, removing redundant information and retaining key samples, it helps the interaction prediction model learn interaction features more efficiently during training, avoiding reduced training efficiency due to redundant data interference, thus improving the overall training effect of the interaction prediction model. In the task execution phase, based on the trained interaction prediction model, it can gradually predict the interactive actions to be executed according to the target application's interactive task and the current interactive interface, and call the target agent to gradually execute the predicted interactive action sequence in the target application, realizing the automated execution of the interactive task. This not only improves the efficiency of task execution but also reduces errors and deviations that may be caused by manual operation. Since the interaction prediction model is trained with pre-constructed first sample interaction data, it improves the accuracy and robustness of the interaction prediction model in predicting interactive actions, enabling it to cope with various complex situations that may occur in actual interaction, and improving the smoothness and accuracy of task execution.

[0154] Please see Figure 14 , Figure 14 This is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of this application. The computer-readable storage medium 140 stores program instructions 141 that can be executed by a processor. The program instructions 141 are used to implement the steps in the above-described method embodiment for acquiring drawing data, or to implement the steps in the above-described data conversion method embodiment.

[0155] In the above scheme, during the interactive data construction phase, the computer-readable storage medium 140 performs field masking based on the correlation between each data field in the interface structure data of the sample interactive interface and the sample interactive interface, and extracts the first similarity between the data feature representations based on the mask structure data of each sample interactive interface for deduplication. This allows for the reduction of essentially identical but redundant collected data in the first interface sequence while retaining key samples as much as possible, based on the essential content of the sample interactive interface represented by the mask structure data feature representation, thus improving the construction quality of the interactive data. On the other hand, by extracting the sequence combination of adjacent interactive interfaces and their triggering actions, the resulting second interface sequence, and the sample interactive task to which the second interface sequence originates from the first interface sequence, ensures the correlation of training samples in task processing logic and covers different interactive scenarios as much as possible, thus helping to improve the training effect of the interactive prediction model. During the interactive prediction model training phase, the interactive prediction model is trained based on the first sample interactive data, including the interaction actions between adjacent sample interactive interfaces when executing sample interactive tasks. Based on the task processing logic in the first sample interaction data, the interaction prediction model can understand the position and role of interactive actions in the task flow, thereby improving the model's prediction accuracy of interactive actions. Furthermore, since the first sample interaction data has undergone deduplication, removing redundant information and retaining key samples, it helps the interaction prediction model learn interaction features more efficiently during training, avoiding reduced training efficiency due to redundant data interference, thus improving the overall training effect of the interaction prediction model. In the task execution phase, based on the trained interaction prediction model, it can gradually predict the interactive actions to be executed according to the target application's interactive task and the current interactive interface, and call the target agent to gradually execute the predicted interactive action sequence in the target application, realizing the automated execution of the interactive task. This not only improves the efficiency of task execution but also reduces errors and deviations that may be caused by manual operation. Since the interaction prediction model is trained with pre-constructed first sample interaction data, it improves the accuracy and robustness of the interaction prediction model in predicting interactive actions, enabling it to cope with various complex situations that may occur in actual interaction, and improving the smoothness and accuracy of task execution.

[0156] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0157] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0160] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0161] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for constructing interactive data, characterized in that, include: The first interface sequence collected by the sample application when it performs a sample interaction task is obtained; wherein, the first interface sequence includes a number of sample interaction interfaces, and adjacent sample interaction interfaces are switched by sample interaction actions. Based on the correlation between each data field in the interface structure data of the sample interactive interface and the sample interactive interface, field masking is performed to obtain the mask structure data of the sample interactive interface. Based on the first similarity between the data feature representations extracted from the mask structure data of each of the sample interaction interfaces, a first interaction interface pair is selected to perform a deduplication operation to obtain a new first interface sequence; wherein, during the deduplication operation, one of the sample interaction interfaces in the first interaction interface pair is replaced by the other sample interaction interface. Based on the new first interface sequence, a second interface sequence is obtained; wherein, the second interface sequence includes at least one pair of adjacent sample interaction interfaces and the sample interaction actions between them; Based on the second interface sequence and the sample interaction task to which the first interface sequence from which the second interface sequence originates, the first sample interaction data of the interaction prediction model is obtained; wherein, the interaction prediction model is used to predict the interaction action based on the task to be interacted with.

2. The method according to claim 1, characterized in that, The step of selecting a first interactive interface pair based on the first similarity between the data feature representations extracted from the mask structure data of each of the sample interactive interfaces, performing a deduplication operation, and obtaining a new first interface sequence includes: Detect whether the first similarity between different sample interaction interfaces meets the screening conditions; wherein, the screening conditions include: the first similarity is not less than a first preset threshold; In response to the first similarity between different sample interaction interfaces not meeting the filtering condition, the process returns to the step of detecting whether the similarity between different sample interaction interfaces meets the filtering condition, until the similarity between any two sample interaction interfaces has been detected. In response to the first similarity between different sample interaction interfaces satisfying the filtering condition, the corresponding sample interaction interface is selected as the first interaction interface pair, and one of the sample interaction interfaces in the first interaction interface pair is replaced with the other sample interaction interface in the first interaction interface pair.

3. The method according to claim 1 or 2, characterized in that, Before dividing the second interface sequence based on the new first interface sequence, the method further includes: Feature extraction is performed on each of the sample interaction interfaces to obtain the image feature representation of each of the sample interaction interfaces; Based on the second similarity between the image feature representations of each of the sample interaction interfaces, a second interaction interface pair is selected to perform a deduplication operation to obtain a new first interface sequence; wherein, during the deduplication operation, one of the sample interaction interfaces in the second interaction interface pair is replaced by the other sample interaction interface.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the sample interaction task, the third interface sequence, the sample memory data at the target time step of executing the sample interaction task, and the historical interaction actions executed at each historical time step before the target time step, sample input data is obtained; wherein, the sample memory data represents cross-step reusable data, and the third interface sequence includes: historical interaction interfaces triggered at several historical time steps before the target time step. Based on the sample input data, a prediction is made to obtain sample output data; wherein, the sample output data includes: the sample interaction action at the target time step, the sample thought process of the sample interaction action, and the sample action description of the sample interaction action; the sample input data serves as the input data for the training phase of the interaction prediction model, and the sample output data serves as the training target for the training phase of the interaction prediction model. Based on the sample input data and the sample output data, the second sample interaction data of the interaction prediction model is constructed.

5. The method according to claim 4, characterized in that, If the sample interaction action at the target time step does not affect the interactive interface, after predicting based on the sample input data to obtain the sample output data, the method further includes: Based on the historical interaction actions and the target interaction interface at the target time step, an extended interaction action that is different from the sample interaction action but does not affect the interaction interface is constructed as a new sample interaction action. Based on the target interactive interface, the historical interactive actions, and the new sample interactive actions, predictions are made to obtain an extended sample task as a new sample interactive task. Based on the new sample interaction task, the third interface sequence, the sample memory data at the target time step of executing the new sample interaction task, and the historical interaction actions, new sample input data is obtained. Based on the new sample input data, prediction is performed to obtain new output data; wherein, the new output data includes: the new sample interaction action, the sample thought process of the new sample interaction action, and the sample action description of the new sample interaction action. Based on the new sample input data and the new sample output data, a new second sample interaction data is constructed for the interaction prediction model.

6. The method according to any one of claims 1 to 5, characterized in that, The first interface sequence collected by the sample application when performing the sample interaction task includes: The sample interaction interface of the sample application when it starts executing the sample interaction task is obtained, and it is used as the first sample interaction interface in the first interface sequence. Based on the interface structure data of the latest sample interaction interface in the first interface sequence, the set of operable elements of the latest sample interaction interface in the first interface sequence is obtained by parsing. Select an operable element from the set of operable elements that is related to the sample interaction task as the target operable element; wherein, the interaction action performed on the target operable element is the sample interaction action; The steps of executing the latest sample interaction action based on the latest sample interaction interface in the first interface sequence, obtaining the new jump interaction interface of the sample application as the new sample interaction interface in the first interface sequence, and returning the interface structure data based on the latest sample interaction interface in the first interface sequence to parse and obtain the set of operable elements of the latest sample interaction interface in the first interface sequence, continue until the sample interaction task is completed.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain the second sample interaction data of the interaction prediction model; wherein, the second sample interaction data includes sample input data and sample output data, the sample input data includes the sample interaction task, the third interface sequence, the sample memory data at the target time step of executing the sample interaction task, and the historical interaction actions executed at each historical time step before the target time step, and the sample output data includes the sample interaction action at the target time step, the sample thought process of the sample interaction action, and the sample action description of the sample interaction action; Error injection is performed based on the second sample interaction data to obtain the third sample interaction data; wherein, the error injection includes: replacing the sample interaction action in the sample output data of the second sample interaction data with an incorrect sample interaction action.

8. The method according to any one of claims 1 to 7, characterized in that, After selecting a first interactive interface pair based on the first similarity between the data feature representations extracted from the mask structure data of each of the sample interactive interfaces, performing a deduplication operation to obtain a new first interface sequence, and before dividing based on the new first interface sequence to obtain a second interface sequence, the method further includes: Obtain the target index score of the new first interface sequence; wherein, the target index score is characterized by a first index score of the data coverage of the new first interface sequence and a second index score of the data deduplication effect of the new first interface sequence. In response to the target index score not meeting the preset conditions, the first similarity between the data feature representations extracted from the mask structure data of each of the sample interaction interfaces is re-executed, and the first interaction interface pair is selected to perform deduplication operation to obtain a new first interface sequence; wherein, the preset conditions include: the first index score is not greater than the second preset threshold and the second index score is not greater than the third preset threshold.

9. A training method for an interactive prediction model, characterized in that, include: Obtain first sample interaction data of the interaction prediction model; wherein, the first sample interaction data includes: the sample interaction actions between adjacent sample interaction interfaces and between them when the sample interaction task is executed, and the first sample interaction data is obtained by the interaction data construction method according to any one of claims 1 to 8; The interaction prediction model is trained based on the first sample interaction data.

10. The method according to claim 9, characterized in that, The step of training the interaction prediction model based on the first sample interaction data includes: Based on the first sample interaction data, pre-training sample data of the interaction prediction model is extracted; wherein, in the pre-training process: any sample interaction interface is used as input data and the interface description text of the sample interaction interface is used as the training target; and / or, the previous sample interaction interface and the sample interaction action in adjacent sample interaction interfaces are used as input data and the interface description text of the next sample interaction interface in adjacent sample interaction interfaces is used as the training target. The interaction prediction model is pre-trained based on the pre-trained sample data; Based on the first sample interaction data, the interaction prediction model is trained in a supervised manner; wherein, during the supervised training process: the sample interaction task and the previous sample interaction interface in the adjacent sample interaction interface are used as input data, and the sample interaction action is used as the training target.

11. The method according to claim 9 or 10, characterized in that, The method further includes: Acquire second sample interaction data and third sample interaction data; wherein, the second sample interaction data includes: sample input data and sample output data, the sample input data includes the sample interaction task, the third interface sequence, sample memory data at the target time step of executing the sample interaction task, and historical interaction actions executed at each historical time step before the target time step, the sample output data includes the sample interaction action at the target time step, the sample thought process of the sample interaction action, and the sample action description of the sample interaction action, the third interface sequence includes historical interaction interfaces triggered at several historical time steps before the target time step, and the third sample interaction data is obtained by error injection based on the second sample interaction data, the error injection including replacing the sample interaction action in the sample output data of the second sample interaction data with an incorrect sample interaction action; Based on the second sample interaction data, the interaction prediction model is subjected to supervised training, and based on the third sample interaction data, the interaction prediction model is subjected to supervised training; wherein, in the supervised training process: the sample input data of the second sample interaction data is used as input data, and the sample output data of the second sample interaction data is used as training target, the third sample interaction data is used as input data, and the sample interaction actions of the third sample data before the error injection are used as training target.

12. A task execution method, characterized in that, include: In response to the interactive task of the target application, obtain the current interactive interface of the target application; The interactive prediction model is used to predict the task to be interacted with and the current interactive interface to obtain the predicted interactive action; wherein the interactive prediction model is trained based on the training method of the interactive prediction model according to any one of claims 9 to 11. The target agent is invoked to perform the predicted interaction action in the target application to obtain the new current interaction interface of the target application; Return to the step of predicting the interactive task and the current interactive interface based on the interaction prediction model to obtain the predicted interactive action, until the interactive task is completed.

13. The method according to claim 12, characterized in that, The step of predicting the interactive task and the current interactive interface based on the interaction prediction model to obtain the predicted interactive action includes: The target agent inputs the task to be interacted with, the current screenshot of the current interactive interface, the interface structure data of the current interface, and the current memory data of the task to be interacted with at the current time step as target input data into the interaction prediction model. Obtain the target output data of the interaction prediction model; wherein, the output data includes the predicted interaction action executed by the target agent at the current time step of the target application, the predictive thinking process of the predicted interaction action, and the interaction action description of the predicted interaction action. After the target agent is invoked to perform the predicted interaction action in the target application to obtain the new current interaction interface of the target application, the method further includes: The system stores the target input data and target output data of the interaction prediction model during the execution of this task; wherein, the stored data is used to: update the training dataset used to train the interaction prediction model, and / or, so that the target object can call the predictive thought process of the predicted interaction action and the interaction action description of the predicted interaction action during the execution of this task.

14. An interactive data construction apparatus, characterized in that, include: The sequence acquisition module is used to acquire a first interface sequence collected by the sample application when the sample application performs a sample interaction task; wherein, the first interface sequence includes a plurality of sample interaction interfaces, and adjacent sample interaction interfaces are switched by sample interaction actions. The field masking module is used to perform field masking based on the correlation between each data field in the interface structure data of the sample interactive interface and the sample interactive interface, so as to obtain the mask structure data of the sample interactive interface. The deduplication module is used to select a first interaction interface pair and perform a deduplication operation based on the first similarity between the data feature representations extracted from the mask structure data of each of the sample interaction interfaces to obtain a new first interface sequence; wherein, when performing the deduplication operation, one of the sample interaction interfaces in the first interaction interface pair is replaced by the other sample interaction interface. A sequence segmentation module is used to segment based on the new first interface sequence to obtain a second interface sequence; wherein, the second interface sequence includes at least one pair of adjacent sample interaction interfaces and the sample interaction actions between them. The sample construction module is used to obtain the first sample interaction data of the interaction prediction model based on the second interface sequence and the sample interaction task to which the second interface sequence originates; wherein, the interaction prediction model is used to predict the interaction action based on the task to be interacted with.

15. A training device for an interactive model, characterized in that, include: A sample acquisition module is used to acquire first sample interaction data of the interaction prediction model; wherein, the first sample interaction data includes: sample interaction actions between adjacent sample interaction interfaces and between them when the sample interaction task is executed, and the first sample interaction data is obtained by the interaction data construction device of claim 14. The model training module is used to train the interaction prediction model based on the first sample interaction data.

16. A task execution device, characterized in that, include: The interface acquisition module is used to acquire the current interactive interface of the target application in response to the interactive task of the target application. An action prediction module is used to predict the task to be interacted with and the current interactive interface based on an interaction prediction model to obtain a predicted interaction action; wherein the interaction prediction model is trained based on the training device of the interaction prediction model according to claim 15. An action execution module is used to invoke the target agent to perform the predicted interaction action in the target application, thereby obtaining a new current interaction interface of the target application. The iterative execution module is used to return to the steps of performing the prediction of the interactive task and the current interactive interface based on the interaction prediction model to obtain the predicted interactive action, until the interactive task is completed.

17. An electronic device, characterized in that, It includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the interactive data construction method according to any one of claims 1 to 8, or to implement the interactive model training method according to any one of claims 9 to 11, or to implement the task execution method according to any one of claims 12 or 13.

18. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the interactive data construction method according to any one of claims 1 to 8, or to implement the interactive model training method according to any one of claims 9 to 11, or to implement the task execution method according to any one of claims 12 or 13.

Images