Data enhancement method and device of operation track, storage medium and program product

Abstract

The embodiment of the invention provides an operation track data enhancement method and device, a storage medium and a program product. The method comprises the following steps: generating a hypothesis instruction which can be completed by an error track according to the error track which enables an intelligent agent not to complete a preset instruction; and based on the error trajectory and the hypothesis instruction, performing data enhancement on the agent. According to the technical scheme, the reasonable hypothesis instruction is generated through reverse reasoning of the error trajectory, original invalid operation data are converted into trainable samples, and the data utilization rate is increased. According to a traditional method, a knowledge base only stores a correct operation path and usually discards wrong trajectory data, so that an intelligent agent lacks adaptive capacity to abnormal conditions, and the method solves the problems that a large amount of correct planning data which depends on manual construction exists in the related technology, automatic learning and improvement from errors are difficult to continuously and automatically learn and improve, and the working efficiency is high. And the technical problems of high data construction cost and poor adaptability are solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device, storage medium, and program product for data augmentation of operational trajectories. Background Technology

[0002] With the rapid development of artificial intelligence technology, Computer Use Agents (CUAs) have begun to be used to automate the operation of graphical user interface environments such as web pages and applications. These agents can simulate user behavior, perform a series of complex tasks, and improve work efficiency to a certain extent. However, in practical applications, CUAs often encounter operational errors or execution failures when performing tasks due to environmental complexity, unclear task objectives, or ambiguous instructions.

[0003] The relevant technologies rely heavily on manual construction of correct trajectories and require a large number of predefined successful operation examples as planning guidance data. Even if a large number of failed operation trajectories are generated during the execution of the intelligent agent, the relevant technologies often ignore the value of these data, lack the ability to summarize the correct tasks and paths from errors, and find it difficult to continuously learn and improve automatically from errors. They also suffer from high data construction costs and poor adaptability. Summary of the Invention

[0004] This application provides a data augmentation method, device, storage medium, and program product for operation trajectories to alleviate or solve the technical problems in related technologies where a large amount of manually constructed "correct" planning data is difficult to continuously learn and improve from errors, resulting in high data construction costs and poor adaptability.

[0005] In a first aspect, embodiments of this application provide a data augmentation method for an operation trajectory, including: Based on the erroneous trajectory that prevents the agent from completing the predetermined instruction, generate a hypothetical instruction that can be completed based on the erroneous trajectory. Based on the error trajectory and the hypothetical instructions, the agent is augmented with data.

[0006] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0008] Fourthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0009] Based on the data augmentation method for operation trajectories described in the first aspect above, this application has at least the following beneficial effects or advantages: By reverse-analyzing failed operations, the originally invalid error trajectories are transformed into valuable training data, enabling the agent to learn from failure experiences and breaking through the limitations of traditional training using only successful data. A mapping relationship between erroneous operations and possible tasks is established, enhancing the agent's adaptability to unexpected operations. Operational patterns that deviate from standard operations but are reasonable are automatically generated, expanding the agent's understanding of task boundary situations. The automated reuse of error trajectories forms a data augmentation closed loop, continuously enriching the coping strategy library with accumulated execution, enabling the agent to autonomously evolve in real-world environments. Error data, traditionally considered "noise," is transformed into "signals" that enhance the agent's capabilities. By establishing a rationalization mechanism for erroneous operations, the adaptability and robustness of the system in non-ideal interactive environments are significantly improved.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0012] Figure 1 A flowchart illustrating the data augmentation method for operation trajectories according to an embodiment of this application is shown; Figure 2 This illustration shows a trajectory reuse diagram of the data augmentation method for operation trajectories according to an embodiment of this application; Figure 3 A block diagram illustrating a data augmentation method for operation trajectories according to an embodiment of this application is shown; Figure 4 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0013] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0014] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0015] The following terms will be used in the following text: A Computer Use Agent (CUA) is a programmable software entity that achieves its goals by sensing the state of its environment and executing a predetermined sequence of actions. It possesses characteristics such as autonomous decision-making, environmental interaction, and learning evolution, serving as the behavioral carrier of artificial intelligence systems. Designed for computer operation, AI agents can understand natural language instructions and automatically perform web page / application interaction tasks (such as clicking and inputting), improving the efficiency of human-computer interaction.

[0016] Large Language Models (LLMs) are deep learning models trained on massive amounts of text. They possess semantic understanding, reasoning, and generation capabilities. Through self-supervised learning, they acquire language representations from text data and have intelligent characteristics such as text generation, semantic understanding, and knowledge reasoning. Their performance exhibits a significant scaling law with the number of parameters and the scale of data.

[0017] Current technologies for automated web page operations and task execution have significant limitations, primarily manifested in their over-reliance on manually predefined data and the inefficient use of erroneous data. While traditional rule-based scripting systems can accurately execute predetermined operations, they require developers to pre-write complete operation procedures, lacking flexibility and adaptability. Although reinforcement learning agents possess a certain degree of generalization, their training process heavily relies on large, meticulously constructed datasets of successful cases, which are not only costly to build but also difficult to cover the various complex scenarios in real-world applications.

[0018] Existing systems generally lack systematic analysis and reuse mechanisms for error operation logs, resulting in the waste of a large amount of potentially valuable error data. Simultaneously, the failure to effectively establish and maintain a persistent mapping between user intent and task execution leads to the need to repeatedly solve similar problems, hindering continuous knowledge accumulation and evolution, and severely restricting the long-term adaptability and execution efficiency improvement of intelligent agents in complex environments. This reliance on perfect training data and neglect of error data often makes existing systems vulnerable and inefficient in practical applications, failing to meet increasingly complex automation demands.

[0019] It should be noted that the application scenarios or examples provided in this application embodiment are for ease of understanding, and this application embodiment does not specifically limit the application of the technical solution. Furthermore, the user information (including but not limited to user device information, user personal information, user operation trajectory information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, operation log data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 A flowchart illustrating the data augmentation method for operation trajectories according to an embodiment of this application is shown, such as... Figure 1 As shown, the method may include steps S101 and S102.

[0022] Step S101: Based on the erroneous trajectory that causes the agent to fail to complete the predetermined instruction, generate a hypothetical instruction that can be completed by the erroneous trajectory; Step S102: Perform data augmentation on the agent based on the error trajectory and the hypothetical instructions.

[0023] In the embodiments provided in this application, by analyzing the erroneous operation trajectories of the agent that failed to complete the predetermined instructions, reasonable task hypotheses corresponding to the erroneous trajectories are derived in reverse, i.e., hypothetical instructions are generated. Data augmentation training is then performed on the agent based on the erroneous trajectories and their corresponding hypothetical instructions. This realizes the value mining of erroneous operation data, transforming traditionally ignored erroneous trajectories into effective training samples. It enhances the agent's fault tolerance, enabling it to identify and adapt to non-standard operation paths. A more comprehensive task knowledge system is constructed, adding a fault-tolerant operation dimension in addition to the standard operation mode, thereby significantly improving the agent's adaptability and execution efficiency in complex environments. Through this positive transformation mechanism of erroneous data, the agent can continuously learn and evolve from execution experience, breaking through the limitations of traditional methods that rely on perfect training data.

[0024] In some embodiments provided in this application, step S101: generating a hypothetical instruction that can be completed based on the erroneous trajectory that causes the agent to fail to complete the predetermined instruction may include the following specific steps: The error trajectory is input into a large model for analysis, and node actions are extracted from the error trajectory; node actions are actions that affect the completion state of the hypothetical instruction. Based on the node actions, determine the error action pattern of the error trajectory; Determine the target action pattern that matches the erroneous action pattern from multiple predefined action patterns; each predefined action pattern corresponds to a different known instruction. Based on the known instructions corresponding to the target action pattern, generate hypothetical instructions.

[0025] In the embodiments provided in this application, the erroneous trajectory of the agent failing to complete a predetermined instruction is input into a pre-trained large model. This large model can be a deep learning model, such as the Transformer architecture, used to process sequence data. The large model analyzes the erroneous trajectory and extracts key node actions. Node actions refer to actions in the trajectory that have a significant impact on the task completion state. Based on the extracted node actions, the patterns of these actions are analyzed, and these node action patterns (i.e., erroneous action patterns) are identified. A predefined library containing multiple action patterns is used, each action pattern corresponding to a known instruction. The target action pattern that best matches the erroneous action pattern is found in the predefined action pattern library. Based on the matched target action pattern, a new hypothetical instruction is generated and output for the agent to execute.

[0026] According to some embodiments provided in this application, a process is further provided that includes processing from user instructions to agent operation logs, and generating multiple trajectories (including error trajectories and correct trajectories) based on the logs. The method also includes the following specific steps: The first user command is input into the large model for processing to generate an operation path plan. The control agent executes processing according to the planned operation path, and the operation log of the agent is obtained. The operation logs are input into a large model for processing to generate multiple trajectories for the agent. These trajectories include error trajectories and correct trajectories, with the correct trajectory being the operation trajectory that enables the agent to complete the predetermined instructions.

[0027] In the embodiments provided in this application, a first user instruction is input into the large model. The user instruction can be a specific task description, such as "Please help me book a high-speed rail ticket to Shanghai tomorrow." The large model generates an operation path plan based on the user instruction. The path plan is the specific steps and action sequence for the agent to complete the task. For example, "Open a travel app → Log in → Search for trains → Place an order." Based on the generated operation path plan, the agent is controlled to execute the corresponding actions, serving as the operation strategy guide for the agent. During the execution process, the agent records an operation log, which includes the agent's mouse clicks, page jumps, input box entries, and other process actions at each time point. The operation log includes multiple operation results, meaning that there may be erroneous and correct operations for the same user instruction. The operation log may include many process actions, and may even include some misoperations. The operation log is input into the large model for processing. The large model can extract a series of process operations into a trajectory, including operation log data of completed or uncompleted user instructions. That is, the corresponding generated error trajectory refers to the operation trajectory of the agent failing to complete the predetermined instruction, while the correct trajectory refers to the operation trajectory of the agent successfully completing the predetermined instruction.

[0028] An example of an operation log is provided for easier understanding: Initial Phase: Assessment: Unknown - Initial state, no operations have been performed yet; Task progress: (empty); Action 1 / 1: {"Open tab":{"URL":"https: / / www.somewebsite.com / v2 / welcome"}}; System memory: (empty).

[0029] Phase 1: Successfully accessing the website: Evaluation result: Success - Successfully accessed the official website of a certain website; Task progress: A website has been opened; Action 1 / 1: {"Click element":{"Index":5,"xpath":null}} (Click the navigation bar); All memories: Website navigation elements are loaded, including the tool access navigation bar.

[0030] Phase Two: Tool Menu Operations Evaluation result: Success - Expand the "Tools" drop-down menu to display a list of functions including "ABA Data Product Selection"; Task progress: 2. Open the website and access the "Tools" drop-down menu; Action 1 / 2: {"Click element":{"Index":19,"xpath":null}}; Action 2 / 2: {"Click element":{"Index":24,"xpath":null}}; All memories: Website navigation elements are available, including tool access paths, and navigation options for ABA data selection are visible.

[0031] Phase 3: Target Positioning (Initial Failure) Evaluation result: Failure - The user mistakenly clicked on "Product Monitoring" instead of the target "ABA Data Product Selection"; Task progress: The website is now open; Action 1 / 3: {"Click element":{"Index":5,"xpath":null}}; Action 2 / 3: {"Scroll down":{}}; Action 3 / 3: {"Click element":{"Index":13,"xpath":null}}; System memory: Website navigation elements are available, including a navigation bar for accessing tools. Tools and options for navigating to ABA Data Selection are visible. The path to "ABA Data Selection" is not displayed on the current page.

[0032] Phase Four: Second Failure Evaluation Result: Failed - Navigation to the wrong section, did not reach "Big Data Product Selection" Task progress: Action 1 / 4: {"Click element":{"Index":5,"xpath":null}} Action 2 / 4: {"Scroll down":{"Amplitude":300}} Action 3 / 4: {"Scroll down":{"Amplitude":200}} Action 4 / 4: {"Scroll down":{"Amplitude":200}} System memory: Website navigation elements are available, including a navigation bar for accessing tools. Tools and options for ABA data selection are visible. The path to "ABA Data Selection" is not displayed on the current page. Tool options for "Big Data Selection" are not displayed.

[0033] Phase 5: Three Failures Evaluation result: Failed - Previous attempt to navigate to "ABA Data Selection" was unsuccessful, possibly due to a click error or the element being unavailable.

[0034] Task progress: Action 1 / 3: {"Upward swipe":{"Amplitude":300}} Action 2 / 3: {"Upward swipe":{"Amplitude":300}} Action 3 / 3: {"Upward swipe":{"Amplitude":300}} System memory: Website navigation elements are available, including a navigation bar for accessing tools. Tools and options for navigating to ABA Data Selection are visible. The path to "ABA Data Selection" is not displayed on the current page. Tool options for "Big Data Selection" are not displayed. Elements related to the target "ABA Data Selection" navigation are not within the view scope.

[0035] Phase 6: Evaluation result: Success - The navigation elements for "Big Data Product Selection" and "ABA Data Product Selection" are now visible.

[0036] Task progress: I opened a website.

[0037] Action 1 / 1: {"Click element":{"Index":24,"xpath":null}} System memory: Website navigation elements are available, including a navigation bar for accessing tools. Tools and options for navigating to ABA Data Selection are visible. The path to "ABA Data Selection" is not currently displayed on the page. The tool options for "Big Data Selection" were not previously displayed. Elements related to the target "ABA Data Selection" navigation were previously out of view. Clickable elements for "ABA Data Selection" are now clearly displayed under the "Big Data Selection" item.

[0038] Phase 7: Evaluation result: Success - Successfully navigated to the ABA Data Selection section.

[0039] Task progress: 1. Open a website. 2. Navigate to the ABA Data Selection section.

[0040] Action 1 / 4: {"Click element":{"Index":32,"xpath":null}} Action 2 / 4: {"Input Text":{"Index":32,"Text":"US Site"}} Action 3 / 4: {"Input Text":{"Index":35,"Text":"Smartwatch"}} Action 4 / 4: {"Click element":{"Index":36,"xpath":null}} System Memory: Website navigation elements are available, including a navigation bar for accessing tools. Tools and options for navigating to ABA Data Selection are visible. The "ABA Data Selection" path was not previously displayed on this page. The "Big Data Selection" tool options were not previously displayed. Elements related to the target "ABA Data Selection" navigation were previously out of view. Clickable elements for "ABA Data Selection" are now clearly displayed under "Big Data Selection". Existing interactive elements are available for selecting markets, entering keywords, and performing search operations.

[0041] Phase 8: Evaluation result: Success - Based on the page element updates, the input operations for "US Site" and "Smartwatch" were successful.

[0042] Task progress: 1. Open a website. 2. Navigate to the ABA Data Product Selection section. 3. Select the "US" market. 4. Enter the keyword "smartwatch".

[0043] Action 1 / 3: {"Click element":{"Index":33,"xpath":null}} Action 2 / 3: {"Input Text":{"Index":33,"Text":"10%"}} Action 3 / 3: {"Click element":{"Index":36,"xpath":null}} System Memory: Website navigation elements are available, including a navigation bar for accessing tools. Tools and options for navigating to ABA Data Selection are visible. The "ABA Data Selection" path was not previously displayed on this page. The "Big Data Selection" tool options were not previously displayed. Elements related to the target "ABA Data Selection" navigation were previously out of view. Clickable elements for "ABA Data Selection" are now clearly displayed under "Big Data Selection". Existing interactive elements are available for selecting markets, entering keywords, and performing search operations. The keyword "smartwatch" has been successfully entered, but the search growth rate setting has not yet been processed. The "Search Now" button is available.

[0044] Phase Nine: Evaluation result: Success - You have successfully clicked the "Search Now" button, and the interface is preparing to perform search growth analysis.

[0045] Task progress: 1. Open a website. 2. Navigate to the ABA Data Product Selection section. 3. Select the market as "US". 4. Enter the keyword "smartwatch". 5. Set the search growth rate to "10%". 6. Click "Search Now".

[0046] Action 1 / 1: {"Extract Content":{"Target":"Identify the fastest-growing keywords related to 'smartwatch'"}} System Memory: Website navigation elements are available, including a navigation bar for accessing tools. Tools and options for navigating to ABA Data Selection are visible. The "ABA Data Selection" path was not previously displayed on this page. The "Big Data Selection" tool options were not previously displayed. Elements related to the target "ABA Data Selection" navigation were previously out of view. Clickable elements for "ABA Data Selection" are now clearly displayed under "Big Data Selection". Existing interactive elements are available for selecting markets, entering keywords, and performing search operations. The keyword "smartwatch" has been successfully entered, but the search growth rate setting has not yet been processed. The "Search Now" button is available. Search results have been generated for the entered "smartwatch", indicating that the search growth analysis page is ready.

[0047] Phase 10: Evaluation result: Success - Detailed information related to the keyword "smartwatch" has been successfully extracted, including search growth metrics and associated brands.

[0048] Task progress: 1. Open a website. 2. Navigate to the ABA Data Product Selection section. 3. Select the market as "US". 4. Enter the keyword "smartwatch". 5. Set the search growth rate to "10%". 6. Click "Search Now". 7. Analyze the keywords with rapid search growth.

[0049] Action 1 / 1: {"Completed":{"Text":"Research found that the keyword 'smartwatch' ranking improved significantly, with a substantial increase in search frequency. Its ranking jumped from 555,986 to 614, a change of 5,756, representing a growth rate of 1.04%. Leading brands associated with this keyword include Mattast, Fitbit, and QONBINK. The keyword had 16,620 impressions, 240 clicks, and a PPC bid range of $1.07 to $1.77."}} System Memory: Website navigation elements are available, including a navigation bar for accessing tools. Tools and options for navigating to ABA Data Selection are visible. The "ABA Data Selection" path was not previously displayed on this page. The "Big Data Selection" tool options were not previously displayed. Elements related to the target "ABA Data Selection" navigation were not previously in view. Clickable elements for "ABA Data Selection" are now clearly displayed under "Big Data Selection". Existing interactive elements are available for selecting markets, entering keywords, and performing search operations. The keyword "smartwatch" has been successfully entered, but the search growth rate setting has not yet been processed. The "Search Now" button is available. Search results have been generated for the entered "smartwatch", indicating that the search growth analysis page is ready. The keyword "smartwatch" is currently ranked 614, previously 555,986, a change of 5,756, or 1.04%. The top brands are Brand 1, Brand 2, and Brand 3.

[0050] The multiple stages recorded in the above operation log can be used by the large model to synthesize hypothetical instructions. The synthesis prompt for the hypothetical instructions can be "Please write a task that can be completed by this trajectory. The task can be explicitly completed by a part of the trajectory. Give the task and the relevant trajectory."

[0051] Based on the above operation logs, the large model can output the following task objectives (i.e., hypothetical instructions) and action trajectories excluding the error correction process, in JSON data format, as shown in the following example: [ { Task Objective: "Query the latest 30 days of electronics product data from the US site on a certain website's ABA data product selection page." "Task Track": [ Enter 'https: / / www.xxx.com / v2 / welcome' in the address bar of your browser's new tab. "Select the corresponding website address from the address bar dropdown suggestions". Click the 'Tools' option in the top navigation bar of the page. "In the pop-up menu of the 'Tools' option, click 'ABA Data Selection'", Click the 'Market Selection' dropdown menu. "Click 'US Marketplace' in the 'Market Selection' dropdown menu". Click the 'Date Selection' dropdown menu. Click 'Last 30 Days' in the 'Date Selection' drop-down menu. Enter 'electronic products' in the 'keyword' input box. Click the 'Search Now' button. ] } ] According to some embodiments provided in this application, the operation log includes the agent's erroneous actions. Inputting the operation log into a large model for processing to generate multiple trajectories of the agent may include the following specific steps: For each of the multiple trajectories, a large model is used to remove erroneous actions, resulting in multiple node actions; node actions are actions that affect the completion state of the hypothetical instruction. Each trajectory is generated according to the temporal sequence between the actions of multiple nodes.

[0052] In the embodiments provided in this application, erroneous actions in the agent's operation log are filtered and temporally reorganized using a large model to generate multiple valid trajectories. The input is an operation log containing erroneous actions, such as actions A, B, X, C, D, Y, and E, which can be represented as [A, B, X, C, D, Y, E], where X and Y are erroneous actions. A pre-trained large model is used to perform semantic and temporal analysis on the operation log to identify actions unrelated to instruction completion, i.e., erroneous actions. By removing noise from the operation log data, the quality of the trajectory is improved, preventing invalid operations from interfering with the agent's learning. A clean sequence of node actions more accurately reflects the instruction completion logic, contributing to the accuracy of action pattern recognition in the operation trajectory. Even if the operation log contains erroneous actions, valid operation trajectories can still be extracted, enabling the agent to recover orderly operations from chaos.

[0053] For example, the large model described above can transform operation logs into a vector space representation, such as identifying outlier actions through clustering or anomaly detection. It analyzes causal dependencies between actions (e.g., "A must be executed before B"), marks actions that violate logic, and thus identifies erroneous actions. Identified erroneous actions are removed from the operation logs, retaining key node actions (e.g., [A, B, C, D, E]). Based on the domain knowledge learned by the large model, the correct temporal sequence of node actions is restored, and the sorted node actions are combined into a complete trajectory.

[0054] For example, by processing large input models, the agent decouples system-level fault tolerance (such as network reconnection and element refresh) from business logic trajectory generation (such as ticket booking processes). The two types of problems are addressed at different levels: low-level fault tolerance and business trajectory. This involves in-depth analysis of various anomalies that the agent may encounter during operation. Low-level fault tolerance specifically handles sudden system problems unrelated to business logic, including but not limited to technical failures such as unresponsive pages, malfunctioning interactive elements, and network interruptions. These system problems are typically sporadic and random, and their solutions are often a series of standardized technical measures. In contrast, business trajectory focuses on handling path deviations caused by misunderstandings of business principles or operational errors. These problems are analyzable, reproducible, and transformable, providing valuable feedback for the agent's continuous evolution.

[0055] For example, the aforementioned intelligence possesses various fault tolerance capabilities to cope with complex and ever-changing environments. Error types include, for example, network errors, interface element errors, data-related errors, permission errors, system resource errors, etc. The following describes the fault tolerance capabilities of the intelligent agent based on different error types and specific operational examples.

[0056] The aforementioned network errors can include network interruptions, high network latency, server connection failures, DNS resolution errors, etc. The agent's fault tolerance methods can include using a retry mechanism: after detecting a network error, it waits a short time (e.g., 3 seconds) and automatically retrys the operation, setting a maximum number of retries (e.g., 3 times). Alternatively, it can utilize caching; if a retry fails, it attempts to retrieve data from the local cache for display or processing. Another approach is to initiate network status monitoring, automatically re-execute the operation once network conditions are restored.

[0057] The aforementioned UI element errors can include missing interactive elements, abnormal UI element interactions, and element position offsets. The agent can reposition elements by analyzing the DOM structure or UI layout, or by finding alternative interaction methods. The agent can also simulate actions, using techniques such as simulated clicks and keyboard input to attempt to trigger element functions.

[0058] The aforementioned data-related errors can include missing data, data conflicts, etc. The agent can perform data validation and repair according to established rules, and supplement missing field default values. If data errors cause the operation to fail, the agent can automatically roll back to the state before the operation to avoid data inconsistency.

[0059] The aforementioned permission errors may include insufficient permissions, not logged in, role-based permission restrictions, expired authorization, etc. The intelligent agent can automatically redirect to the login or authorization page for such errors, provide a user interaction interface, and assist the user in completing the permission acquisition process.

[0060] The aforementioned system resource errors may include insufficient device memory, excessive CPU usage, and insufficient storage space. When the intelligent agent can identify the above errors, it can close idle background processes or clear the cache to release system resources. It can also reduce operational complexity or data quality, such as loading low-resolution images.

[0061] According to some embodiments provided in this application, when a user issues a new second user instruction, the intelligent agent, after data augmentation of the error trajectory and the corresponding hypothetical instruction, can retrieve the reused operation trajectory in real time from the knowledge base. The method also includes the following processing: Receive instructions from the second user; If the second user instruction matches the assumed instruction, the controlling agent completes the second user instruction according to the erroneous trajectory.

[0062] In some embodiments provided in this application, the intelligent agent receives a second user instruction and matches it with existing hypothetical instructions. Once a match is found, the intelligent agent invokes the error trajectory corresponding to the hypothetical instruction and directly completes the user instruction according to this trajectory. This allows the intelligent agent to make full use of historical error experience and transform previous ineffective operations into effective strategies. By reusing error trajectories to execute the second user instruction, the intelligent agent achieves an evolution from passively responding to errors to actively utilizing error experience. This not only improves task execution efficiency but also represents a deep application of data augmentation technology in intelligent systems.

[0063] According to some embodiments provided in this application, the matching of a user instruction with multiple hypothetical instructions may stem from the ambiguity of the instruction semantics, the multi-path nature of the task objective, or the diversity of historical error trajectories. The method may further include the following steps: When the second user instruction matches multiple hypothetical instructions, the multiple hypothetical instructions and the error trajectories corresponding to the multiple hypothetical instructions are input into the large model for processing. Using a large model, the target trajectory for completing the second user instruction is obtained based on the error trajectory with the highest confidence among the error trajectories corresponding to multiple hypothetical instructions.

[0064] In the embodiments provided in this application, the same operation trajectory can be assigned different semantics by different instructions, and the same instruction may also have different implementation methods. For example, if the user instruction is to buy a train ticket to a certain place at a certain time, the agent may have multiple trajectories of operations in different ticket-buying apps, all of which can realize the aforementioned user instruction. Multiple erroneous trajectories corresponding to hypothetical instructions can be input into a large model as an optional method. The large model then performs a confidence assessment and selects the trajectory with the highest confidence as the target trajectory. The aforementioned trajectory can be an erroneous trajectory previously derived from an instruction. When the current user instruction matches its corresponding hypothetical instruction, the previously erroneous trajectory becomes a correct trajectory with a verified processing method, which can be reused in the execution of a second user instruction.

[0065] To illustrate, consider this example: An agent receives the instruction "Book a high-speed rail ticket from Beijing to Shanghai," but the execution path is incorrect: [Log in to a travel app → Enter date → Mistakenly click the "Hotel Booking" tab → Select a hotel in Beijing]. Following this incorrect path, if the instruction were "Book a hotel in Beijing," the incorrect path, while not matching the actual booking instruction, is a correct path for the hypothetical instruction and has been verified as executable, thus achieving the goal of "Booking a hotel in Beijing." Associating this incorrect path with the hypothetical instruction enriches the agent's knowledge base. It allows the agent to reuse existing "incorrect" paths to execute the correct processing when a user has a similar or identical need to the hypothetical instruction. By establishing a mapping between incorrect paths and hypothetical instructions, the agent's knowledge base is enhanced; simply replacing key parameters (e.g., changing "Beijing" to the target city) allows for efficient task completion. This mechanism not only improves the agent's utilization of erroneous operations but also continuously expands the agent's capability boundaries through the accumulated "error-hypothetical instruction" mapping.

[0066] According to some embodiments provided in this application, in step S102: data augmentation is performed on the agent based on the error trajectory and the hypothetical instructions.

[0067] Based on the hypothetical instructions and error trajectories, the mapping data is obtained; Updating the agent's knowledge base based on the mapped data is a way to augment data.

[0068] In this embodiment, by analyzing the erroneous trajectory generated when the agent executes the original predetermined instruction (such as the example of mistakenly entering a hotel booking process mentioned above), the hypothetical instruction that the trajectory can actually complete (such as "book a hotel") is derived in reverse. Then, a mapping relationship is established between the two, associating the semantic representation of the erroneous trajectory with the hypothetical instruction to form structured mapping data. Through this process, knowledge acquisition efficiency is improved, erroneous execution records are transformed into effective training data, the reusable trajectories of executable tasks are expanded, and the cost of repeated trial and error is avoided.

[0069] It's important to clarify that the knowledge base and training set of an intelligent agent are essentially two complementary but distinct data carriers within a machine learning system. The knowledge base, as a structured repository of experiential knowledge, stores explicit rules and validated operational templates, such as the complete trajectory of "clicking the hotel tag → completing the booking," common error patterns and remedial solutions, and domain constraints. Its core value lies in providing real-time decision support to the agent, dynamically retrieving relevant knowledge during execution through efficient retrieval and reasoning mechanisms (e.g., directly returning pre-stored operation flows when matching user command semantics, or triggering preset error correction schemes when operational deviations are detected). The training set, on the other hand, is a raw dataset, such as massive user command-operation trajectory pairs and system operation logs. It's primarily used for offline model parameter training, mining potential patterns in the data through statistical learning, but lacks the ability to directly guide real-time operations. The key difference lies in the knowledge base's emphasis on interpretable deterministic knowledge and online service capabilities, with updates relying on logical rule injection or incremental mechanisms such as error trajectory mapping as described in this application. The training set, however, focuses on discovering statistical patterns and requires batch data import and model retraining for updates. In a typical scenario, when a user issues the instruction "book a hotel in Beijing", the knowledge base can immediately return a verified operation template. Through the transformation of error trajectories into hypothetical instructions, the rules of the knowledge base are expanded, and reusable trajectory mappings are added, forming a closed loop of co-evolution.

[0070] Based on the above embodiments and optional embodiments, this application also provides an optional implementation method. This optional implementation method is used to generate web page instructions based on computer-based intelligent agents and large language models. By having computer-based intelligent agents execute web page tasks, record execution logs, analyze error trajectories, and synthesize correct task trajectories, a knowledge base of user operation intentions and tasks is accumulated, thereby ultimately improving the success rate of computer-based intelligent agent operations.

[0071] Figure 2 This illustration shows a trajectory reuse diagram of the data augmentation method for operation trajectories according to an embodiment of this application, as shown below. Figure 2 As shown, the process of an optional implementation includes the following steps: 1. User input of webpage commands.

[0072] Users input their desired action on the webpage using natural language or structured statements. For example: "Please help me book a high-speed train ticket to Shanghai for tomorrow."

[0073] 2. Large-scale model LLM for task planning.

[0074] By using a large language model to understand and decompose user commands, an operation path plan is generated, such as: opening a website → logging in → searching for train schedules → placing an order. This path serves as the operation strategy guide for the intelligent agent.

[0075] 3. The agent executes the operation strategy.

[0076] The agent executes web page operations based on the path strategy generated by the large model, and the system records the entire operation process, including mouse clicks, page jumps, and input box filling.

[0077] 4. Record the execution process log (including error traces).

[0078] During operation, the agent may make mistakes, such as clicking the wrong button, entering incorrect content, or the page failing to load. The system records all processes to form an operation log.

[0079] 5. Incorrect trajectories are reverse-engineered to form correct task trajectories.

[0080] A novel task generation prompt is proposed: "Based on the following error operation trajectory, write a task that can be completed by this trajectory. The task can be explicitly completed by part of the trajectory. Give the task and the relevant trajectory." The error trajectory is analyzed by a large model, and a reasonable task (hypothetical instruction) corresponding to the error trajectory is synthesized.

[0081] 6. Establish task trajectory mapping relationships.

[0082] The correct task (i.e. the hypothetical instruction) derived from the erroneous trajectory is bound to the erroneous trajectory to form mapping data, which is then stored in the knowledge base.

[0083] 7. Knowledge accumulation and reuse mechanism.

[0084] When subsequent users issue similar or identical operation requests, the system can quickly retrieve historical mapping records, reuse existing task planning paths, and improve execution efficiency and accuracy.

[0085] Through the above processing, by automatically analyzing the error trajectories during CUA execution, usable correct task data is synthesized in reverse, avoiding reliance on large-scale manually constructed "successful operation samples" and significantly reducing the initial investment in data construction. Using LLM for semantic analysis and reconstruction of error trajectories enables the computer system to learn from failures, summarize operational paths, and continuously evolve. User-issued natural language operation commands can be continuously mapped to task trajectories by the system, forming an "intent-execution" knowledge mapping, allowing for direct reuse of existing trajectories or efficient retrieval and combination in the future.

[0086] Figure 3 A block diagram illustrating a data augmentation method for operation trajectories according to an embodiment of this application is shown, such as... Figure 3 As shown, corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a data enhancement device for operation trajectories, including: The trajectory inversion module 301 is used to generate hypothetical instructions that can be completed based on the erroneous trajectory that caused the agent to fail to complete the predetermined instructions. The data augmentation module 302 is used to augment the agent based on error trajectories and hypothetical instructions.

[0087] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0088] Figure 4 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 4 As shown, the electronic device includes a memory 401 and a processor 402. The memory 401 stores a computer program that can run on the processor 402. When the processor 402 executes the computer program, it implements the method described in the above embodiments. The number of memories 401 and processors 402 can be one or more. In a specific implementation, the electronic device may also include a communication interface 403 for communicating with external devices and performing data exchange and transmission.

[0089] In practical implementation, if the memory 401, processor 402, and communication interface 403 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0090] Optionally, in a specific implementation, if the memory 401, processor 402 and communication interface 403 are integrated on a single chip, the memory 401, processor 402 and communication interface 403 can communicate with each other through an internal interface.

[0091] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0092] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0093] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0094] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0095] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0096] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0097] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0098] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0099] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0100] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0101] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0102] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0104] The above are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data augmentation method for an operation trajectory, characterized in that, include: Based on the erroneous trajectory that prevents the agent from completing the predetermined instruction, generate a hypothetical instruction that can be completed based on the erroneous trajectory. Based on the error trajectory and the hypothetical instructions, the agent is augmented with data.

2. The method according to claim 1, characterized in that, The data augmentation of the agent based on the error trajectory and the hypothesized instructions includes: Based on the hypothetical instructions and the error trajectory, mapping data is obtained; The agent's knowledge base is updated based on the mapping data as a form of data augmentation.

3. The method according to claim 1, characterized in that, The step of generating hypothetical instructions that can be completed based on the erroneous trajectory that prevents the agent from completing the predetermined instructions includes: The error trajectory is input into a large model for analysis, and node actions in the error trajectory are extracted; the node actions are actions that affect the completion state of the hypothetical instruction. Based on the node actions, determine the erroneous action pattern of the erroneous trajectory; A target action pattern matching the erroneous action pattern is determined from a plurality of predetermined action patterns; each of the plurality of predetermined action patterns corresponds to a different known instruction. Based on the known instructions corresponding to the target action pattern, the hypothetical instructions are generated.

4. The method according to claim 1, characterized in that, The method further includes: The first user command is input into the large model for processing to generate an operation path plan. Control the intelligent agent to execute processing according to the operation path plan, and obtain the operation log of the intelligent agent; The operation log is input into the large model for processing to generate multiple trajectories for the agent; the multiple trajectories include the error trajectory and the correct trajectory, and the correct trajectory is the operation trajectory that enables the agent to complete the predetermined instruction.

5. The method according to claim 4, characterized in that, The operation log includes the agent's erroneous actions. The step of inputting the operation log into the large model for processing to generate multiple trajectories for the agent includes: For each of the multiple trajectories, the large model is used to remove the erroneous actions, resulting in multiple node actions; the node actions are actions that affect the completion state of the hypothetical instruction. Each trajectory is generated according to the temporal sequence between the actions of the multiple nodes.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Receive instructions from the second user; If the second user instruction matches the hypothetical instruction, the control agent completes the second user instruction according to the erroneous trajectory.

7. The method according to claim 6, characterized in that, The method further includes: If the second user instruction matches multiple hypothetical instructions, the multiple hypothetical instructions and the error trajectories corresponding to the multiple hypothetical instructions are input into the large model for processing. Using the large model, based on the error trajectory with the highest confidence among the error trajectories corresponding to the multiple hypothetical instructions, a target trajectory for completing the second user instruction is obtained.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images