GUI-Agent trajectory data generation method and system based on multi-agent collaborative reasoning

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a GUI-Agent trajectory data generation method and system based on multi-agent collaborative reasoning. The method comprises the following steps: S1, capturing and analyzing a page state and a structure, and constructing an initial graph; s2, analyzing semantics of interactive elements based on VLM and generating an initial action instruction set; s3, executing the instruction, recording state change, automatically deducing a target intention, forming an initial track, and storing the initial track in a track pool; s4, for the current track, predicting and executing the next action according to the target and the page state of the current track, and updating the track; s5, unexplored elements are recognized, new actions and intentions are generated in combination with the VLM, and a new track is created and added into a track pool; s6, evaluating whether the action result achieves a target or not, and deciding to terminate, adjust or continue to explore according to the target; and S7, circularly executing the steps S4 to S6, performing parallel iteration processing on the trajectory pool until a termination condition is met, and outputting a trajectory data set.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for generating GUI-Agent trajectory data based on multi-agent collaborative reasoning. Background Technology

[0002] With the rapid development of artificial intelligence technology, graphical user interface (GUI) agent systems based on vision-language models (VLM) have become an important technological direction in the field of digital automation. These systems can simulate the operational behavior of human users and autonomously execute complex computer tasks in various applications and operating systems, providing strong technical support for application scenarios such as robotic process automation (RPA).

[0003] In the development and optimization of GUI Agent systems, trajectory data is the core support for achieving automated task execution. Trajectory data records the complete operational process that the GUI Agent undergoes when completing a specific task, mainly including the following four parts: 1. Environment State: This refers to the current context information of the interface where the GUI Agent is located, such as the state of page elements, whether controls are visible, and the content of input boxes. For example, when a user logs into the system, the environment state may include "login page has been loaded, username input box is empty, password input box is obscured, and 'login' button is available".

[0004] 2. High-level instructions: These are abstract target commands generated by the task planning module or user intent. They describe "what to do" without specifying how to do it. For example, "Open the settings menu and turn off notifications" is a high-level instruction; it expresses the ultimate goal but does not specify the specific click path.

[0005] 3. Low-level instructions: These correspond to specific action commands at the actual execution level and are usually called directly by the automation engine, such as mouse clicks, keyboard inputs, scrolling, etc. For example, "Send a left mouse click event to coordinates (150, 200)" is a low-level instruction.

[0006] 4. Specific operation actions: These are the actual operations performed, such as clicking, inputting, and swiping. These actions are driven by low-level instructions and act on interface elements.

[0007] The RPA system can directly utilize trajectory data generated by the GUI Agent to automatically reproduce the same operational processes, thereby achieving unmanned operation of business processes. This method not only improves execution efficiency but also significantly reduces the risk of errors caused by human operation, providing reliable technical support for enterprise digital transformation.

[0008] However, existing trajectory data acquisition technologies have significant drawbacks, as follows: Traditional task-driven methods rely on manual calibration or pre-defined task execution processes, resulting in high costs and low efficiency. This new method requires professionals to manually design task scenarios and meticulously record the complete sequence of operations performed by the GUI agent during execution, encompassing multiple aspects such as environmental state, instruction mapping, and specific action execution. Specifically: environmental state refers to the visual and interactive state of the GUI interface at a given moment, such as the layout of page elements, the visibility of controls, the content of input boxes, and the clickability of buttons. This information forms the basic context for agent decision-making; instruction mapping refers to the process of translating high-level task intents (such as "login system") into executable low-level operation instructions, for example, mapping the semantics of "click the login button" to specific coordinate clicks or element ID trigger actions.

[0009] Due to the high complexity and dynamic nature of the GUI environment, and the diverse task scenarios, manually completing the above calibration process is time-consuming and labor-intensive, making it difficult to guarantee the consistency of large-scale data annotation, which seriously restricts the efficiency and economic feasibility of trajectory data acquisition.

[0010] Automatic model generation methods attempt to reduce the cost of manual annotation through artificial intelligence. A typical implementation utilizes large language models or visual-language models for autonomous reasoning to generate the operational trajectory of a GUI agent. For example, the system can input a natural language task description (such as "login account and view notifications"), and the large model, combined with the current interface state analysis, outputs a series of low-level operation instructions (such as clicking the login button, entering a username, etc.), thereby automatically constructing a complete execution path. However, this method still faces significant technical challenges. First, because the generation process highly depends on the understanding of the task intent and accurate perception of the environmental state, existing models typically rely on predefined task templates to constrain the inference scope and ensure the controllability of the output results. This rigid design makes it difficult for the system to adapt to dynamically changing real-world application scenarios, severely limiting the scalability and content diversity of the generated data. Second, due to the high complexity and uncertainty of the GUI environment, and the inherent instability of model inference itself, problems such as errors in intermediate steps and deviations from the task objective often occur during automatic execution. For example, the model may misjudge the function of a button or fail to correctly identify new elements after page navigation, leading to trajectory interruptions or logical confusion. These problems directly lead to quality issues in the generated trajectory data, such as missing, incomplete, or logical errors, which seriously affect the usability of the data and the subsequent training results.

[0011] Therefore, it is very important to design a GUI-Agent trajectory data generation method and system based on multi-agent collaborative reasoning that can generate high-quality, diverse, and logically complete GUI operation trajectory datasets without human intervention or predefined templates. Summary of the Invention

[0012] This invention aims to overcome the technical shortcomings of existing GUI agent trajectory data generation methods, such as high cost, difficulty in guaranteeing data quality, and poor scalability. Traditional manual calibration methods require professionals to manually design task scenarios and record operation sequences one by one, which is costly and inefficient. On the other hand, model automatic generation methods based on predefined tasks rely too much on fixed task templates, which limits the diversity and scalability of data. At the same time, due to the complexity of the GUI environment and the uncertainty of model inference, data loss or logical confusion are prone to occur. This invention provides a GUI-Agent trajectory data generation method and system based on multi-agent collaborative reasoning that can generate high-quality, diverse, and logically complete GUI operation trajectory datasets without manual intervention or predefined templates.

[0013] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A method for generating GUI-Agent trajectory data based on multi-agent collaborative reasoning includes the following steps; S1. Open the target webpage, obtain a complete screenshot of the current page, and use front-end parsing tools to extract the page DOM structure, identify and record the attribute information of all interactive elements, and construct the initial page state graph. S2, based on the initial page state graph, semantic analysis and value assessment of interactive elements are performed using the visual-language model (VLM) to generate a corresponding set of initial action instructions; S3, execute the actions in the initial action instruction set, record the page state changes before and after each action execution, automatically infer the target intent for each execution path, form an initial trajectory record, and store it in the trajectory pool; S4. For a current trajectory in the trajectory pool, combine the target intent corresponding to the trajectory and the current page state to predict and execute the next optimal interaction action, and update the trajectory record at the same time. S5: Analyze the unexplored interactive elements on the current page, combine VLM to generate potential new action commands and infer the new target intent, form a new initial trajectory record and add it to the trajectory pool; S6, After the action is completed, determine whether the action execution result of the current trajectory has achieved the preset goal, and decide on the subsequent processing strategy; S7. Repeat steps S4 to S6 to perform parallel or iterative execution, expansion and evaluation of multiple trajectories in the trajectory pool until the preset iteration termination condition is met, and output the final GUI operation trajectory dataset.

[0014] Preferably, in step S1, the attribute information of the interactive element includes the type, position, identifier, and text content attributes of the interactive element.

[0015] Preferably, in step S2, the semantic analysis and value assessment include the following process: Based on the semantic information, layout position and text content of interactive elements, all interactive elements are evaluated by the visual-language model (VLM), and a preset number of elements with the highest evaluation results are selected as the basis for generating the initial action instructions. The initial action instructions include clicking, entering text, selecting, or scrolling.

[0016] Preferably, in step S3, the action is executed by directly calling page elements for interaction through a standard browser automation interface; the automatically inferred target intent is generated by VLM based on the sequence of executed actions; the initial trajectory record includes the target intent corresponding to the trajectory, the historical action sequence, and page state changes.

[0017] Preferably, step S4 specifically includes the following steps: S41, input the target intent, page snapshot and DOM structure information of the current trajectory into VLM, and VLM outputs the next operation instruction that best matches the current context; S42 executes the output next operation instruction in a real or simulated environment; S43, after execution, the system automatically captures the operation result feedback and updates the latest action item in the trajectory record; the operation result feedback includes page jump and element state change.

[0018] Preferably, in step S5, generating potential new action instructions includes the following process: Construct a set of non-interactive elements on the current page, generate possible actions for each element using VLM, evaluate the rationality and potential value of each action based on the existing task context, and select the actions with the highest potential value as candidates.

[0019] Preferably, step S6, determining whether the action execution result of the current trajectory has achieved the preset goal, specifically includes the following process: Check whether the current page state meets the semantics of the task objective, or call VLM to perform semantic analysis on the current interface to determine the objective completion status; If the preset goal is achieved, the corresponding trajectory is marked as a complete trajectory and the expansion is terminated; If the preset goal is not achieved, assess whether the action result meets expectations: If the match is found, the action is retained and step S4 is executed; if the match is not found, the target intent is dynamically adjusted according to the current page state. If a new target is found, the trajectory target is updated and the exploration continues; otherwise, the subsequent exploration of the current trajectory is abandoned. The dynamic adjustment of target intent includes calling a large language model to match the current page state with possible task intents.

[0020] Preferably, in step S7, the iteration termination condition includes at least one of the following: all potential paths have been explored, no new valid trajectories are added in N consecutive iterations, or a preset time limit or computing resource limit is reached.

[0021] This invention also provides a GUI-Agent trajectory data generation system based on multi-agent cooperative reasoning, comprising: The initialization module is used to load the target webpage, analyze interactive elements, generate and execute initial action instructions to form multiple initial trajectory records; The trajectory pool is used to store and manage the initial trajectory records and the trajectories generated during the iteration process; The trajectory generation agent module is used for iterative processing of the trajectory pool, specifically including: The analysis and decision-making unit is used to predict the next interaction action based on the target intent and the current page state. The trajectory extension unit is used to mine potential interactions based on the current page state and generate new trajectories. The evaluation filtering and correction unit is used to evaluate the results of actions and to make decisions such as goal achievement judgment, trajectory correction, or abandonment.

[0022] Preferably, the initialization module includes an analysis and decision-making agent module and a trajectory initialization agent module; the analysis and decision-making unit in the trajectory generation agent module shares an analysis and decision-making agent module with the initialization module; and the agent modules communicate with each other through a message passing mechanism.

[0023] Compared with the prior art, the beneficial effects of this invention are: (1) De-manualization: No need for manual design of task scenarios or labeling of operation sequences, significantly reducing data generation costs; (2) High generalization ability: Get rid of dependence on fixed task templates and support cross-application and cross-interface universality; (3) Adaptive error correction: Possesses a target dynamic adjustment and trajectory elimination mechanism, effectively improving data quality and logical consistency; (4) High efficiency and scalability: Supports batch web page automated processing and is suitable for large-scale GUI agent training data construction. Attached Figure Description

[0024] Figure 1 This is a principle block diagram of the GUI-Agent trajectory data generation system based on multi-agent collaborative reasoning in this invention; Figure 2 This is a flowchart illustrating a GUI-Agent trajectory data generation method based on multi-agent collaborative reasoning in this invention. Figure 3 This is a flowchart illustrating the result evaluation and trajectory correction process in this invention. Detailed Implementation

[0025] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0026] This invention provides a two-level hierarchical Multi-Agent system, comprising the following two main modules and internal intelligent agents, such as... Figure 1 As shown: (1) Initialization Agent Responsible for webpage loading, state awareness, and initial trajectory construction, comprising two sub-agents: Analysis and Decision Agent: This agent performs semantic understanding of page elements and generates initial action commands. It plays a continuous role throughout the system's operation, initially used to generate the first operation command during the initialization phase. Trajectory Initialization Agent: Receives the above action instructions, performs the first operation, and establishes the first trajectory record.

[0027] The aforementioned "analysis and decision-making agent" is reused in the subsequent trajectory generation stage, serving as the core reasoning unit to participate in the next step of action prediction and evaluation, thus achieving cross-stage functional continuity.

[0028] (2) Trajectory Generation Agent Responsible for continuously expanding, evaluating, and optimizing existing trajectories, comprising three sub-agents: Analysis and Decision Agent: Predicts the optimal next action based on goal orientation; Trajectory Expansion Agent: Similar to the analysis and decision-making agent in the initialization phase, it is used to perform semantic parsing of the current environmental state and support the generation of new actions; Evaluation, Filtering and Correction Agent: Determines whether the objective has been achieved and decides whether to terminate, correct, or continue the exploration.

[0029] The "analysis and decision-making agent" used in the initialization and trajectory generation phases is actually the same agent module, invoked in different task phases to complete the corresponding decision-making tasks. While their functions are consistent, they differ in input context and output objectives, demonstrating the agent's reusability and adaptability to different phases.

[0030] The agents communicate with each other through a message passing mechanism, forming a closed-loop process of "perception-decision-execution-evaluation-re-decision".

[0031] like Figure 2 As shown, this invention provides a GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning, which is specifically performed according to the following steps: Step 1, Webpage Initialization and Structure Capture Executor: Analytical and decision-making intelligent agent; Function Description: Open the target webpage, obtain a complete screenshot of the current page, and use a front-end parsing tool to extract the page's DOM structure, record the position, type, and attribute information of all interactive elements, and construct an initial page state graph.

[0032] The operation process, for example: (1) Open the target webpage and obtain a complete screenshot of the current page; (2) Use front-end parsing tools (such as Chrome DevTools Protocol) to extract the page DOM structure; (3) Iterate through all interactive elements (such as buttons, input boxes, links, etc.) and record their: Position coordinates (x, y); Element type (button, input, a, etc.); Attribute information (id, class, text content, aria-label, etc.); (4) Use VLM to evaluate the value of each interactive element (based on element semantics, layout position, text content, etc.) and generate corresponding actions.

[0033] (5) Select the 20-30 most promising high-value elements and their corresponding actions; (6) Perform these high-value actions sequentially (such as clicking the "Login" button, entering "XXX place" into the search box, etc.); capture a new page snapshot after each operation and update the DOM structure; then return to the initial page and perform the corresponding action on the next candidate element until all elements have been operated on. Output: Page snapshot, initial page state graph (including a list of interactive elements and their attributes).

[0034] Step 2, Interaction Element Analysis and Action Decision Executor: Analytical and Decision-Making Intelligent Agent Function Description: Iterate through every interactive element on the page, and combine the visual-language model (VLM) with contextual semantic analysis to generate appropriate operation instructions: If it is an input box element, then VLM is called to predict reasonable input content; If the element is a trigger element such as a button or link, a click action instruction will be generated directly.

[0035] The operation process, for example: (1) Automatically infer the target intent for each execution path (generated by the VLM model based on the action sequence, such as "complete user login" or "query tourist information for XXX place"); (2) Initialize the trajectory recording mechanism and record: Comparison of target intent, historical action sequence, and page state changes before and after; (3) Store each independent trajectory in the trajectory pool for easy traversal next time.

[0036] Output: A set of action instructions (such as "Click button A" or "Enter text C in input box B").

[0037] Step 3, Action Execution and Trajectory Initialization Executor: Trajectory Initialization Agent Function Description: This module is responsible for receiving action commands and executing corresponding operations in a real or simulated browser environment. The execution process directly calls page elements for interaction based on standard browser automation interfaces.

[0038] Specifically, the system implements control operations in the following ways: For button controls (such as "Login" and "Submit"), call the "click()" method to trigger the click event; For input box controls (such as username and password fields), first locate the element, and then call the "type()" method to input the text content; For drop-down menus or tabs, you can activate and select the target item using the "select()" or "hover()" operation; For scrollable areas, use "scrollTo()" or "executeScript()" to scroll the page; For pop-ups or modals, they are automatically identified and handled by listening for DOM changes or event callbacks.

[0039] After each operation is completed, the system automatically captures a snapshot of the current page, updates the page structure information (such as the DOM tree and changes in element attributes), and records the status changes of this operation.

[0040] Simultaneously, the initial trajectory recording mechanism is activated to generate a trajectory record containing the following content: The target intent of the current operation (automatically inferred and generated by the large model based on the action); Historical action sequences and their corresponding page state changes; Operation timestamp and execution result feedback.

[0041] Output: Initial trajectory record (target intent + historical action sequence), updated page state graph.

[0042] Step 4, Goal-oriented action execution Executor: Analytical and Decision-Making Intelligent Agent Function Description: For each initialized trajectory, based on its preset target intent (such as "login to the system" or "submit form"), the Visual-Language Model (VLM) is invoked to perform semantic understanding and behavioral reasoning of the current interface state. This process includes: Enter a screenshot of the current page and the corresponding DOM structure information; The VLM model is invoked to generate the optimal interaction action that best fits the current context; Execute selected action commands in a real or simulated environment via the Chrome DevTools Protocol (CDP).

[0043] After execution, the system automatically captures the feedback of the operation results (such as page jumps, element state changes) and updates the latest action items in the trajectory record.

[0044] The operation process, for example: (1) The target intent of the current trajectory is received by the "analysis and decision-making agent" in the "trajectory generation agent"; (2) Call the Visual-Language Model (VLM) and combine the semantic context of the current page with the element layout features to predict the next optimal interaction action; (3) Generate specific operation instructions based on the prediction results (such as clicking a button, entering specified text, etc.); (4) Pass the action instruction to the execution module to drive the browser to perform the corresponding interactive behavior; (5) After execution, obtain a new page snapshot and update the DOM structure using a front-end parsing tool; (6) Record the state changes after the action is performed and feed the results back to the "evaluation, filtering and correction agent". Output: Updated trajectory record, including the latest action execution results and page state changes; Step 5, Dynamic Path Extension of Trajectory Executor: Trajectory Extension Intelligent Agent Function Description: For the remaining elements on the current interface that have not yet been interacted with (such as unclicked buttons, unfilled input fields, hidden dropdown menus, etc.), the VLM model is invoked to mine potential behaviors. The specific process is as follows: Construct a collection of currently uninterrupted elements on the screen; Generate possible actions for each element (such as "click", "input", "expand" etc.); Based on the existing task objectives and contextual semantics, evaluate the rationality and potential value of each action; Select several candidate actions from high-value actions and infer the new task scenarios they may trigger. For example, when the "Settings" button is detected, sub-tasks such as "Turn off notifications" and "Change password" can be associated, and new target intents can be generated accordingly. With the new task as the objective, construct a new initial trajectory and add it to the trajectory pool for use in subsequent iterations.

[0045] This process enables the expansion from a single task to a multi-task space, significantly improving the diversity and coverage of trajectory data.

[0046] The operation process, for example: (1) Start the "Trajectory Extension Agent" to traverse all interactive elements on the current page; (2) Use VLM to evaluate the value of candidate elements, taking into account factors such as semantic meaning, positional weight, and text content; (3) Filter out unexplored interactive elements with high potential; (4) Generate corresponding initial action instructions (such as click, input, etc.) for each candidate element; (5) Automatically infer the potential target intent for each new path (e.g., "redirect to user center", "submit form information", etc.); (6) Initialize a new trajectory record, including the target intent, initial action sequence, and current page state; (7) Store these newly generated trajectories into the trajectory pool as candidate paths for subsequent iterative expansion. Output: One or more new trajectories, including the new task objective, initial action sequence and corresponding environmental state.

[0047] Step 6, Result Evaluation and Trajectory Correction Implementer: Evaluation and filtering correction agent; Function Description: After the action is executed, the evaluation and filtering correction agent initiates a result evaluation process to determine whether the current operation has achieved the preset goal and to decide on subsequent processing strategies. The specific process is as follows: Figure 3 As shown, the process is as follows: (1) Judgment of goal achievement: Check if the current page status meets the predetermined task objectives (such as "Login successful" or "Form submission completed"). If the objective has been achieved, the current trajectory is marked as a complete trajectory, and further expansion of the branch is terminated to prevent waste of resources.

[0048] (2) If incomplete, proceed to the reflection stage: If the goal is not achieved, the process enters a reflection and correction phase, where the current actions are evaluated to determine whether they meet the expected outcome. The evaluation methods include: Compare the actual page changes with the expected changes (e.g., clicking the "Login" button should redirect to the homepage); The Visual-Language Model (VLM) is invoked to analyze the semantics of the current interface and determine whether there are any discrepancies. Check for error messages, loading failures, or pop-up blocking.

[0049] If they match: This action step is included in the current trajectory record; The remaining tasks are then performed by the next analytical decision-making agent to maintain task continuity.

[0050] If inconsistent: Further, the large model is invoked to rematch the task intent and determine whether there is an adjustable target space. a. If it can match the current state: then the trajectory target will be dynamically adjusted (e.g., the original target was "login", now it is changed to "password retrieval"), and the subsequent action sequence will be replanned; b. Unmatchable: The trajectory branch is deemed invalid and abandoned to avoid the accumulation of invalid paths and improve system efficiency.

[0051] The operation process, for example: (1) The "evaluation, filtering and correction agent" determines whether the current page state meets the preset target: If the goal has been achieved, mark the trajectory as "completed" and terminate further expansion of that branch; If the goal is not achieved, then proceed to the reflection phase; (2) During the reflection phase, assess whether the results of the current action meet expectations: If the results are consistent, the action is retained and included in the trajectory record; If the results are inconsistent, then the larger model is invoked to analyze whether there is room for target adjustment: If the current state can match a new reasonable goal (for example, the original goal was "login to the system", but it actually redirects to the "password recovery" page), then the trajectory goal is dynamically adjusted, that is: the original goal is replaced with the new goal (such as "password recovery"), and the branch path is explored. If no reasonable target can be matched, the trajectory branch is deemed invalid and abandoned to prevent the accumulation of invalid paths and improve system efficiency.

[0052] (3) Perform quality screening on all active trajectories, eliminate inefficient or erroneous paths, and retain high-quality candidate trajectories.

[0053] Output: Complete trajectory or terminated branch; updated trajectory pool state.

[0054] Step 7: Iterate until the termination condition is met. enter: All active trajectory records in the current trajectory pool; The latest page state (snapshot and DOM structure) for each track; Historical action sequence and target intent; Preset termination condition parameters (such as maximum number of iterations, threshold N for no new valid trajectories, time limit, etc.).

[0055] Operation process: (1) The "trajectory generation agent" coordinates and controls the entire process, initiating multiple rounds of iterative cycles; (2) In each iteration, the following sub-steps are executed sequentially: Select a trajectory to be expanded from the trajectory pool; The "analysis and decision-making intelligent agent" is invoked to predict the next optimal interaction action based on the current objective; Perform this action and obtain a snapshot of the new page and the updated DOM structure; The execution results are sent to the "Evaluation Filtering and Correction Agent" for validity assessment; If the action is effective but the goal is not achieved, the "trajectory extension agent" is invoked to generate 3 to 5 new candidate paths and stored in the trajectory pool. If the trajectory has achieved its goal, it is marked as "completed" and removed from the active trajectory list; If the trajectory is invalid or cannot match the target, the branch is abandoned to prevent waste of resources.

[0056] (3) The system continuously monitors whether any of the following termination conditions are met: All potential paths have been explored; The number of newly added valid trajectories is zero in N consecutive iterations; The preset time limit or computing resource limit has been reached.

[0057] (4) When any termination condition is met, stop the iteration and enter the final output stage.

[0058] Output: The complete GUI operation trajectory dataset, stored in structured JSON format, includes: The target intent of each trajectory, the historical action sequence, the comparison before and after page state changes, whether the completion mark is set, and the snapshots and DOM information of key nodes.

[0059] The resulting high-quality trajectory set can be used for subsequent automated testing, task modeling, or human-computer interaction research.

[0060] To better understand the technical solution of the present invention, the present invention will be further described below with reference to specific embodiments: Taking the website address of a travel company as an example, this example illustrates the entire process from initial page loading to the final generation of a complete operation trajectory. This embodiment demonstrates the actual operational mechanism of collaborative work among various intelligent agents, conforming to the interaction logic in a real-world web environment. 1. Input Information Target webpage URL: The official website address of a travel company; System Configuration: Browser environment: Chrome browser (supports DevTools Protocol); Visual-Language Model (VLM): A large model deployed on a local or cloud server (such as qwen2.5-VL); Track pool capacity limit: 50 active tracks Iteration termination condition: No new valid trajectories are added for three consecutive rounds. 2. Step 1: Webpage Initialization Input: The official website URL of a travel company; Operation process: (1) Initialize the agent to open the official website of a tourism company and obtain a complete screenshot of the homepage; (2) Use Chrome DevTools Protocol to extract the DOM tree structure and identify all interactive elements; (3) Traverse and record the key element attributes: Element types: button, input, a, div, etc. Attribute information: `id="searchBtn"`, `class="search-box"`, `text="train ticket"`; Position coordinates: `(x=400, y=150)`; (4) Call VLM to evaluate the value of each element, focusing on: Navigation buttons containing keywords such as "train tickets," "air tickets," and "hotels"; Highly interactive components such as search boxes and city selection dropdown menus; (5) 25 high-value elements were selected, including: "Train Tickets" navigation button; departure point input box; arrival point input box; search button; (6) Perform these actions in sequence (such as clicking the "train ticket" button), capture a new page snapshot and update the DOM structure after each operation, and then return to the homepage to continue the next candidate action.

[0061] Output: Page snapshots (homepage and multiple subpages); initial DOM tree structure; list of interactive elements and their corresponding actions (e.g., "click the 'train ticket' button"); page state snapshots after multiple interactions.

[0062] 3. Step Two: Interactive Element Analysis and Action Decision Input: The set of action instructions from the previous step (containing 25 candidate operations); Operation process: (1) Invoke VLM for each execution path to infer the target intent: Click "Train Tickets" → Infer target: "Enter the train ticket search page"; Input "initial location" to departure location → infer target: "Set departure city to a certain location"; (2) Initialize the trajectory recording mechanism and record: Target Intent; Historical action sequence (e.g., "click on train ticket → enter initial location"); Comparison of page status before and after changes (e.g., changes to the content of the search box); (3) Store the above trajectories in the trajectory pool to form multiple initial paths for parallel exploration; Output: Multiple initial trajectory records (e.g., "Query initial location → destination high-speed rail ticket"); The trajectory pool contains 25 trajectories to be expanded.

[0063] 4. Step 3: Trajectory Generation and Dynamic Extension Input: A trajectory from the trajectory pool (e.g., "Click on the train ticket → Enter the initial location"); Operation process: (1) The analytical decision-making agent receives the current trajectory target: "Query the high-speed rail ticket from the initial location to the destination"; (2) The current page is a train ticket search page. VLM analysis found that the "Destination" input box is empty; (3) Predict the next optimal action: "Enter the specific 'destination' in the destination input box"; (4) Perform this action to obtain a new page snapshot and update the DOM structure; (5) Feed the results back to the evaluation, filtering and correction agent.

[0064] Output: Updated trajectory records (new action: "Input destination"); new page snapshot and DOM structure; action execution results are used for subsequent evaluation.

[0065] 5. Step Four: Trajectory Expansion and Branch Initialization Input: Current page (origin and destination already filled in); Operation process: (1) Start the trajectory extension agent to identify potential interactive elements on the current page; (2) The VLM assessment identified the following high-value elements: "Search" button; departure date selector; passenger number selection box; (3) Select 3 candidate actions: Click the "Search" button; select "Tomorrow" as the departure date; set the number of passengers to "1 person"; (4) Generate a new trajectory for each action: Track A: Click "Search" → Target: "Display Train List"; Track B: Select Date → Target: "Set Travel Time"; (5) Add the three new trajectories to the trajectory pool.

[0066] Output: Three new track records; updated track pool (28 active tracks in total).

[0067] 6. Step Five: Evaluate Filtering and Trajectory Correction Input: The page status after clicking "Query"; Operation process: (1) Assess whether the objective has been achieved: "Is the list of high-speed train services from the initial location XXX to the destination XXX displayed?"; Yes → Mark the trajectory as "completed" and terminate the branch; (2) If the goal is not achieved (e.g., redirection fails or page is abnormal): Entering the reflection phase; Determine whether the result of the action meets expectations; If it does not meet the requirements, check if there is room for adjustment of the target: If the current page is a login page, the target will be dynamically adjusted to "login first, then query"; If no reasonable target can be matched, the trajectory is abandoned.

[0068] (3) Perform quality screening on all trajectories and retain valid paths.

[0069] Output: Corrected trajectory records; optimized trajectory pool (only valid paths are retained).

[0070] 7. Step Seven: Iterate until the termination condition is met. Input: Current trajectory pool (containing multiple active trajectories); Operation process: (1) The system initiates multiple iterations: Each round selects one trajectory and executes prediction → action → evaluation → extension; Continuously generate new paths to gradually approach the final goal; (2) In the 10th iteration, a certain trajectory successfully completed the task of "querying the initial location → destination high-speed rail ticket"; (3) The system detects that no new valid trajectories are added for three consecutive rounds, thus meeting the termination condition; (4) Stop the iteration and summarize all completed trajectories.

[0071] This embodiment demonstrates the complete operational flow of the present invention in a real-world, complex web scenario. Through a closed-loop mechanism of "initialization → analysis → execution → expansion → evaluation → iteration," the system can autonomously explore, dynamically adjust its objectives, and efficiently converge to an effective operation path, ultimately generating a high-quality GUI operation trajectory dataset.

[0072] This method is applicable to various scenarios such as automated testing of web applications, task learning of intelligent robots, and human-computer interaction modeling, and has good versatility and practicality.

[0073] The innovative aspects of this invention are as follows: 1. Multi-agent collaborative architecture design: The system adopts a hierarchical agent architecture, including an initialization agent, a trajectory generation agent, and an evaluation and correction agent. Each agent has a clear responsibility and achieves closed-loop collaboration through state information sharing, supporting the automated exploration and execution of complex web page interaction tasks.

[0074] 2. Automatic generation and dynamic adjustment of target intent: The system does not require manual setting of specific operation paths. During execution, the large model automatically infers the target intent of the current trajectory based on the action sequence. When the execution result does not meet expectations, it can enter the reflection stage to determine whether it can match the current page state to dynamically adjust the target, avoid path rigidity, and enhance the system's adaptability and robustness.

[0075] 3. Trajectory pool-driven multi-path parallel exploration mechanism: Each initial trajectory is stored in the trajectory pool, supporting the simultaneous expansion of multiple paths to form an exploration mode of "main trunk + branches", effectively preventing a single path from getting stuck in a dead end and significantly improving the task completion rate and exploration efficiency.

[0076] 4. Reflective trajectory correction strategy: Introducing a "reflection" mechanism, the system does not terminate directly after an action fails. Instead, it calls the large model to determine if there is a reasonable new target. If a match is found, the target is dynamically adjusted and the exploration continues. Otherwise, the branch is abandoned, effectively preventing the accumulation of invalid paths and improving the overall resource utilization rate.

[0077] 5. State-aware dynamic path expansion: Traverse all unexplored interactive elements on the current page, sort them by value using VLM, select 3-5 high-potential elements to generate new trajectories, and achieve a gradual and in-depth exploration from local to global, taking into account both breadth and depth.

[0078] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating GUI-Agent trajectory data based on multi-agent cooperative reasoning, characterized in that, Includes the following steps; S1. Open the target webpage, obtain a complete screenshot of the current page, and use front-end parsing tools to extract the page DOM structure, identify and record the attribute information of all interactive elements, and construct the initial page state graph. S2, based on the initial page state graph, semantic analysis and value assessment of interactive elements are performed using the visual-language model (VLM) to generate a corresponding set of initial action instructions; S3, execute the actions in the initial action instruction set, record the page state changes before and after each action execution, and automatically infer the target intent for each execution path to form an initial trajectory record and store it in the trajectory pool; S4. For a current trajectory in the trajectory pool, combine the target intent corresponding to the trajectory and the current page state to predict and execute the next optimal interaction action, and update the trajectory record at the same time. S5: Analyze the unexplored interactive elements on the current page, combine VLM to generate potential new action commands and infer the new target intent, form a new initial trajectory record and add it to the trajectory pool; S6, After the action is completed, determine whether the action execution result of the current trajectory has achieved the preset goal, and decide on the subsequent processing strategy; S7. Repeat steps S4 to S6 to perform parallel or iterative execution, expansion and evaluation of multiple trajectories in the trajectory pool until the preset iteration termination condition is met, and output the final GUI operation trajectory dataset.

2. The GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning according to claim 1, characterized in that, In step S1, the attribute information of the interactive element includes the type, position, identifier, and text content attribute of the interactive element.

3. The GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning according to claim 2, characterized in that, In step S2, the semantic analysis and value assessment include the following processes: Based on the semantic information, layout position and text content of interactive elements, all interactive elements are evaluated using a visual-language model (VLM), and a preset number of elements with the highest evaluation results are selected as the basis for generating the initial action instructions. The initial action instructions include clicking, entering text, selecting, or scrolling.

4. The GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning according to claim 3, characterized in that, In step S3, the action is executed by directly calling page elements for interaction through a standard browser automation interface; the automatically inferred target intent is generated by VLM based on the sequence of executed actions; the initial trajectory record includes the target intent corresponding to the trajectory, the historical action sequence, and page state changes.

5. The GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning according to claim 4, characterized in that, Step S4 specifically includes the following steps: S41, input the target intent, page snapshot and DOM structure information of the current trajectory into VLM, and VLM outputs the next operation instruction that best matches the current context; S42 executes the output next operation instruction in a real or simulated environment; S43, after execution, the system automatically captures the operation result feedback and updates the latest action item in the trajectory record; the operation result feedback includes page jump and element state change.

6. The GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning according to claim 5, characterized in that, In step S5, generating potential new action instructions includes the following process: Construct a set of non-interactive elements on the current page, generate possible actions for each element using VLM, evaluate the rationality and potential value of each action based on the existing task context, and select the actions with the highest potential value as candidates.

7. The GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning according to claim 6, characterized in that, In step S6, determining whether the action execution result of the current trajectory has achieved the preset goal specifically includes the following process: Check whether the current page state meets the semantics of the task objective, or call VLM to perform semantic analysis on the current interface to determine the objective completion status; If the preset goal is achieved, the corresponding trajectory is marked as a complete trajectory and the expansion is terminated; If the preset goal is not achieved, assess whether the action result meets expectations: If the match is found, the action is retained and step S4 is executed; if the match is not found, the target intent is dynamically adjusted according to the current page state. If a new target is found, the trajectory target is updated and the exploration continues; otherwise, the subsequent exploration of the current trajectory is abandoned. The dynamic adjustment of target intent includes calling a large language model to match the current page state with possible task intents.

8. The GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning according to claim 7, characterized in that, In step S7, the iteration termination condition includes at least one of the following: all potential paths have been explored, no new valid trajectories are added in N consecutive iterations, or the preset time limit or computing resource limit is reached.

9. A GUI-Agent trajectory data generation system based on multi-agent cooperative reasoning, used to implement the GUI-Agent trajectory data generation method based on multi-agent cooperative reasoning as described in any one of claims 1-8, characterized in that, The GUI-Agent trajectory data generation system based on multi-agent collaborative reasoning includes: The initialization module is used to load the target webpage, analyze interactive elements, generate and execute initial action instructions to form multiple initial trajectory records; The trajectory pool is used to store and manage the initial trajectory records and the trajectories generated during the iteration process; The trajectory generation agent module is used for iterative processing of the trajectory pool, specifically including: The analysis and decision-making unit is used to predict the next interaction action based on the target intent and the current page state. The trajectory extension unit is used to mine potential interactions based on the current page state and generate new trajectories. The evaluation filtering and correction unit is used to evaluate the results of actions and to make decisions such as goal achievement judgment, trajectory correction, or abandonment.

10. The GUI-Agent trajectory data generation system based on multi-agent cooperative reasoning according to claim 9, characterized in that, The initialization module includes an analysis and decision-making agent module and a trajectory initialization agent module; the analysis and decision-making unit in the trajectory generation agent module shares an analysis and decision-making agent module with the initialization module; the agent modules communicate with each other through a message passing mechanism.

Images