Gui agent task planning method and system based on multi-agent cooperation

Abstract

This invention belongs to the field of artificial intelligence technology, specifically relating to a GUIAgent task planning method and system based on multi-agent collaboration. The method includes: S1, constructing multi-agent collaborative data: through multi-agent collaboration and feedback control closed loop, automatically generating training data that has been verified, repaired, and optimized by rules; the multi-agent includes a training sample generation module, a data quality verification module, a data repair module, and a rule extraction module; S2, using the endpoint prediction data format and model training method, based on unique offline map nodes, constructing training samples with endpoint nodes as prediction targets, and using a no-thinking training method to fine-tune the pre-trained large language model to obtain the fine-tuned model; S3, constructing a training set, a same-domain test set, and a cross-domain test set, and performing performance evaluation and generalization capability verification on the fine-tuned model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a GUI Agent task planning method and system based on multi-agent collaboration. Background Technology

[0002] With the rapid development of artificial intelligence technology, the application of intelligent agent systems in online and private deployment scenarios is increasing. In general intelligent agents, especially in the field of office automation, the agent needs to understand the user's natural language input and, combined with the application's offline map, perform correct task planning to guide the execution of automated processes. The core of GUI Agent (Graphical User Interface Agent) task planning is: given the user's natural language input for a single application and the application's offline map, the model needs to output the correct execution steps to fulfill the user's intent. Currently, this task mainly relies on general-purpose large language models such as GPT.

[0003] However, existing methods have the following fundamental problems at the technical principle level: Question 1: The fundamental difference between language modeling and programmatic planning. Large language models like GPT are trained based on a language modeling objective function, with the optimization goal of maximizing the conditional probability of the next lexical unit. Essentially, this aims to generate fluent text sequences, not to perform programmatic task planning. Language modeling focuses on local coherence, while task planning requires searching for the globally optimal path. This mismatch in objective functions leads to structural limitations in the model's planning capabilities, and even fine-tuning cannot fundamentally solve the problem of insufficient planning ability.

[0004] Question 2: Technical limitations at the training data level. High-quality "step-supervised data" is extremely scarce, and manual annotation requires a deep understanding of business logic, which is very costly; it needs to cover scenarios of varying complexity, and manual creation makes it difficult to ensure data consistency and integrity; the lack of automated data quality verification mechanisms leads to low-quality data entering the training set, limiting the learning ceiling of the model.

[0005] Problem 3: Structural flaws in the reasoning method. Existing methods use autoregressive generation to output the complete execution path. Autoregressive generation is a local decision-making process performed on a word-by-word basis, which cannot perform global optimal path search. This leads the model into the dilemma of "local optimum ≠ global correctness". It may make seemingly reasonable but overall wrong decisions in intermediate steps. The errors accumulate in subsequent steps, eventually causing the entire path planning to fail.

[0006] Question 4: The True Upper Limit of Inference Training Methods. Existing inference training methods, such as Chain-of-Thought (CoT), NoThink, RLHF, and GRPO, while improving model interpretability and local consistency, do not fundamentally solve the "planning" problem. These methods mainly optimize the expression of the inference process and the coherence of local logic, rather than the global planning capability itself, and therefore have a clear technical limit in improving model accuracy.

[0007] Therefore, there is an urgent need for a GUI Agent task planning method and system that can fundamentally solve the mismatch between language modeling and procedural planning, automatically generate high-quality training data, optimize training objective design, and improve global planning capabilities. Summary of the Invention

[0008] This invention aims to overcome the problems of existing GUI Agent task planning methods, such as the inherent mismatch between language modeling and procedural planning, the scarcity of high-quality training data, structural defects in autoregressive generation, and the technical limitations of inference training methods. It provides a GUI Agent task planning method and system based on multi-agent collaboration that can fundamentally solve the mismatch between language modeling and procedural planning, automatically generate high-quality training data, optimize training objective design, and improve global planning capabilities.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A GUI Agent task planning method based on multi-agent collaboration includes the following steps; S1, Constructing multi-agent collaborative data: Through multi-agent collaboration and feedback control closed loop, training data that has been verified, repaired and optimized by rules is automatically generated; the multi-agent includes a training sample generation module, a data quality verification module, a data repair module and a rule extraction module; S2 adopts the data format and model training method of endpoint prediction. Based on unique offline map nodes, it constructs training samples with endpoint nodes as prediction targets and uses the NoThink training method to fine-tune the pre-trained large language model to obtain the fine-tuned model. S3 constructs training sets, same-domain test sets, and cross-domain test sets to evaluate the performance and verify the generalization ability of the fine-tuned model.

[0010] Preferably, step S1 includes the following steps: S11, Initialize the data source, which includes manually labeled seed data and offline map structured information from the application; S12, through the training sample generation module, generates training samples of different dimensions and complexities based on the offline map; S13, through the data quality verification module, performs format verification on the generated training samples and outputs data that does not meet the requirements and corresponding abnormal feedback information; S14, the data repair module regenerates the data that does not meet the requirements based on the abnormal feedback information, and the generated data is then input into the data quality verification module for verification. S15, Calculate the verification failure rate. When the verification failure rate is higher than the preset threshold, proceed to step S16; when the verification failure rate is lower than the preset threshold, proceed to step S17. S16, through the rule extraction module, the abnormal feedback information is extracted into rules, and the extracted rules are injected back into the training sample generation module to update the generation strategy, forming a feedback control closed loop; S17. Manually label the remaining data that does not meet the requirements to complete the construction of the training dataset.

[0011] Preferably, step S16 further includes the following steps: Deeply analyze multiple rounds of abnormal feedback information to summarize the constraint rules and error patterns of data generation; dynamically update the extracted rules into the instructions of the training sample generation module; iteratively execute the training sample generation, data quality verification, and data repair steps until the verification failure rate is reduced to below the preset threshold.

[0012] Preferably, step S2 includes the following steps: S21, Offline map preprocessing: Perform node uniqueness processing on the offline map to ensure the uniqueness of all node names in the offline map; S22, Training Sample Construction: Construct training samples that include the system instruction domain, user input domain, and model output domain; S23, Model Training: The NoThink training method is adopted, which enables the pre-trained large language model to directly map from user input and offline map structure to the endpoint node, establishing a direct correspondence between "intent and result". S24, Endpoint Node Post-processing and Path Reconstruction: Based on the endpoint nodes predicted by the fine-tuned model, a complete executable path is generated by backtracking the tree structure of the offline map.

[0013] Preferably, in step S21, the node uniqueness processing refers to handling the case of duplicate nodes on different pages in the offline map through path encoding or renaming, in order to ensure that each node has a unique identifier.

[0014] Preferably, in step S22, the instruction domain is used to define the model role as a GUI Agent expert, and specify the task as understanding the user's intent and locating the most matching endpoint node in the offline map; The user input field contains user natural language commands and a structured offline map description. The user natural language commands describe the user's operational intent. The offline map uses a nested dictionary structure to represent a tree structure. The dictionary keys are operation names, and the dictionary values ​​are child node dictionaries, forming a complete path tree from the root node to the leaf node. The model output field contains a unique endpoint node identifier, which is a unique node identifier after node uniqueness processing.

[0015] Preferably, step S24 includes the following steps: S241, Node localization: Based on the unique endpoint node identifier output by the fine-tuned model, accurately locate the target node in the offline map after unique processing; S242, Path backtracking: Starting from the located target node, traverse upwards along the parent node pointers of each node to the root node to obtain the reverse path sequence; S243, Sequence Reversal: Reverse the reversed path sequence to generate a complete execution path from the root node to the target node; S244, Integrity Verification: Through the structural constraints of the offline map, ensure that the complete execution path generated in step S243 is consistent with the node hierarchy of the corresponding offline map.

[0016] Preferably, step S3 includes the following steps: S31, Training set construction: Directly use the training dataset generated in step S1; the training dataset covers different application software and scenarios of different complexities; S32, Test Set Construction: Constructing same-domain and cross-domain test sets; The same-domain test set includes application software that appeared in the training set but used different user inputs or offline map versions, and is used to evaluate the performance of the fine-tuned model in similar scenarios. The cross-domain test set includes offline maps of application software that did not appear in the training set, used to verify the generalization ability of the fine-tuned model. S33, Model Performance Evaluation: The performance of the fine-tuned model is evaluated using endpoint accuracy, path completeness, and generalization accuracy as evaluation metrics.

[0017] Preferably, in step S33, the accuracy of the endpoint node is the proportion of the final execution node output by the model that is completely consistent with the standard answer; The path completeness rate is the proportion of all steps in the reconstructed complete execution path that are consistent with the node hierarchy of the corresponding offline map. The generalization accuracy is the accuracy on the cross-domain test set.

[0018] This invention also provides a GUI Agent task planning system based on multi-agent collaboration, comprising: A multi-agent collaborative data construction module is used to automatically generate training data that has been verified, repaired, and optimized by means of multi-agent collaboration and feedback control closed loop; the multi-agent includes a training sample generation module, a data quality verification module, a data repair module, and a rule extraction module; The endpoint prediction and model training module is used to construct training samples with endpoint nodes as prediction targets based on unique offline map nodes, using the endpoint prediction data format and model training method. It also uses the NoThink training method to fine-tune the pre-trained large language model to obtain the fine-tuned model. The model evaluation and generalization testing module is used to construct training sets, same-domain test sets, and cross-domain test sets to evaluate the performance and verify the generalization ability of the fine-tuned model.

[0019] Compared with the prior art, the beneficial effects of this invention are: (1) Through the multi-agent collaborative mechanism and feedback control closed-loop mechanism, the automatic generation, cleaning and optimization of training data are realized. The rule extraction module can automatically extract rules according to the abnormal feedback information and inject them back into the training sample generation module, realizing the adaptive optimization and self-evolution capability of the system. When the failure rate of verification is higher than the set threshold, the closed-loop optimization is automatically triggered, which significantly improves the data quality and production efficiency and reduces the cost of manual annotation; (2) Through the unique processing of offline map nodes, the problem of duplicate node names is solved, ensuring the uniqueness of all node names in the offline map, and providing a technical basis for endpoint prediction and path reconstruction; (3) Through the NoThink training method and the innovative method of endpoint prediction data format, the model directly establishes the mapping relationship of "intent -> result", skips the intermediate inference steps, and avoids the autoregressive generation of the model. The local optimum problem simplifies the model's learning objective and avoids redundancy and error accumulation caused by the complete path output. Compared with the traditional full path generation method, the accuracy is improved by more than 4 percentage points; (4) Through the end node post-processing mechanism, starting from the end node output by the model, it traverses upwards along the parent node pointer to the root node, and after obtaining the path sequence, it reverses it. It can automatically generate a complete execution path based on the end node output by the model, which simplifies the training objective and ensures the path integrity in actual application. The structural constraints of the offline map forcefully ensure the logical compliance of the generated path and completely eliminate the "illusion" or "interruption" problem that may occur in the intermediate steps of autoregressive generation; (5) Through multi-dimensional data production and generalization test design, the model's cross-application generalization ability is improved, enabling the model to accurately plan paths on offline maps of application software that have not been trained. The fundamental reason for the generalization ability is that the model learns the "semantic and functional point matching logic", that is, understands the semantic relationship between the user's natural language intent and the functional nodes in the offline map, rather than memorizing the specific menu positions or path structures, so that the model can transfer the learned matching logic to new application software. Attached Figure Description

[0020] Figure 1 This is a flowchart of a GUI Agent task planning method based on multi-agent collaboration in this invention; Figure 2 This is a flowchart illustrating the multi-agent collaborative data construction process in this invention. Figure 3 This is a flowchart of the feedback control closed-loop mechanism of the rule extraction module in this invention; Figure 4 This is a flowchart of offline map preprocessing in this invention; Figure 5 This is a flowchart of a data format method for endpoint prediction in this invention; Figure 6 This is a flowchart of the post-processing procedure for the endpoint node in this invention. Detailed Implementation

[0021] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0022] like Figure 1 As shown, the present invention provides a GUI Agent task planning method based on multi-agent collaboration, including the following steps; 1. Constructing multi-agent collaborative data: Through multi-agent collaboration and feedback control closed loop, training data that has been verified, repaired and optimized by rules is automatically generated; the multi-agent includes a training sample generation module, a data quality verification module, a data repair module and a rule extraction module; 2. Using the data format and model training method of endpoint prediction, based on unique offline map nodes, training samples with endpoint nodes as prediction targets are constructed, and the pre-trained large language model is fine-tuned using the NoThink training method to obtain the fine-tuned model; 3. Construct training sets, same-domain test sets, and cross-domain test sets to evaluate the performance and verify the generalization ability of the fine-tuned model.

[0023] For step 1, as Figure 2 As shown, the specific process includes the following: Step 1-1: Initialization Phase We collect basic data for manually annotated offline maps, including structured information from offline maps of various applications (such as digital shopping platforms and e-commerce platforms). The offline maps are represented using a tree structure. Each node consists of a dictionary whose keys are operation names (which may contain parameter placeholders such as <parameter X>), and the dictionary values ​​are child node dictionaries, forming a complete path tree from the root node to the leaf nodes.

[0024] Step 1-2: Training Sample Generation Module The prompt design for the training sample generation module follows these principles: the module's role is clearly defined as a GUIAgent, an expert in creating training data. It is required to generate training data based on the given application software's offline map and common scenarios. Samples of varying complexity need to be generated, including single short paths, single long paths, and multiple paths. User input should describe the specific operation intent in natural language. The generated training samples should include user input, offline map structure representation, and endpoint node annotations. The output format must conform to structured data specifications.

[0025] The training sample generation module generates data for an e-commerce platform application, producing an example: A single, shorter path: When a user enters "search for mobile phones on the electronics and digital shopping platform", the corresponding path is "open the electronics and digital shopping platform webpage" → "search for product <parameter 1>", and the final node is "search for product <parameter 1>".

[0026] A single, relatively long path: When a user enters "Contact the customer service of a certain eyewear official flagship store on the electronic digital shopping platform and ask what models of sunglasses they have," the corresponding path is "Open the electronic digital shopping platform webpage" → "Click to enter the online customer service page" → "Contact the store's customer service <parameter 18> and send a message <parameter 19>", and the final node is "Contact the store's customer service <parameter 18> and send a message <parameter 19>".

[0027] Multiple paths: The user inputs "search for mobile phones on the electronics shopping platform and sort them by price from low to high, then search for computers and filter out computers priced below 5000", which involves two offline map branches.

[0028] The prompt word for the training sample generation module undergoes a warm-up and instruction optimization phase: the first run generates initial data, analyzes the results, and optimizes the prompt word; the second run generates data again and further optimizes; the third run serves as the official version. The warm-up and instruction optimization phases ensure that the instruction design of each module reaches its optimal state, laying the foundation for subsequent data generation.

[0029] Steps 1-3: Data Quality Verification Module The prompt design for the data quality verification module follows these principles: clearly defining the module's role as a data quality verification expert, requiring checks on data format compliance, node names, and offline location. Figure 1 Consistency, parameter numbering consistency with supplementary knowledge, logical integrity, etc., and specific abnormal feedback information should be output for data that does not meet the requirements.

[0030] The data quality verification module verifies the data generated by the training sample generation module. Example output: Data that meets the requirements is marked as "passed" and added to the training set.

[0031] Data that does not meet the requirements: Mark it as "Not Passed" and output abnormal feedback information, such as the node name "Open Enterprise Communication Software" being inconsistent with "Start Enterprise Communication Software" in the offline map, or "Parameter Number <Parameter 18>" not existing in the supplementary knowledge, etc.

[0032] Steps 1-4: Data Repair Module The prompt design for the data repair module follows these principles: clearly define the module's role as a data repair expert, require the regeneration of compliant data based on anomaly feedback, maintain the user's intent, correct all non-compliant items, ensure that node names are completely consistent with the offline map, and ensure that parameter numbers are strictly consistent with supplementary knowledge.

[0033] The data repair module regenerates the data based on the anomaly feedback information output by the data quality verification module. For example, if the node name in the original data is "Open Enterprise Communication Software" while the offline map shows "Start Enterprise Communication Software", the data repair module will correct it to the correct node name.

[0034] Steps 1-5: Iterate The data regenerated by the data repair module is then input into the data quality verification module for verification. The verification failure rate is calculated as follows: If the failure rate of the verification is higher than 10%, proceed to steps 1-6.

[0035] If the failure rate of the verification is less than 10%, proceed to steps 1-7.

[0036] Steps 1-6: Rule Extraction Module The prompt design for the rule extraction module follows these principles: clearly defining the module's role as a rule extraction expert, requiring analysis of the anomaly feedback information from the first two rounds, and summarizing the specific rules for data generation, including node names and offline locations. Figure 1 Consistency rules, parameter numbering and supplementary knowledge consistency rules, path reachability rules, and other common error modes and corresponding correction methods.

[0037] like Figure 3As shown, the rule extraction module constitutes the core component of the feedback control closed loop: when the failure rate of verification exceeds 10%, the rule extraction module performs in-depth analysis of the abnormal feedback information from the first two rounds, extracting the constraint rules and error patterns generated from the data; the extracted rules are then injected back into the Prompt function of the training sample generation module to dynamically update the generation strategy and achieve adaptive optimization of the system; the rule extraction module generates data for the third round, and the generated data is input again into the data quality verification module for verification, returning to steps 1-5. This feedback control closed loop mechanism enables the system to automatically adjust the generation strategy according to the error patterns, achieving self-optimization and continuous improvement, demonstrating the system's adaptability and self-evolutionary capabilities.

[0038] Steps 1-7: Manual annotation When the failure rate of validation is below 10%, the remaining non-compliant data is manually annotated. Manual annotation mainly handles problems that are difficult to automatically correct, such as complex logical errors and semantic comprehension biases.

[0039] The result is a high-quality training dataset that covers different application software and scenarios of varying complexity.

[0040] Step 2 specifically includes the following process: Step 2-1: Offline map preprocessing Ensuring node uniqueness in offline maps is a prerequisite for destination prediction. The offline map preprocessing workflow is as follows: Figure 4 As shown, offline maps may contain nodes with duplicate names. For example, different pages may have a "back" button. If these nodes are not uniquely named, the model will not be able to accurately distinguish and locate the target node. The system needs to ensure the uniqueness of all node names in the offline map, ensuring that each node has a unique identifier. Node uniqueness can be achieved through path encoding, renaming, or other methods, as long as the node names in the offline map are unique. In practical applications, if the given offline map itself does not have duplicate names, the original node names can be used directly.

[0041] By uniquely identifying each node in the offline map, the endpoint node output by the model can be accurately located, providing a foundation for subsequent reverse backtracking algorithms.

[0042] Step 2-2: Construct training samples like Figure 5As shown, the data structure of the training samples consists of three parts: (1) System instruction field, which defines the model role as GUI Agent expert, and clarifies the task as understanding user intent and locating the most matching endpoint node in the offline map; (2) Input field, which contains user natural language instructions and structured offline map descriptions, where user natural language instructions describe the user's operation intent, and the offline map adopts a nested dictionary structure to represent a tree structure, where the key of the dictionary is the operation name (which may contain parameter placeholders such as <parameter X>), and the value of the dictionary is the child node dictionary, forming a complete path tree from the root node to the leaf node; (3) Output field, which contains a unique endpoint node identifier, which is a unique node identifier after node uniqueness processing, rather than a complete execution path.

[0043] The construction of training samples follows these design principles: (1) The Input domain contains the user’s natural language input and a complete structured representation of the offline map, explicitly requiring the model to understand the user’s intent and locate the most matching endpoint node in the offline map.

[0044] (2) The Output field contains only a unique endpoint node identifier and does not contain the complete execution path. The node identifier must be completely consistent with the node identifier in the offline map that has been uniquely processed.

[0045] (3) The target of the training sample is the endpoint node in the offline map. This node can be any node in the offline map, as long as it can fulfill the user's intent.

[0046] (4) If a matching node cannot be found in the offline map, output the preset failure flag.

[0047] (5) The NoThink training method is adopted. The model directly establishes the mapping relationship of "intent -> result", that is, it directly maps from the user's natural language input and offline map structure to the endpoint node, skips the intermediate reasoning steps, and directly learns to output the final node identifier. During evaluation, only the accuracy of the final node identifier is considered.

[0048] Steps 2-3: Model Training This embodiment uses a pre-trained large language model for fine-tuning, such as the Qwen3-8B model. The relevant parameters that need to be set during training include learning rate, batch size, gradient accumulation steps, number of training epochs, optimizer type, learning rate scheduling strategy, gradient pruning parameters, and mixed precision training method. The loss function is cross-entropy loss, and only the loss of the final node identifier portion is calculated.

[0049] During training, the model adopts the NoThink training method, which directly establishes the mapping relationship between "intent -> result". It directly locates the final execution node from user input and offline map, skipping intermediate inference steps, thus avoiding the complexity of the complete path output and the local optimum problem caused by autoregressive generation.

[0050] Steps 2-4: Post-processing of the endpoint node The endpoint node output by the model is post-processed to generate a complete execution path. Since each node in the offline map has been uniquely identified through step 2-1, the system can accurately locate the node based on the unique node identifier output by the model. Figure 6 As shown, the specific process is as follows: (1) Node localization: Locate the node in the tree structure of the offline map based on the unique identifier of the endpoint node. For example, if the model outputs "Contact store customer service <parameter 18> and send message <parameter 19>", the system can accurately locate the node in the offline map using the unique node identifier.

[0051] (2) Path backtracking: Starting from the endpoint node output by the model, traverse upwards along the parent node pointer of each node until the root node is reached, thus obtaining the path sequence from the endpoint to the root node. Then, reverse the reverse sequence to generate the complete execution path from the root node to the endpoint node.

[0052] (3) Path output: Output the complete execution step sequence after reversal. For example, for the endpoint node "Contact store customer service <parameter 18> and send message <parameter 19>", the complete path generated by path backtracking and sequence reversal is: "Open the electronic digital shopping platform webpage" → "Click to enter the online customer service page" → "Contact store customer service <parameter 18> and send message <parameter 19>".

[0053] This path reconstruction mechanism ensures that the execution path can be fully recovered from the simplified endpoint output, achieving a balance between training simplification and application integrity. Through node uniqueness processing and path reconstruction, the system simplifies the model's training objective (learning only the output endpoint) while guaranteeing that a complete execution path can be obtained in practical applications, thus solving the technical requirement of GUIAgent execution requiring a complete path.

[0054] Step 3 specifically includes the following process: Step 3-1: Training Set Construction The high-quality training dataset generated in step 1 covers different applications and scenarios of varying complexity, including training samples of various types such as single short paths, single long paths, and multiple paths. The training set includes interaction data from different scenarios such as e-commerce platforms, social media apps, and food review apps.

[0055] Step 3-2: Test Set Construction The test set is divided into two parts: (1) The first part is the same domain test set, which includes application software that has appeared in the training set (such as electronic digital shopping platform, e-commerce platform), but uses different offline map versions or different user inputs to evaluate the model’s performance in similar scenarios.

[0056] (2) The second part is the cross-domain test set, which contains offline maps of application software that did not appear in the training set. This is used to verify the generalization ability of the model and ensure that the model can accurately plan paths on application software that it has never seen before.

[0057] Step 3-3: Model Performance Evaluation The following evaluation indicators will be used: (1) Endpoint node accuracy: The proportion of the final execution node output by the model that is completely consistent with the standard answer.

[0058] (2) Path completeness rate: The proportion of all steps in the complete execution path generated by post-processing that are reachable in the offline map.

[0059] (3) Generalization accuracy: The accuracy on cross-domain test sets, used to measure the model’s ability to adapt to untrained application software.

[0060] After training the model on the training set, its performance is evaluated on both the same-domain test set and the cross-domain test set. The evaluation process includes: the model outputs the endpoint node, generating the complete execution path through post-processing, comparing it with the standard answer, and calculating various evaluation metrics.

[0061] Based on the technical solution of this invention, the following case scenario illustrates the implementation process of this invention in practical applications. The specific application implementation scheme is as follows: This embodiment uses the task of "users wanting to complete product retrieval, filtering, and adding to cart on an e-commerce platform application" as an example to illustrate the end-to-end implementation process of the method of the present invention in real business. This process includes four stages: training dataset preparation, model fine-tuning, model inference, and result post-processing.

[0062] Preparation of training dataset Step A1: Data Source Initialization. Offline map structured information from multiple application software is selected as the basic corpus, and a small number of manually labeled seed samples are introduced as initial constraints. The offline map is represented using a tree-like nested dictionary structure, where nodes represent executable operations and edges represent reachability relationships.

[0063] Step A2: Automatic Generation of Training Samples by Multi-Agents. The training sample generation module automatically generates samples of varying complexity based on offline maps, covering scenarios of single short paths, single long paths, and multi-branch paths. Sample fields include the system command domain, user input domain (user intent + offline map), and model output domain (endpoint node identifier). For example, for an e-commerce platform scenario, a sample can be constructed: "Open e-commerce platform webpage - search for product <parameter 1> - click range - set minimum price range to <parameter 2> - set maximum price range to <parameter 3> - click OK," with the endpoint node identifier being the unique node identifier corresponding to "click OK."

[0064] Step A3: Quality Verification and Repair. The data quality verification module verifies the generated samples for format standardization, node consistency, parameter consistency, and path reachability. For samples that fail the verification, the data repair module performs regeneration based on the anomaly feedback, maintaining the user's intent and correcting erroneous fields.

[0065] Step A4: Feedback Control Closed-Loop Optimization. When the failure rate exceeds a preset threshold of 10%, the rule extraction module performs attribution analysis on multiple rounds of abnormal feedback, forms executable rules, and injects them back into the prompt words of the training sample generation module to iteratively update the generation strategy; when the failure rate drops below the threshold, the remaining complex samples are manually labeled to complete the construction of a high-quality training dataset.

[0066] Model fine-tuning Step B1: Offline map preprocessing. Perform node uniqueness processing on the offline map to eliminate ambiguity caused by duplicate node names across pages and ensure that each node has a unique identifier.

[0067] Step B2: Constructing Endpoint Prediction Samples. Using the endpoint prediction data format, the model output target is set to a "unique endpoint node identifier," instead of directly outputting the complete path. This focuses the learning objective on the mapping from "user intent to endpoint node."

[0068] Step B3: NoThink fine-tuning. A Qwen pre-trained model is used for supervised fine-tuning, employing the cross-entropy loss function to calculate the loss only at the position corresponding to the endpoint identifier. During training, cue words and hyperparameters are set, and the weights corresponding to the optimal metrics on the validation set are used as the final model.

[0069] Step B4: Convergence Determination and Model Consolidation. When the accuracy of the endpoint node reaches a stable improvement on the validation set and the loss converges, the fine-tuned parameters are consolidated to form a deployable model.

[0070] Model Inference Step C1: Input Construction. During the online phase, the system receives the user's natural language request and the corresponding offline map description for the application. The input is then concatenated using a training isomorphic template to ensure consistency between training and inference. For example, the user input could be "Search for sneakers on an e-commerce platform and add the top-selling item to your cart."

[0071] Step C2: Endpoint Node Prediction. Feed the input into the fine-tuned model and output the unique endpoint node identifier "Add to Cart".

[0072] Step C3: Same-domain and cross-domain execution. In same-domain applications, the endpoint node is located directly based on the current application's offline map in the input. In cross-domain applications, supplementary knowledge of the target application's offline map (including its complete structured representation) is injected into the input content. This allows the model to perform "user intent-functional node" semantic matching and output the target endpoint node within a unified inference framework, thereby verifying the model's adaptability to offline maps of unseen applications. Taking a music player not included in the training data as an example, if the user inputs "play the song 'Rice Fragrance' on the music player," the model can identify the endpoint path as "find and click play." Combined with post-processing operations, it can output the complete path: "Step 1, start the music player; Step 2, search for music <parameter 1>; Step 3, find and click play." Post-processing of results Step D1: Node localization. Based on the unique endpoint node identifier output by the model, perform a precise search in the unique offline map to locate the target node.

[0073] Step D2: Path backtracking and reconstruction. Starting from the target node, backtrack step by step along the parent node pointers to the root node to obtain the reverse path sequence. Then, reverse the sequence to generate the complete execution path from the root node to the target node.

[0074] Step D3: Completeness and Compliance Verification. Verify the reachability and hierarchical consistency of each step in the reconstruction path on the offline map; if there are broken links, skipped levels, or invalid nodes, revert to the candidate node for reconstruction or trigger manual review.

[0075] Step D4: Output Results. Output a structured execution path for the GUI Agent to execute directly. Parameter placeholders are reserved in the path and populated by a separate parameter extraction model. The output path is: "Open e-commerce platform webpage" → "Search for product <parameter1>" → "Select sort by sales volume in the filter" → "Click to enter the page of the top-selling product" → "Add to cart".

[0076] Inference tests using 327 data points across 165 different offline maps used in the training process showed that the endpoint prediction accuracy was 5.8% higher than that of traditional full path generation methods.

[0077] Inference tests were conducted on 200 data points from five offline maps that were not included in the training data. The results show that the cross-domain test accuracy is still higher than that of traditional full path generation methods.

[0078] The results of the above process show that: On the training set, compared with traditional full path generation methods, the method of this invention has achieved a significant improvement in the accuracy of destination prediction.

[0079] (2) On the same test set, the method of the present invention can maintain stable and high prediction accuracy, indicating that the model has good robustness and consistency in similar application scenarios.

[0080] (3) On the cross-domain test set, the method of this invention can still achieve effective task planning, verifying the applicability of the method in different application scenarios and its good generalization ability. The fundamental reason for the generalization ability is that the model learns the "semantic and functional point matching logic", that is, it understands the semantic relationship between the user's natural language intent and the functional nodes in the offline map, rather than memorizing the specific menu locations or path structures. This semantic matching ability enables the model to transfer the learned matching logic to new application software. Even if the structure and node names of the offline map are different, as long as the semantic functions are similar, the model can still accurately identify and locate the target node.

[0081] The above experimental results verify the effectiveness of the endpoint prediction data format method and the multi-agent collaborative data construction process: (1) Validation of the endpoint prediction data format method: The results show that, compared with the baseline method, the present invention improves the accuracy by 4 percentage points by simplifying the output target, which significantly improves the model performance.

[0082] (2) Validation of the effectiveness of multi-agent collaborative data construction: Method A: Manual annotation only. Data quality: Medium; Production efficiency: Low.

[0083] Method B (this invention): Multi-agent collaborative construction. Data quality: High; Production efficiency: High; Validation failure rate reduced from an initial 40% to below 10% through a feedback control closed-loop mechanism.

[0084] The results show that the multi-agent collaborative mechanism of the present invention significantly improves data quality and production efficiency.

[0085] In addition, the present invention also provides a GUI Agent task planning system based on multi-agent cooperation, including: A multi-agent collaborative data construction module is used to automatically generate training data that has been verified, repaired, and optimized by means of multi-agent collaboration and feedback control closed loop; the multi-agent includes a training sample generation module, a data quality verification module, a data repair module, and a rule extraction module; The endpoint prediction and model training module is used to construct training samples with endpoint nodes as prediction targets based on unique offline map nodes, using the endpoint prediction data format and model training method. It also uses the NoThink training method to fine-tune the pre-trained large language model to obtain the fine-tuned model. The model evaluation and generalization testing module is used to construct training sets, same-domain test sets, and cross-domain test sets to evaluate the performance and verify the generalization ability of the fine-tuned model.

[0086] The offline map in this invention is a structured method for representing application operation paths. It organizes the application's functional points into a tree structure, with each node representing an executable operation step. This method includes: a multi-agent collaborative data construction process, which automatically generates high-quality training data through iterative iterations of a training sample generation module, a data quality verification module, a data repair module, and a rule extraction module; a endpoint prediction data format method that ensures each step in the offline map has unique names, and uses the final execution node as the prediction target in the training data, rather than the complete execution path, simplifying the model's learning objective; and a NoThink training method based on pre-trained large language models (such as Qwen, DeepSeek, etc.) to improve inference efficiency. This invention significantly improves the quality of training data through a multi-agent collaborative mechanism. Through innovative endpoint prediction data format, it improves model accuracy by more than 4% compared to traditional full-path generation methods and possesses good cross-application generalization ability. The fundamental reason for this generalization ability is that the model learns the "semantic matching logic between functional points," rather than rote memorization of specific menu locations.

[0087] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A GUI Agent task planning method based on multi-agent collaboration, characterized in that, Includes the following steps; S1, Constructing multi-agent collaborative data: Through multi-agent collaboration and feedback control closed loop, training data that has been verified, repaired and optimized by rules is automatically generated; the multi-agent includes a training sample generation module, a data quality verification module, a data repair module and a rule extraction module; S2 adopts the data format and model training method of endpoint prediction. Based on unique offline map nodes, it constructs training samples with endpoint nodes as prediction targets and uses the NoThink training method to fine-tune the pre-trained large language model to obtain the fine-tuned model. S3 constructs training sets, same-domain test sets, and cross-domain test sets to evaluate the performance and verify the generalization ability of the fine-tuned model. Step S2 includes the following steps: S21, Offline map preprocessing: Perform node uniqueness processing on the offline map to ensure the uniqueness of all node names in the offline map; S22, Training Sample Construction: Construct training samples that include the system instruction domain, user input domain, and model output domain; S23, Model Training: The NoThink training method is adopted, which enables the pre-trained large language model to directly map from user input and offline map structure to the endpoint node, establishing a direct correspondence between "intent and result"; S24, Endpoint Node Post-processing and Path Reconstruction: Based on the endpoint nodes predicted by the fine-tuned model, a complete executable path is generated by backtracking the tree structure of the offline map.

2. The GUI Agent task planning method based on multi-agent collaboration according to claim 1, characterized in that, Step S1 includes the following steps: S11, Initialize the data source, which includes manually labeled seed data and offline map structured information from the application; S12, through the training sample generation module, generates training samples of different dimensions and complexities based on the offline map; S13, through the data quality verification module, performs format verification on the generated training samples and outputs data that does not meet the requirements and corresponding abnormal feedback information; S14, the data repair module regenerates the data that does not meet the requirements based on the abnormal feedback information, and the generated data is then input into the data quality verification module for verification. S15, Calculate the verification failure rate. When the verification failure rate is higher than the preset threshold, proceed to step S16; when the verification failure rate is lower than the preset threshold, proceed to step S17. S16, through the rule extraction module, the abnormal feedback information is extracted into rules, and the extracted rules are injected back into the training sample generation module to update the generation strategy, forming a feedback control closed loop; S17. Manually label the remaining data that does not meet the requirements to complete the construction of the training dataset.

3. The GUI Agent task planning method based on multi-agent collaboration according to claim 2, characterized in that, Step S16 further includes the following steps: Perform in-depth analysis of multiple rounds of abnormal feedback information to summarize the constraint rules and error patterns of data generation; dynamically update the extracted rules into the instructions of the training sample generation module; The process involves iteratively executing steps such as training sample generation, data quality verification, and data repair until the verification failure rate is reduced to below a preset threshold.

4. The GUI Agent task planning method based on multi-agent collaboration according to claim 1, characterized in that, In step S21, the node uniqueness processing refers to handling the case of duplicate nodes on different pages in the offline map through path encoding or renaming, in order to ensure that each node has a unique identifier.

5. The GUI Agent task planning method based on multi-agent collaboration according to claim 4, characterized in that, In step S22, the instruction domain is used to define the model role as a GUI Agent expert, and to specify the task as understanding the user's intent and locating the most matching endpoint node in the offline map; The user input field includes user natural language commands and structured offline map descriptions. The user natural language commands describe the user's operational intentions. The offline map uses a nested dictionary structure to represent a tree structure, where the dictionary keys are operation names and the dictionary values ​​are child node dictionaries, forming a complete path tree from the root node to the leaf node. The model output domain contains a unique endpoint node identifier, which is a unique node identifier after node uniqueness processing.

6. The GUI Agent task planning method based on multi-agent collaboration according to claim 5, characterized in that, Step S24 includes the following steps: S241, Node localization: Based on the unique endpoint node identifier output by the fine-tuned model, accurately locate the target node in the offline map after unique processing; S242, Path backtracking: Starting from the located target node, traverse upwards along the parent node pointers of each node to the root node to obtain the reverse path sequence; S243, Sequence Reversal: Reverse the reversed path sequence to generate a complete execution path from the root node to the target node; S244, Integrity Verification: Through the structural constraints of the offline map, ensure that the complete execution path generated in step S243 is consistent with the node hierarchy of the corresponding offline map.

7. The GUI Agent task planning method based on multi-agent collaboration according to claim 6, characterized in that, Step S3 includes the following steps: S31, Training set construction: Directly use the training dataset generated in step S1; the training dataset covers different application software and scenarios of different complexities; S32, Test Set Construction: Constructing same-domain and cross-domain test sets; The same-domain test set includes application software that appeared in the training set but used different user inputs or offline map versions, and is used to evaluate the performance of the fine-tuned model in similar scenarios. The cross-domain test set includes offline maps of application software that did not appear in the training set, used to verify the generalization ability of the fine-tuned model. S33, Model Performance Evaluation: The performance of the fine-tuned model is evaluated using endpoint accuracy, path completeness, and generalization accuracy as evaluation metrics.

8. The GUI Agent task planning method based on multi-agent collaboration according to claim 7, characterized in that, In step S33, the accuracy of the endpoint node is the proportion of the final execution node output by the model that is completely consistent with the standard answer; The path completeness rate is the proportion of all steps in the reconstructed complete execution path that are consistent with the node hierarchy of the corresponding offline map. The generalization accuracy is the accuracy on the cross-domain test set.

9. A GUI Agent task planning system based on multi-agent collaboration, used to implement the GUI Agent task planning method based on multi-agent collaboration as described in any one of claims 1-8, characterized in that, The GUIAgent task planning system based on multi-agent collaboration includes: A multi-agent collaborative data construction module is used to automatically generate training data that has been verified, repaired, and optimized by means of multi-agent collaboration and feedback control closed loop; the multi-agent includes a training sample generation module, a data quality verification module, a data repair module, and a rule extraction module; The endpoint prediction and model training module is used to construct training samples with endpoint nodes as prediction targets based on unique offline map nodes, using the endpoint prediction data format and model training method. It also uses the NoThink training method to fine-tune the pre-trained large language model to obtain the fine-tuned model. The model evaluation and generalization testing module is used to construct training sets, same-domain test sets, and cross-domain test sets to evaluate the performance and verify the generalization ability of the fine-tuned model.

Images