A gui operation intelligent agent training and automation method based on visual page understanding

Abstract

The application provides a GUI operation intelligent agent training and automation method based on visual page understanding, S1, multi-modal environment perception and state representation; S2, task logic analysis and state machine modeling; S3, virtual task synthesis; S4, hierarchical model architecture design; S5, collaborative training of imitation learning and reinforcement learning. The application realizes a complete technical link from a small amount of human demonstration to large-scale automatic execution by constructing a multi-modal environment perception system, a task logic analysis engine, a virtual task synthesis module and a hierarchical model architecture.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction automation, specifically to a method for training and automating GUI operation agents based on visual page understanding, applicable to various application scenarios such as automated testing of mobile and PC software, robotic process automation, accessibility assistance technology, and intelligent teaching and operation guidance. Background Technology

[0002] With the deepening of information technology construction, graphical user interfaces (GUIs) have become the main form of human-computer interaction. During enterprise digital transformation, a large number of repetitive GUI operations need to be automated, such as the execution of regression test cases in software testing, and data entry and report generation in business processes. Traditional GUI automation technologies mainly rely on script recording and playback, element location, and operation command binding, but they have the following prominent problems:

[0003] First, it is highly dependent on the method of locating interface elements and has poor adaptability. Traditional methods are usually based on element attributes (such as ID, XPath, CSS selectors) for location. When the interface layout is adjusted, element attributes are changed, or cross-platform deployment is carried out, the script often fails and requires manual maintenance. Especially in old systems or closed third-party applications, element location is more difficult and the cost of automation is high due to the lack of accessible interface support.

[0004] Second, it struggles to handle complex interactive operations. Traditional scripting methods are ill-equipped to handle complex tasks involving multimodal input, such as password input requiring switching between Chinese and English, or between uppercase and lowercase letters, or interactive flows that need to be dynamically adjusted based on interface feedback. These operations not only involve precise coordinate clicks and keyboard input, but also require understanding the current interface state and making correct decisions, placing higher demands on the cognitive abilities of the intelligent agent.

[0005] Third, the deployment cost of large-scale pre-trained models is high. In recent years, vision-based multimodal large models have shown great potential in GUI understanding and operation tasks, but these models have a huge number of parameters and consume a lot of computing resources, making them difficult to deploy and run on terminal devices. At the same time, large models have high inference latency, which cannot meet the performance requirements of real-time interactive scenarios.

[0006] Fourth, lightweight models lack generalization ability. To reduce deployment costs, some solutions use lightweight visual models for GUI operation decisions. However, due to limited training data and incomplete coverage of task scenarios, these models lack generalization ability when faced with unfamiliar interface layouts or operation processes, making it difficult to guarantee accuracy.

[0007] Fifth, there is a lack of effective data generation and augmentation mechanisms. Training intelligent agents requires a large amount of labeled data covering diverse scenarios, but manual collection and labeling are costly. Existing methods lack effective mechanisms to automatically generate rich training data from a small number of human demonstrations, resulting in limited training data scale and incomplete scenario coverage.

[0008] Sixth, the system lacks strong task logic comprehension and transferability. Existing automation systems often employ "memory-based" learning, merely recording operation sequences without understanding the underlying logical intent of the task. When task conditions change (such as moving input boxes or changing button styles), the system cannot adaptively adjust according to the task logic, leading to interruptions in the automation process.

[0009] Therefore, there is an urgent need for an agent training and automation method that can deeply understand GUI page states and task logic, automatically generate diverse training data, and achieve a balance between high accuracy and low resource consumption. This invention effectively solves the above-mentioned technical challenges by constructing a "demonstration-parsing-synthesis" data generation paradigm and a layered model architecture of "cloud-edge collaboration, emphasizing understanding over execution," providing an efficient and reliable technical solution for GUI operation automation. Summary of the Invention

[0010] To address the aforementioned issues, this invention provides a method for training and automating GUI operation agents based on visual page understanding. By constructing a multimodal environment perception system, a task logic parsing engine, a virtual task synthesis module, and a hierarchical model architecture, it achieves a complete technical chain from a small number of human demonstrations to large-scale automated execution.

[0011] The specific plan is as follows:

[0012] A method for training and automating a GUI operation agent based on visual page understanding includes the following steps:

[0013] S1. Multimodal environment perception and state representation: Synchronously collect screen pixel data, accessibility tree information and window layout metadata to construct a unified structured page state representation;

[0014] S2. Task Logic Analysis and State Machine Modeling: Analyze the demonstration process using a multimodal model, extract the task logic, and construct a state machine model.

[0015] S3. Virtual Task Synthesis: Based on the task state machine, diverse virtual tasks are generated through controllable perturbation, element replacement, and scene combination.

[0016] S4. Layered Model Architecture Design: A layered architecture using a general understanding model and lightweight dedicated executors is adopted to achieve an optimal balance between computing resources and execution efficiency.

[0017] S5. Collaborative training of imitation learning and reinforcement learning: First, initialize the strategy through behavior cloning, and then optimize it through reinforcement learning;

[0018] S6. Simulation-Reality Transfer Mechanism: Enhance the robustness of the model in real-world environments through domain randomization techniques.

[0019] Furthermore, step S1 specifically includes:

[0020] Set time Screen pixel observation is ,in and These represent the screen height and width, respectively, with 3 indicating the RGB three-channel configuration; simultaneously, accessibility tree information is obtained. It includes the hierarchical structure of interface elements, type identifiers, text content, bounding box coordinates, and interactivity attributes;

[0021] By fusing pixel observations with structured information, a page state vector is constructed:

[0022]

[0023] in For a moment The number of interactive elements on the page For the first An embedding representation vector of elements. The element embedding dimension; each element embedding vector encodes the element's position coordinates, type features, textual semantics, and visual appearance information.

[0024] Operation sequence is defined as Each operation Includes the operation type (click, input, swipe, etc.), target element index, and operation parameters; the state-operation trajectory generated during a human demonstration is denoted as... .

[0025] Furthermore, step S2 specifically includes:

[0026] First, identify the key state nodes in the demonstration process, including: focus switching state (input box gains focus), pop-up appearance state, text content update state, and page jump state; let the set of key states be . ,in Number of critical states;

[0027] For the sequence of operations between adjacent critical states, analyze their operational intent and logical constraints; define the operational intent vector:

[0028]

[0029] in For the first The intention of the step operation is indicated. The intent embedding dimension is used; the intent encoder adopts the Transformer architecture, which integrates information from the current state, the executed operation, and the next state to infer the purposeful semantics of the operation.

[0030] Construct a task state machine model based on key states and operational intentions. ,in For the set of state transition functions, Let be the set of intent vectors; state transitions satisfy the following definition:

[0031]

[0032] in From state Transition to state Required sequence of operations The state transition function, i.e., the set Members.

[0033] Furthermore, step S3 specifically includes:

[0034] Element attribute perturbation: Randomly perturbs the attributes of page elements, including position offset, size scaling, color adjustment, and style changes; assuming the original element attributes are... The properties after perturbation are:

[0035]

[0036] in Let be the perturbation vector, which follows a mean of zero and a covariance matrix of... Gaussian distribution, The disturbance intensity coefficient is... This is an attribute correlation matrix;

[0037] Semantic equivalence replacement: While maintaining the semantic meaning of the task, replace the text content or visual style of interface elements; for example, replace the "Username" input box with the "Account" input box, and change the style of the login button from a rounded rectangle to a right rectangle. The replacement process ensures that:

[0038]

[0039] That is, the semantic functions of the elements before and after the replacement are equivalent;

[0040] Scene Combination Extension: Combine the state machines of multiple basic tasks to generate composite task scenarios; let the state machines of the two basic tasks be respectively... and The combined state machine is as follows:

[0041]

[0042] in This indicates a state machine connection operation, which achieves task chaining by sharing intermediate state nodes;

[0043] Through the above-mentioned synthesis mechanism, from Individual human demonstration samples can be generated A virtual task sample, satisfying This greatly expanded the scale of training data.

[0044] Furthermore, step S4 specifically includes:

[0045] General Understanding Model Layer: Deployed in the cloud or on edge servers, this layer processes raw visual input and extracts structured page state representations. The model employs a visual Transformer architecture, with screen pixel images as input. The output is a sequence of element-embedded vectors:

[0046]

[0047] The general understanding model has the following parameter count: It has the ability to understand pages across tasks and applications, and provides input for multiple dedicated executors as a shared service;

[0048] Lightweight dedicated actuator layer: Deployed on terminal devices, optimized for specific tasks; the actuator receives a structured state representation output from a general understanding model, rather than raw pixels, greatly reducing the input dimensionality; the actuator model is defined as:

[0049]

[0050] in For the policy function, For actuator parameters, Output layer weight matrix; number of actuator parameters It has extremely low inference latency, making it suitable for real-time execution scenarios.

[0051] Furthermore, step S5 specifically includes:

[0052] Imitation learning phase: Using human demonstration data and synthetic task data, the policy network is initialized through behavior cloning; the loss function is defined as:

[0053]

[0054] in The total number of training samples; behavioral cloning enables the agent to quickly learn basic operational skills, providing good initial policy values ​​for subsequent reinforcement learning;

[0055] Reinforcement learning phase: In the synthetic task environment, the proximal policy optimization algorithm is used to further optimize the policy; the reward function is defined as follows:

[0056]

[0057] in Rewards for completing the task. Incentives are given for operational efficiency (to encourage the reduction of redundant operations). Incentives are given for smooth operation (to encourage continuous and stable operation). and These are the weighting coefficients;

[0058] The objective function for reinforcement learning is:

[0059]

[0060] in For trajectory, For the trajectory length, The discount factor is used; the parameters are optimized using the policy gradient method. This maximizes the expected cumulative reward.

[0061] Furthermore, step S6 specifically includes: introducing domain randomization techniques into the synthesis task to simulate the diversity of real-world environments, including:

[0062] Resolution randomization: The resolution of the synthesized page is randomly adjusted to simulate the screen specifications of different devices.

[0063] Style randomization: Randomize the visual style (color, font, border, etc.) of interface elements to enhance the model's tolerance to visual differences;

[0064] Delay simulation: Introducing random delays after the operation is executed to simulate the response time fluctuations of a real system;

[0065] By training with domain randomization, the actuator model can effectively transfer to real-world environments and maintain stable operational performance.

[0066] Through the above technical solution, this invention constructs a complete technical chain from human demonstration to automated execution of intelligent agents, realizing the organic unity of efficient data generation, lightweight model deployment, and precise operation execution, and providing an innovative solution for GUI operation automation.

[0067] The beneficial effects of this invention are as follows:

[0068] 1. By employing a "demonstration-analysis-synthesis" data generation paradigm, massive amounts of training data (e.g., 500 samples) can be automatically generated from a small number of human demonstrations (e.g., 5 sets of samples), achieving a data augmentation ratio of over 100 times. Compared to traditional methods that require a large amount of manually labeled data, this invention significantly reduces data collection and labeling costs, making agent training more efficient and convenient, and drastically reducing data requirements.

[0069] 2. This invention adopts a layered model architecture, deploying a large-parameter general understanding model (86M parameters) on cloud or edge servers for shared use, while the terminal only needs to deploy a lightweight dedicated executor (2.5M parameters). The terminal model size is only about 2.5MB, with an inference latency as low as 4ms, which can run smoothly on ordinary office devices without the need for a dedicated hardware accelerator, significantly reducing model deployment costs.

[0070] 3. This invention generates training data covering diverse scenarios through a virtual task synthesis engine, enabling the agent to adapt to unfamiliar interface layouts, element styles, and operational changes. In cross-scenario testing, the agent still achieves a task success rate of over 87% on new systems that were not previously trained on, demonstrating excellent generalization performance.

[0071] 4. This invention is specifically optimized for complex operation tasks involving multimodal input (such as password input with Chinese / English switching and case switching). The intelligent agent can understand the input method state, correctly execute the switching operation, and accurately input complex passwords. The operation accuracy in such tasks reaches over 99%, demonstrating outstanding complex operation processing capabilities that are significantly superior to traditional script methods.

[0072] 5. State machine-based task logic parsing enables the agent to understand the intent behind operations, rather than mechanically memorizing operation sequences. When task conditions change (such as the input box position moving), the agent can adaptively adjust according to the task logic and continue to complete the operation process, exhibiting cognitive flexibility similar to humans, namely, the ability to understand and transfer task logic.

[0073] 6. End-to-end response latency is approximately 65ms in edge deployment mode and approximately 25ms in local caching mode, meeting the performance requirements of real-time interaction scenarios and demonstrating excellent real-time performance. The intelligent agent executes slightly faster than human operators while maintaining high accuracy, effectively improving the efficiency of automated operations.

[0074] 7. The intelligent agent possesses a certain degree of exception handling capability. When encountering abnormal situations such as pop-up interference or interface loading delays, it can correctly handle and continue the task in 85% of cases. Compared to traditional script methods that interrupt upon encountering an exception, this invention significantly improves the robustness of automated processes.

[0075] 8. The technical framework of this invention has good scalability: it can support more types of GUI operations (such as drag, scroll, gestures, etc.); it can be adapted to different operating system platforms (Windows, macOS, Linux, Android, iOS); it can integrate more perceptual modalities (such as voice commands, eye tracking, etc.); and it provides a solid foundation for subsequent function enhancements and technology iterations.

[0076] 9. This invention can be applied to multiple fields such as software automated testing, robotic process automation, accessibility assistance technology, and intelligent teaching guidance, and has broad market application prospects and commercial value. Especially in scenarios where traditional technologies are difficult to solve, such as the transformation of outdated systems and the automation of third-party closed applications, this invention has unique technical advantages. Attached Figure Description

[0077] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0078] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0079] As shown in the figure, this invention provides a method for training and automating GUI operation agents based on visual page understanding. By constructing an environment that can perceive, understand and record changes in the state of graphical user interface pages and operation sequences in real time, and adopting a technical route of "human demonstration - process understanding - task synthesis - hierarchical model training", it realizes the training and automated execution of agents for complex GUI operation tasks.

[0080] This embodiment details the specific implementation process of a GUI operation agent training and automation method based on visual page understanding, taking the automation of login processes involving complex password input in enterprise information systems as an application scenario.

[0081] I. System Architecture and Data Acquisition

[0082] A GUI-based intelligent agent training and execution system is constructed, comprising an environment perception module, a task parsing module, a data synthesis module, a model training module, and an execution and deployment module.

[0083] Environment awareness configuration: (1) Screen capture: The GUI interface is captured in real time using the screenshot API with a resolution of 1920×1080 pixels and a capture frame rate of 10fps; (2) Accessibility information: The element hierarchy tree is obtained through the operating system accessibility interface (Windows UIAutomation or Android Accessibility Service); (3) Operation record: The mouse click coordinates, keyboard input sequence, and focus change events are recorded.

[0084] The system records the entire operation process of the human demonstrator and generates state-operation trajectory data.

[0085] II. Implementation of Multimodal Environment Perception

[0086] 2.1 Screen Pixel Acquisition and Preprocessing

[0087] Take a screenshot of the login screen and obtain pixel observations. Image preprocessing includes normalization and resizing:

[0088]

[0089] The preprocessed image is then input into the visual encoder.

[0090] 2.2 Accessibility Tree Resolution

[0091] Obtain the accessibility tree of the login screen via the UI Automation interface. Extract key element information:

[0092] Username input field: Type is Edit, bounding box coordinates Interactivity is set to True

[0093] Password input box: Type is Edit, bounding box coordinates Interactivity is set to True

[0094] Login button: Type is Button, bounding box coordinates Interactivity is set to True

[0095] Input method switching prompt: Type is Text, content displays the current input method status.

[0096] Encode the attributes of each element into an embedding vector:

[0097]

[0098] in Encoding the element type, Encode the bounding box coordinates. Encoding text semantics Encode the visual appearance.

[0099] 2.3 Page State Vector Construction

[0100] By fusing pixel features and structured information, a page state vector is constructed:

[0101]

[0102] In the login interface of this embodiment, (Including all interactive elements such as input boxes, buttons, and labels), each element has an embedded dimension. .

[0103] III. Task Logic Analysis and Implementation

[0104] 3.1 Human Demonstration Data Collection

[0105] Invite an operator to demonstrate the login process; the system records the complete operation sequence:

[0106] 1. Click the username input box → switch the focus to the username input box.

[0107] 2. Switch the input method to English mode → The input method status changes to English.

[0108] 3. Enter username "admin" → Text content update

[0109] 4. Click the password input box → switch the focus to the password input box.

[0110] 5. Switch the input method to Chinese mode → The input method status changes to Chinese.

[0111] 6. Enter the password in pinyin "mima123" → the pinyin candidate box appears.

[0112] 7. Select candidate words → Password box displays ciphertext

[0113] 8. Switch between uppercase and lowercase letters → Enter uppercase letters

[0114] 9. Enter the remaining password characters → Password input complete

[0115] 10. Click the login button → Submit login request

[0116] Generate state - operation trajectory .

[0117] 3.2 Key State Identification

[0118] Utilizing a multimodal understanding model to analyze the demonstration trajectory and identify key state nodes:

[0119] Initial state: The cursor is at the default position.

[0120] Username input is ready; focus is on the username input box.

[0121] Username input complete status

[0122] Password input ready state, focus is on the password input box.

[0123] Password input complete

[0124] Login submission status

[0125] Key state set .

[0126] 3.3 Analysis of Operation Intent

[0127] Intent encoding is performed for each step of the operation. Taking step 5, "Switch the input method to Chinese mode," as an example:

[0128]

[0129] Intent encoder analysis: The current state is password input ready, the operation is input method switching, and the next state will show Chinese input method activated. The inferred intent is "preparing to input Chinese characters".

[0130] Constructing a task state machine model This provides a complete description of the state transition logic of the login process.

[0131] IV. Implementation of Virtual Task Synthesis

[0132] 4.1 Element Attribute Perturbation

[0133] By perturbing the attributes of the login interface elements, a variety of variants can be generated:

[0134] (1) Username input box position offset: original coordinates After the disturbance

[0135] (2) Scaling the login button: Original size After the disturbance

[0136] (3) Interface color scheme adjustment: The background color was changed from white to light gray.

[0137] Disturbance intensity coefficient This ensures that the interface remains recognizable and operable even after disturbance.

[0138] 4.2 Semantic Equivalence Substitution

[0139] Perform semantically equivalent element replacement:

[0140] (1) The username input box label has been changed from "Username" to "Account".

[0141] (2) Replace the text on the login button with "Login" (add a space).

[0142] (3) The border style of the input box has been changed from solid line to dashed line.

[0143] The substitution process verifies semantic equivalence:

[0144]

[0145] 4.3 Scene Combination Expansion

[0146] Combine login tasks with other common tasks:

[0147] (1) Login + Data Query: After successful login, you will be automatically redirected to the query page.

[0148] (2) Login + Report Export: After successful login, perform the report export operation.

[0149] (3) Login + System Settings: After successfully logging in, enter the settings page to modify the configuration.

[0150] By combining and extending, multiple complex task scenarios can be generated from a single login demo.

[0151] 4.4 Statistical Analysis of Composite Data

[0152] Training data was generated using a virtual task synthesis engine based on five human demonstration samples.

[0153] (1) Original demonstration sample: 5 complete login process trajectories

[0154] (2) Element perturbation variants: 50 variants are generated for each group, for a total of 250 groups.

[0155] (3) Semantic substitution variants: 20 variants are generated for each group, for a total of 100 groups.

[0156] (4) Scene combination expansion: Generate 150 sets of composite task trajectories

[0157] Total Synthesis The training samples were expanded by 100 times compared to the original data.

[0158] V. Implementation of Hierarchical Model Training

[0159] 5.1 Deployment of the General Understanding Model

[0160] The general understanding model uses qwen 3 VL and is deployed on an edge server. The model input is... RGB image, output as a sequence of element embedding vectors.

[0161] Encode the login screen screenshot:

[0162]

[0163] Each element is embedded in a vector dimension. The inference latency for the general understanding model is approximately 45ms.

[0164] 5.2 Training of Dedicated Actuators

[0165] The dedicated actuator adopts a small ViT combined with an LSTM architecture, with a small number of parameters. It is deployed on terminal devices.

[0166] Imitation learning stage:

[0167] Using 500 sets of training data, the executor is trained via behavior cloning:

[0168]

[0169] Training parameters: learning rate Batch size Number of training rounds After training, the actuator achieved a motion prediction accuracy of 92.3% on synthetic data.

[0170] Reinforcement learning phase:

[0171] The PPO algorithm was used to further optimize the strategy in a simulation environment. Reward function settings:

[0172]

[0173] in This indicates that the task has been successfully completed. , .

[0174] PPO training parameters: discount factor Cutting parameters Learning rate Number of training iterations .

[0175] VI. Real-world environment execution testing

[0176] 6.1 Test Environment Configuration

[0177] Testing agent performance in a real enterprise information system:

[0178] (1) Test system: Enterprise Resource Management System (ERP)

[0179] (2) Test scenario: Login process involving complex password input

[0180] (3) Test equipment: ordinary office computer, Intel i5 processor, 8GB memory

[0181] (4) Number of tests: 100 independent executions

[0182] 6.2 Performance Evaluation

[0183] Test result statistics:

[0184] (1) Task success rate: 96% (96 out of 100 login attempts were successful)

[0185] (2) Average execution time: 8.2 seconds (the average human execution time is about 12 seconds)

[0186] (3) Operation accuracy: Input content accuracy was 99.2%, and click location accuracy was 97.8%.

[0187] (4) Anomaly handling capability: When encountering pop-up interference, it can handle it correctly and continue the task in 100% of cases.

[0188] 6.3 Cross-Scenario Generalization Testing

[0189] Testing generalization ability in systems that were not trained:

[0190] (1) New System A (OA Office System): Task success rate 89%

[0191] (2) New System B (CRM Customer Management System): Task success rate 91%

[0192] (3) New System C (Financial Management System): Task success rate 87%

[0193] The results show that the agent possesses good cross-scenario generalization ability and can adapt to unseen interface layouts and operation processes. Through the detailed description of this embodiment, those skilled in the art can clearly understand the technical solution of the present invention and can implement the present invention in practical application scenarios according to the described method and system configuration to achieve efficient and reliable GUI operation automation.

[0194] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A method for training and automating a GUI operation agent based on visual page understanding, characterized in that, Includes the following steps: S1. Multimodal environment perception and state representation: Synchronously collect screen pixel data, accessibility tree information and window layout metadata to construct a unified structured page state representation; S2. Task Logic Analysis and State Machine Modeling: Analyze the demonstration process using a multimodal model, extract the task logic, and construct a state machine model. S3. Virtual Task Synthesis: Based on the task state machine, diverse virtual tasks are generated through controllable perturbation, element replacement, and scene combination. S4. Layered Model Architecture Design: A layered architecture using a general understanding model and lightweight dedicated executors is adopted to achieve an optimal balance between computing resources and execution efficiency. S5. Collaborative training of imitation learning and reinforcement learning: First, initialize the strategy through behavior cloning, and then optimize it through reinforcement learning; S6. Simulation-Reality Transfer Mechanism: Enhance the robustness of the model in real-world environments through domain randomization techniques.

2. The method for training and automating a GUI operation agent based on visual page understanding according to claim 1, characterized in that, Step S1 specifically includes: Set time Screen pixel observation is ,in and These represent the screen height and width, respectively, with 3 indicating the RGB three-channel configuration; simultaneously, accessibility tree information is obtained. It includes the hierarchical structure of interface elements, type identifiers, text content, bounding box coordinates, and interactivity attributes; By fusing pixel observations with structured information, a page state vector is constructed: in For a moment The number of interactive elements on the page For the first An embedding representation vector of elements. The element embedding dimension; each element embedding vector encodes the element's position coordinates, type features, textual semantics, and visual appearance information. Operation sequence is defined as Each of the operations Includes operation type, target element index, and operation parameters; the state-operation trajectory generated during the human demonstration is denoted as... .

3. The method for training and automating a GUI operation agent based on visual page understanding according to claim 2, characterized in that, Step S2 specifically includes: First, identify the key state nodes in the demonstration process, including: focus switching state, pop-up appearance state, text content update state, and page jump state; let the set of key states be . ,in Number of critical states; For the sequence of operations between adjacent critical states, analyze their operational intent and logical constraints; define the operational intent vector: in For the first The intention of the step operation is indicated. The intent embedding dimension is used; the intent encoder adopts the Transformer architecture, which integrates information from the current state, the executed operation, and the next state to infer the purposeful semantics of the operation. Construct a task state machine model based on key states and operational intentions. ,in For the set of state transition functions, Let be the set of intent vectors; state transitions satisfy the following definition: in From state Transition to state Required sequence of operations The state transition function, i.e., the set Members.

4. The method for training and automating a GUI operation agent based on visual page understanding according to claim 3, characterized in that, Step S3 specifically includes: Element attribute perturbation: Randomly perturbs the attributes of page elements, including position offset, size scaling, color adjustment, and style changes; assuming the original element attributes are... The properties after perturbation are: in Let be the perturbation vector, which follows a mean of zero and a covariance matrix of... Gaussian distribution, The disturbance intensity coefficient is... This is an attribute correlation matrix; Semantic equivalence replacement: Replace the text content or visual style of UI elements while maintaining the semantics of the task; the replacement process ensures that: That is, the semantic functions of the elements before and after the replacement are equivalent; Scene Combination Extension: Combine the state machines of multiple basic tasks to generate composite task scenarios; let the state machines of the two basic tasks be respectively... and The combined state machine is as follows: in This indicates a state machine connection operation, which achieves task chaining by sharing intermediate state nodes; Through the above-mentioned synthesis mechanism, from Individual human demonstration samples can be generated A virtual task sample, satisfying This greatly expanded the scale of training data.

5. The method for training and automating a GUI operation agent based on visual page understanding according to claim 4, characterized in that, Step S4 specifically includes: General Understanding Model Layer: Deployed in the cloud or on edge servers, this layer processes raw visual input and extracts structured page state representations. The model employs a visual Transformer architecture, with screen pixel images as input. The output is a sequence of element-embedded vectors: The general understanding model has the following parameter count: It has the ability to understand pages across tasks and applications, and provides input for multiple dedicated executors as a shared service; Lightweight dedicated actuator layer: Deployed on terminal devices, optimized for specific tasks; the actuator receives a structured state representation output from a general understanding model, rather than raw pixels, greatly reducing the input dimensionality; the actuator model is defined as: in For the policy function, For actuator parameters, Output layer weight matrix; number of actuator parameters It has extremely low inference latency, making it suitable for real-time execution scenarios.

6. The method for training and automating a GUI operation agent based on visual page understanding according to claim 5, characterized in that, Step S5 specifically includes: Imitation learning phase: Using human demonstration data and synthetic task data, the policy network is initialized through behavior cloning; the loss function is defined as: in The total number of training samples; Reinforcement learning phase: In the synthetic task environment, the proximal policy optimization algorithm is used to further optimize the policy; the reward function is defined as follows: in Rewards for completing the task. Incentives for operational efficiency As a reward for smooth operation, and These are the weighting coefficients; The objective function for reinforcement learning is: in For trajectory, For the trajectory length, The discount factor is used; the parameters are optimized using the policy gradient method. This maximizes the expected cumulative reward.

7. The method for training and automating a GUI operation agent based on visual page understanding according to claim 6, characterized in that: Step S6 specifically includes: introducing domain randomization techniques into the synthesis task to simulate the diversity of real-world environments, including: Resolution randomization: The resolution of the synthesized page is randomly adjusted to simulate the screen specifications of different devices; Style randomization: Randomly vary the visual style of interface elements to enhance the model's tolerance to visual differences; Delay simulation: Introducing random delays after the operation is executed to simulate the response time fluctuations of a real system; By training with domain randomization, the actuator model can effectively transfer to real-world environments and maintain stable operational performance.

Images