An automatic GUI interaction execution system based on multi-modal perception and end-to-end VLA

Abstract

The application discloses an automatic GUI interaction execution system based on multi-modal perception and an end-to-end VLA, which comprises seven core modules, namely, a visual perception module, a language understanding module, a VLA decision module, an action execution module, a timing modeling module, a cross-application coordination module and a correction recovery module. By constructing a unified visual-language-action end-to-end model, the application integrates multi-modal perception, intelligent decision and physical operation capabilities, and realizes intelligent, flexible and robust automatic execution on various GUI interfaces.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, natural language processing and automated testing technology, and specifically to an automated GUI interactive execution system based on multimodal perception and end-to-end VLA (Vision-Language-Action). Background Technology

[0002] With the explosive growth of mobile internet and applications, GUI automation testing and execution technologies play a crucial role in software quality assurance and business process automation. Traditional GUI automation technologies primarily rely on methods such as coordinate-based script execution, UI tree-based element location, and image recognition-based control matching. However, existing technologies face numerous technical challenges in practical applications that urgently need to be addressed.

[0003] Question 1: Device Security Checks Limit Current automated testing tools typically interact with devices through system interfaces such as adb commands, XCUITest, and AccessibilityService. These interfaces are strictly limited by the operating system's security mechanisms. While rooted devices can obtain system-level privileges, on unrooted Android and iOS devices, the system detects the automated testing tool's process signature and application signature, triggering security checks that cause tests to fail. For example, Android's security enhancements identify emulator hardware characteristics, and iOS devices' anti-debugging and anti-tampering mechanisms prevent automated operations from non-system-signed applications. This limits automated testing to specific environments, making deployment on real user devices impossible.

[0004] Question 2: High Cost of Multi-Platform Adaptation. The GUI architectures of different operating systems (iOS, Android, HarmonyOS, Windows, Web) differ significantly. iOS uses the UIKit / SwiftUI framework, Android uses View / Compose, and Web uses HTML / CSS / JavaScript. The element positioning methods and event triggering mechanisms are completely different across platforms. Traditional methods require writing separate test scripts and maintaining test frameworks for each platform. When applications iterate across platforms, the test scripts need to be modified extensively, resulting in high maintenance costs. Statistics show that the average maintenance workload for an automated test script for a cross-platform application is three times that of a traditional application.

[0005] Question 3: Inaccurate Action Prediction. Traditional methods rely on predefined action rules and static coordinate positioning, which cannot intelligently predict and adapt to dynamically changing interfaces. When control positions shift with changes in screen size, resolution, and system version, scripts based on fixed coordinates will fail. Even when using image recognition to match controls, factors such as changes in lighting, icon tweaks, and interface updates reduce the accuracy of feature extraction and template matching, leading to click position shifts or accidental touches on other elements, resulting in a test failure rate as high as 15%-20%.

[0006] Question 4: Inability to Simulate Complex Operations. Existing tools primarily simulate user actions by sending simple touch events such as clicks, swipes, and long presses. However, their ability to simulate complex operations such as dragging, multi-touch, continuous scrolling, and curved swiping is limited. For example, route planning in map applications, handwritten signatures in drawing applications, and gradual volume adjustment in music applications require precise control of the movement trajectory, speed changes, and pressure changes of the touch point. Traditional touch event injection methods struggle to accurately simulate the finger movements of real users.

[0007] Question 5: Inability to Capture Dynamic Interface Changes. Traditional methods rely solely on single-frame static screenshots for perception, failing to identify and handle dynamic interface changes such as loading animations, automatic ad slideshows, pop-up fade-in / fade-out windows, and real-time progress bar growth. When the system needs to wait for the interface to stabilize before performing an operation, it often has to rely on hard-coded fixed delay times, leading to excessively long testing times or erroneous operations due to waiting too long. The lack of time-series modeling capabilities for dynamic interfaces makes it impossible to predict interface state transitions and proactively plan operation timings.

[0008] Question 6: Difficulty in Executing Cross-Application Workflows. Real-world business processes often require collaboration across multiple applications or pages, such as navigating from the main application to an SMS app to obtain a verification code, from an e-commerce platform to a payment gateway, or from a social media app to a photo album to select photos. Existing tools struggle to track the global application stack state, cannot remember the interface state before the navigation, and cannot automatically identify navigation patterns between different applications. This results in automating cross-application workflows requiring significant manual intervention or complex conditional scripts, making implementation extremely difficult.

[0009] Question 7: Inconvenient to extend to web and mobile automation tools. Mobile automation tools communicate with devices via adb or XCUITest, while web automation typically uses the WebDriver protocol to access the page's DOM structure through the browser. The underlying mechanisms of the two are completely different, making it difficult to use a unified testing framework and technology stack. Achieving end-to-end automated testing across mobile and web platforms requires integrating two different technology systems, increasing system complexity and maintenance costs.

[0010] Question 8: Lack of a robust error correction mechanism. When anomalies occur during automated execution (such as network timeouts, permission pop-ups, forced updates, or ad obstruction), existing tools typically report errors and terminate the test directly, lacking intelligent error correction and anomaly recovery capabilities. The system cannot automatically identify error types, nor can it attempt recovery operations such as retrying, clicking cancel, or skipping ads. The robustness and reliability of the test are poor, requiring significant manual intervention to handle anomalies.

[0011] To address the aforementioned technical challenges, there is an urgent need for an automated GUI execution system that can overcome software-layer limitations, support multi-platform unification, intelligently predict actions, simulate complex operations, understand dynamic interfaces, achieve cross-application collaboration, and possess intelligent error correction capabilities. This invention proposes an automated GUI interactive execution system that integrates multimodal perception, end-to-end VLA learning, and external device physical operation, providing an innovative technical solution to the aforementioned problems. Summary of the Invention

[0012] To address the aforementioned issues, this invention provides an automated GUI interaction execution system based on multimodal perception and end-to-end VLA. By constructing a unified vision-language-action end-to-end model, it integrates multimodal perception, intelligent decision-making, and physical manipulation capabilities to achieve intelligent, flexible, and robust automated execution of various GUI interfaces.

[0013] The specific plan is as follows:

[0014] An automated GUI interaction execution system based on multimodal perception and end-to-end VLA comprises seven core modules: a visual perception module, a language understanding module, a VLA decision-making module, an action execution module, a temporal modeling module, a cross-application coordination module, and an error correction and recovery module. The visual perception module captures dynamic changes and spatial layout of the interface through multi-frame video streams. The language understanding module parses the natural language descriptions of test cases, extracting task objectives and operational constraints. The VLA decision-making module uses a unified Multimodal Large Language Model (MLLM), taking interface images and task instructions as input, and directly outputs structured motion trajectories through toolcalling capabilities. These trajectories implicitly encode information such as action type, operation coordinates, and touch pressure, achieving end-to-end learning from perception to decision-making. The action execution module controls external physical devices such as dexterous hands, directly contacting the touchscreen of real terminal devices. The temporal modeling module models the dynamic changes and temporal context of the UI interface, predicting interface state transitions. The cross-application coordination module monitors the global application stack state, enabling automatic execution of cross-application workflows. The error correction and recovery module identifies the execution results by comparing the interface states before and after execution, and uses a built-in strategy library for anomaly recovery. The modules are closely integrated to form a complete closed loop of perception-decision-execution-verification.

[0015] Furthermore, the VLA decision module adopts a unified multimodal large language model architecture, which can directly process image and text inputs and output structured action trajectories through built-in tool calling capabilities. Unlike traditional multi-stage prediction methods, the system does not need to train components such as visual encoders, language encoders, and attention fusion modules separately. Instead, it uses a pre-trained multimodal base model (such as the Qwen-VL series) and fine-tunes it to enable it to understand and plan GUI operations.

[0016] Suppose the system is at the th The input received at any given time includes: the current interface image. ,in and These are the height and width of the image, respectively; the natural language description of the test cases. Historical interface feature sequence Multimodal large language models encode these inputs uniformly into an internal representation:

[0017]

[0018] in Represents the encoding function of a multimodal large language model. It serves as a unified representation within the model; the model automatically fuses multimodal information through its inherent vision-language fusion mechanism (usually composed of a ViT visual encoder and a Transformer decoder), without the need for an additional cross-modal attention module.

[0019] After fine-tuning, the model can correctly understand the position, function, and interaction methods of interface elements based on the input interface images and task instructions, and convert them into executable motion trajectories. Compared with the traditional multi-module architecture, the unified multimodal model has stronger generalization ability and can adapt to different platforms and different styles of interface design.

[0020] Furthermore, the VLA decision module outputs structured motion trajectories through a toolcalling mechanism; the system defines a trajectory generation tool, execute_trajectory, for the model, which accepts structured trajectory parameters as input:

[0021] toolexecute_trajectory{

[0022] "trajectory":[

[0023] {"x":1248,"y":378,"pressure":1.2,"contact":0},

[0024] {"x":1245,"y":375,"pressure":0.5,"contact":1},

[0025] ...,

[0026] {"x":1248,"y":378,"pressure":1.2,"contact":1},

[0027] {"x":1248,"y":378,"pressure":0.8,"contact":1},

[0028] {"x":1249,"y":379,"pressure":0.3,"contact":0}

[0029] ],

[0030] "screen_size":{"width":2400,"height":1080}

[0031] }

[0032] Wherein, trajectory is an array of trajectory points, each trajectory point contains four parameters: x: horizontal coordinate (pixels), range [0, W]; y: vertical coordinate (pixels), range [0, H]; pressure: touch pressure (Newtons), range [0, 5]; contact: contact state, 0 indicates floating, 1 indicates touching the screen;

[0033] screen_size is the screen size, used to calibrate the trajectory coordinates;

[0034] The trajectory output by the model is directly parsed and executed by the action execution module, without the need for additional coordinate transformations or parameter processing. The trajectory itself implicitly encodes the operation type information.

[0035] Click operation: The trajectory includes three stages: hovering approach, contact click, and lifting away. The contact point coordinates are concentrated and the number of contact points is small (about 3-5).

[0036] Sliding operation: The trajectory includes a continuous movement path from the starting point to the ending point, with a large number of contact points (about 10-20), and the coordinates change linearly;

[0037] Drag operation: The trajectory is similar to sliding, but a large touch pressure is maintained throughout;

[0038] Text input operation: The trajectory consists of multiple independent click trajectories, and each click trajectory corresponds to a virtual keyboard key;

[0039] This tool-based trajectory output method has the following advantages: 1. Unified output format: All operations are represented by trajectories, simplifying the interface between modules; 2. Flexible operation expression: Trajectories can represent arbitrarily complex operations, including curve scrolling, multi-point gestures, etc.; 3. Easy debugging and verification: Trajectories are output in a structured format, which is convenient for manual inspection and verification; 4. Scalability: The system capabilities can be extended by defining new tools, such as cross-application operations.

[0040] Furthermore, in practical applications, video streams are input in real-time as continuous frames (e.g., 60fps, frame interval approximately 16.67ms), while the VLA decision module's model inference suffers from computational latency (typically 300-800ms). To address the time inconsistency between inference latency and real-time execution, the system is designed with trajectory buffering and real-time control mechanisms to ensure that the dexterous hand can smoothly and accurately follow the trajectory.

[0041] Problem Analysis:

[0042] Assuming the video stream frame rate is (frames / second), then the frame interval is The system continuously captures video frames. When the first When the model is inference with a frame input, the model outputs the trajectory. However, at this point the video had already progressed to the [number]th [stage / section]. Frame, in which The number of frames corresponding to the model inference latency:

[0043]

[0044] in For model inference time; if fps and ms, then frame;

[0045] Using the trajectory output by the model directly may result in:

[0046] 1. Time base offset: The timestamp of the trajectory does not match the current actual time;

[0047] 2. Discontinuous execution: If the trajectory is recalculated each time, the movement of the dexterous hand will be intermittent;

[0048] 3. Accumulated position error: Accumulated delay causes the execution position to deviate from the target.

[0049] Solution: Track buffering and real-time interpolation

[0050] The system employs a trajectory buffer queue combined with real-time interpolation.

[0051] Trajectory timestamp mapping: Each trajectory point output by the model is accompanied by an absolute timestamp, rather than a relative time proportion; assuming the model is at the [missing information]th ... If the trajectory is output at time 1, then the 1st... The expected execution time for each trajectory point is:

[0052]

[0053] in The trajectory execution delay is approximately 50-100ms, used to prepare the dexterous hand. For relative time (from 0 to 1). The total duration of the trajectory (e.g., 400ms);

[0054] Track Buffer Queue: The system maintains a track point buffer queue. Each buffer item contains a timestamp of a trajectory point. ,coordinate ,pressure Contact status :

[0055]

[0056] When the model outputs a new trajectory, the trajectory points are inserted into the buffer queue in timestamp order;

[0057] Real-time trajectory query: Dexterous hand controller with fixed control frequency (e.g., 100Hz, 10ms interval) Query the current time The status of the trajectory points to be executed; the query method is:

[0058] like Equal to the timestamp of a certain trajectory point Then use that point directly.

[0059] like lie in and Between these points, linear interpolation is performed:

[0060]

[0061]

[0062]

[0063]

[0064] Trajectory smoothing and path planning: To avoid abrupt changes in dexterity hand movements, the interpolated coordinates are smoothed using a filtering process.

[0065]

[0066] in For smoothing coefficients, For the first Smoothing position of each control cycle;

[0067] Trajectory overlap handling: When a new trajectory arrives, if there are still unexecuted points in the buffer queue, the system determines whether to discard the old trajectory (indicating that the new user command overwrites it) or perform a smooth transition (such as in key input scenarios); for continuous operations on input text, the endpoint of the old trajectory is retained. As the starting point of a new trajectory :

[0068]

[0069] Timeout handling and safety mechanism: If the buffer queue has no new trajectory points within a preset time (e.g., 100ms), the dexterous hand will automatically lift the finger and return to a safe position to avoid accidental operation caused by prolonged contact with the screen;

[0070] Execution flow example:

[0071] Taking the "click the register button" action as an example:

[0072] 1. : The 0th frame of the video is captured and input into the model of the VLA decision module;

[0073] 2. Model inference complete, output trajectory (expected execution time: to );

[0074] 3. The trajectory point is injected into the buffer queue, and the dexterous hand begins to move from its current position to approach the registration button.

[0075] 4. : Apply pressure by actually touching the screen;

[0076] 5. Lift your finger off the screen;

[0077] 6. Return to a safe location;

[0078] 7. The trajectory has been executed. The system is waiting for the interface to respond and prepare for the next task.

[0079] Technical Effects: Through trajectory buffering and real-time interpolation mechanisms, the system can: decouple the time constraints of model inference and trajectory execution; ensure the smooth continuity of dexterous hand movements; support the smooth execution of operation sequences (such as quickly inputting multiple characters); and provide real-time responsiveness to adapt to dynamically changing interfaces. This mechanism is the key technical aspect of this invention enabling end-to-end real-time automatic GUI execution, solving the problems of execution discontinuity and accuracy loss caused by inference latency in traditional methods.

[0080] Furthermore, the temporal modeling module aims to capture the dynamic changes of the GUI interface in the time dimension; the system takes multiple consecutive frames of images as input, enabling the VLA decision module's model to understand and predict the temporal evolution of the interface state;

[0081] Let the input video frame sequence be ,in This refers to the timing window length; in actual deployment, the system chooses one of the following two methods to provide timing information:

[0082] Multi-frame image input: The multi-frame image sequence is directly input into the VLA decision module model. The model captures the inter-frame dependencies through its built-in attention mechanism and identifies the timing of the completion of the loading animation, the update rhythm of the ad carousel, and the dynamic changes in the disappearance process of the pop-up.

[0083] Text description assistance: Describe the interface state change information in the task instructions, such as "The interface loading animation has ended", "The ad carousel has been completed", "The pop-up has disappeared", etc., for the model's reference.

[0084] The VLA decision module's model, through its pre-trained visual understanding capabilities, can identify dynamic elements in the interface. Combined with the timing constraints in natural language commands, the model can intelligently determine the optimal timing for operations, avoiding execution when the interface is unstable. For example, when the command is "Wait for loading to complete before clicking the register button," the model will first observe the interface state, identify whether the loading animation has disappeared, and then plan the click trajectory.

[0085] In this way, the system achieves intelligent perception and understanding of dynamic interfaces without the need for an additional independent timing modeling module. The timing capability is entirely achieved through the inherent mechanism of the multimodal large language model.

[0086] Furthermore, the cross-application coordination module works closely with the language understanding module, visual perception module, and action execution module to monitor the global application stack status, enabling automatic execution and navigation backtracking of cross-application workflows; the system maintains an application stack. ,in Indicates the first in the application stack One application, The top of the stack (the currently active application). This represents the application stack depth. When an application switching event is detected (such as launching a messaging app or opening a browser), the system records the current application stack state and a snapshot of the interface to provide a basis for subsequent navigation.

[0087] To support intelligent decision-making across application scenarios, the system constructs an application switching graph. , where the node set Application, edge set This represents the navigation relationship between applications; each edge Associate a transition probability , indicating in application Jump to application The probability; it is obtained through statistical analysis of historical execution data:

[0088]

[0089] in For application Jump to application The historical frequency, where the denominator represents the number of times the application has been used. Total number of historical jumps across all applications; this probability is used to predict the most likely jump path across application scenarios.

[0090] When navigation is required across application scenarios, the system plans the optimal jump path based on the current application stack and task objectives; for scenarios that require jumping to other applications to obtain information (such as verification codes) and then returning, the system learns the "jump-operation-return" pattern.

[0091] Specifically, define the jump-back strategy. , indicating in application and mission objectives The following sequence of operations should be executed:

[0092]

[0093] in The length of the operation sequence. For the first Step-by-step operation For the task objective (e.g., "obtain a verification code"); the strategy is learned from demonstration data of human experts through imitation learning, consistent with the aforementioned training method;

[0094] When a verification code is detected, the system executes the following cross-application coordination process: First, the language understanding module, combined with OCR, recognizes the verification code prompt (e.g., "Please enter the SMS verification code"). Then, the action execution module launches the SMS app, the visual perception module reads the SMS interface, extracts the verification code using regular expressions, and the action execution module returns to the original application, enters the verification code into the corresponding input box, and completes the cross-application verification. The cross-application coordination module coordinates the work of each module throughout the process, ensuring that data is correctly transmitted between different applications.

[0095] Furthermore, the error correction and recovery module forms a closed loop with the visual perception module and the action execution module. By comparing the interface state before and after execution, the execution result is identified, and an error recovery is performed using a built-in strategy library.

[0096] The motion execution module follows the predicted motion trajectory. After the operation is executed, the visual perception module captures the resulting interface image; the system determines whether the operation was successful using the following two methods:

[0097] 1. Interface similarity detection: Calculate the pixel or feature differences between the interface images before and after execution. When the difference exceeds the threshold, it is considered that the operation has caused a change in the interface and is judged as successful.

[0098] 2. Model result verification based on VLA decision module: Input the interface image after execution and the expected result description (such as "should enter the registration page") into the VLA decision module model, and the model judges whether the current state meets the expectations;

[0099] Meanwhile, the system detects the characteristics of common abnormal scenarios; for network timeouts, it checks whether the loading indicator has been spinning for a long time; for permission pop-ups, it checks whether a "Allow access" or "Deny" dialog box has appeared; for forced updates, it checks whether "Update now" and "Update later" buttons have appeared.

[0100] The system maintains an error correction strategy library. ,in For the number of strategies, A single error correction strategy includes the criteria for identifying the type of exception and the corresponding recovery action:

[0101]

[0102] in For abnormal identification conditions (such as detecting a permission pop-up). The recovery action is either a new trajectory or a recovery command sent to the model in the VLA decision module to generate a new trajectory.

[0103] When an interface anomaly is detected, the system executes the corresponding recovery strategy, such as retrying clicks, clicking the cancel button, or clicking the skip button. After the recovery operation is executed, the interface status is checked again to determine if it has returned to normal. If it has returned to normal, subsequent tasks continue; if multiple retries still fail, an error log is recorded and manual intervention is notified.

[0104] Furthermore, the VLA decision module's model is based on a pre-trained multimodal large language model (such as the Qwen-VL series), which is then fine-tuned under supervision to master GUI operation capabilities. The training data is collected from the operation trajectory records of real users, and each training sample consists of an interface image, task instructions, and target trajectory.

[0105] Training data format:

[0106] Each training sample contains three parts: 1. Interface image: a screenshot of the current moment; 2. Task instruction: text describing the current operation, such as "Click the register button"; 3. Target trajectory: the actual operation trajectory, presented in tool call format. During training, the model receives the interface image and task instruction as input, and the target output is the correct tool call.

[0107] enter:

[0108] Image: [Screenshot of the current interface]

[0109] Instruction: "Click the register button"

[0110] Target output:

[0111] execute_trajectory({trajectory:[...],screen_size:{width:2400,height:1080}})

[0112] The training process employs supervised fine-tuning, using standard language model loss functions (such as cross-entropy loss) to optimize model parameters; the loss function calculates the difference between the model's output tokens and the target trajectory token sequence.

[0113]

[0114] in To output the sequence length, For the first One token, For the model in a given input image and instructions Output under the condition The probability of.

[0115] This concise end-to-end training method eliminates the need for complex loss function design, allowing the model to directly learn trajectory output through the language model's generative capabilities. The following enhancement strategies are employed during training: 1. Data augmentation: Randomly rotating, scaling, and adjusting the colors of the interface image; 2. Trajectory perturbation: Adding slight noise to the real trajectory to improve model robustness; 3. Multi-task learning: Simultaneously training data of different operation types to enhance generalization ability.

[0116] Compared to traditional multi-task loss function design, the training method based on pre-trained multimodal models is simpler and more efficient. It makes full use of the visual and language understanding capabilities of the pre-trained models and can obtain excellent GUI operation capabilities with only a small amount of fine-tuning data.

[0117] Through the above technical solution, this invention achieves end-to-end unified modeling of visual perception, language understanding, and motion trajectory prediction. Combined with external device physical operation and intelligent error correction mechanism, it breaks through many limitations of traditional GUI automation technology and provides an efficient, robust, and scalable technical solution for scenarios such as automated testing and RPA robots.

[0118] The beneficial effects of this invention are as follows:

[0119] 1. This invention uses external physical devices such as dexterous hands to directly operate real terminal devices, completely bypassing software-layer restrictions and detection such as system security checks, keyboard locks, black screens, and anti-debugging measures. The operation is triggered by real physical contact, leaving no characteristic processes or application signatures of the automated script on the device. This allows for deployment on real user devices, solving the problem that traditional automation tools can only run in specific environments.

[0120] 2. Through visual-language fusion using a multimodal large language model, the system can intelligently identify the position and function of interface elements, no longer relying on fixed coordinates or predefined UI tree structures. Even if the position of controls changes with screen size, resolution, and system version, the system can still accurately identify and locate the target element. The unified trajectory prediction method encodes information such as action type, operation coordinates, and touch pressure into a continuous trajectory, avoiding errors accumulated independently at multiple stages, achieving a trajectory prediction accuracy of over 94%.

[0121] 3. The dexterous hand robotic arm supports 6 degrees of freedom of motion, enabling precise control of touch intensity (0.1N-5N), contact angle (deviation <2 degrees), sliding speed, and path. It can simulate complex operations such as clicking, double-clicking, long-pressing, sliding, dragging, curved scrolling, and multi-touch. Compared to traditional touch event injection, physical operations can realistically simulate human finger behavior, supporting complex scenarios such as route dragging planning in map applications and handwritten signatures in drawing applications.

[0122] 4. By inputting multiple frames of images into a multimodal large language model, the system can recognize and process dynamic interface changes, including loading animations, ad carousels, pop-up fade-in / fade-out windows, and real-time progress bar growth. The system can predict interface state transitions and execute operations at the optimal time, avoiding accidental operations on unstable interfaces, reducing fixed waiting time, and shortening test execution time by more than 30%.

[0123] 5. Establish a global application stack monitoring and application switching graph. The system can automatically identify cross-application scenarios and execute jump-back patterns. Cross-application processes such as CAPTCHA acquisition, file selection, and payment redirection do not require manual intervention. The automation capability of cross-application workflows significantly reduces the complexity of test script writing and maintenance costs.

[0124] 6. By comparing the changes in the interface state before and after execution, the system identifies the execution result. With a built-in error correction strategy library, the system can automatically identify abnormal scenarios such as network timeouts, permission pop-ups, forced updates, and ad obstruction, and perform recovery operations such as retrying, canceling, and skipping. The system has self-diagnosis and automatic recovery capabilities, reducing manual intervention and significantly improving the success rate and stability of testing.

[0125] 7. Employing a pre-trained multimodal large language model, and using supervised fine-tuning with cross-platform data, a single model can be adapted to multiple platforms such as iOS, Android, HarmonyOS, Windows, and Web, eliminating the need to write and maintain separate test scripts for each platform. When applications iterate across platforms, the model does not require large-scale modifications, reducing maintenance costs by more than 60%.

[0126] 8. Supports natural language descriptions for test cases. A multimodal large language model automatically parses task objectives, operation objects, and operation parameters, supporting semantic matching of fuzzy descriptions (such as "click the login button"). Testers do not need professional knowledge of coordinate positioning and API calls, significantly lowering the technical threshold for test case writing and improving the readability and maintainability of test cases.

[0127] 9. Through human-computer collaborative demonstrations and learning, operational data from human experts is collected. A supervised fine-tuning strategy is employed for continuous optimization, resulting in constantly improving trajectory prediction accuracy and execution efficiency as data accumulates. Online learning and adaptive adjustments are supported to adapt to new applications and interface styles.

[0128] 10. The GUI automated execution system provided by this invention can be widely used in fields such as automated testing, RPA robots, intelligent customer service, regression testing, compatibility testing, performance testing, teaching demonstrations, and accessibility assistance, and has broad market application prospects and commercial value. The system's high accuracy, high robustness, and cross-platform capability make it an effective alternative to traditional automation technologies. Attached Figure Description

[0129] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0130] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0131] As shown in the figure, this embodiment provides an automated GUI interaction execution system based on multimodal perception and end-to-end VLA, taking the automated registration process testing of a mobile e-commerce application as the application scenario. The specific solution is as follows:

[0132] I. System Hardware and Software Configuration

[0133] Hardware configuration:

[0134] Dexterous Hand Robot: It adopts a 6-DOF robotic arm, equipped with soft fingertips, which can precisely control the touch force (0.1N-5N), with a positioning accuracy of 0.1mm and a contact angle deviation of <2 degrees;

[0135] Camera equipment: Industrial-grade high-speed camera, 1920×1080 resolution, 60fps frame rate, supports real-time video stream capture;

[0136] Device under test: Android smartphone, 6.5-inch screen, 2400×1080 pixel resolution;

[0137] Control computer: NVIDIA RTX 4090 GPU (24GB VRAM), Intel i9-13900K CPU, 64GB RAM

[0138] Software configuration:

[0139] Operating system: Ubuntu 22.04 LTS;

[0140] Deep learning framework: PyTorch 2.0 or the Transformers library (for loading pre-trained models);

[0141] The VLA decision module model uses a pre-trained multimodal large language model (such as Qwen-VL-Chat) with approximately 7B-14B parameters. It undergoes supervised fine-tuning to support GUI operation trajectory generation. The temporal window length is: frame;

[0142] II. Test Cases and Natural Language Understanding The test cases are for the registration process of an e-commerce application, described in natural language as follows:

[0143] "Open the e-commerce app, click the registration button, enter your mobile number 13800138000, click to get the verification code, you will be redirected to the SMS app, get the verification code, return to the e-commerce app, enter the verification code, set the password Abc123456, click the confirm registration button to complete the registration."

[0144] The VLA decision module's model can directly understand the above natural language description without additional word segmentation and encoding steps. Through its built-in language understanding capabilities, the model extracts semantic information such as the task objective (registration), the operation objects (registration button, phone number input box, verification code input box), and operation parameters (phone number, password).

[0145] In actual deployment, the system breaks down complete test cases into multiple sub-steps, considering only the currently needed step during each model inference. For example, when executing the first step, the model receives the task instruction "click the register button," rather than the complete test case. This step-by-step execution approach reduces the complexity of a single inference iteration and improves execution accuracy.

[0146] III. Implementation of Visual Perception and Multimodal Fusion

[0147] The visual perception module captures the screen video stream of the terminal device in real time. The camera captures images of the phone screen at a frame rate of 60fps, and the system processes one frame every 100ms. Before executing the "click the registration button" operation, the system captures a screenshot of the most recent frame, and at the same time, in order to provide timing context, the system retains the image history of the most recent 5 frames.

[0148] The VLA decision module uses a unified multimodal processing approach, simultaneously receiving image and text inputs and automatically performing visual-language fusion.

[0149] enter:

[0150] Image: [Screenshot of the current interface, resolution 2400×1080]

[0151] Instruction: "Click the register button"

[0152] History: [Screenshots of the last 4 frames, used for timing context]

[0153] Internally, the model automatically performs feature extraction and fusion through its pre-trained vision-language fusion mechanism (typically using a ViT visual encoder to process images and a Transformer decoder to process multimodal input). This mechanism can recognize the location, shape, color, and text label of the registration button on the interface, and determine the operation to be performed based on the instruction to "click the registration button".

[0154] For temporal modeling, the model captures dynamic changes in the interface by either using historical frames as additional image input or through its built-in memory mechanism. The system can choose to input temporal information into the model in the form of a multi-frame image sequence, or it can describe interface state changes in the instructions, such as "The interface has finished loading, and a blue registration button is displayed at the bottom," for the model to reference.

[0155] IV. Motion Trajectory Prediction and Execution

[0156] Trajectory prediction process:

[0157] The VLA decision module's model directly outputs structured motion trajectories through a tool invocation mechanism. For the "Click the register button" instruction, the model generates the following trajectory output:

[0158] {

[0159] "trajectory":[{"x":1200,"y":300,"pressure":0.0,"contact":0},

[0160] {"x":1230,"y":340,"pressure":0.1,"contact":0},

[0161] {"x":1245,"y":375,"pressure":0.5,"contact":1},

[0162] {"x":1248,"y":378,"pressure":1.2,"contact":1},

[0163] {"x":1248,"y":378,"pressure":1.2,"contact":1},

[0164] {"x":1248,"y":378,"pressure":1.2,"contact":1},

[0165] {"x":1248,"y":378,"pressure":0.8,"contact":1},

[0166] {"x":1248,"y":378,"pressure":0.3,"contact":0},

[0167] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0168] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0169] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0170] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0171] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0172] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0173] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0174] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0175] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0176] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0177] {"x":1250,"y":380,"pressure":0.0,"contact":0},

[0178] {"x":1250,"y":380,"pressure":0.0,"contact":0}

[0179] ],

[0180] "screen_size":{"width":2400,"height":1080}

[0181] }

[0182] The trajectory shows that:

[0183] Trajectory points 3-7 represent the contact phase, with coordinates concentrated around (1248, 378), which is the center of the registration button;

[0184] The touch pressure gradually increases from 0.5N to a peak of 1.2N, and then decreases to 0.8N;

[0185] The contact state changes from point 3 to 1 (contact) and point 8 to 0 (lifted).

[0186] This trajectory output method unifies the representation of all operation types, eliminating the need for additional action type classification or parameter regression modules.

[0187] Trajectory execution:

[0188] The motion execution module parses the trajectory JSON output by the model and controls the dexterity hand's movement according to the time sequence and coordinates of the trajectory points. The motion parameters are converted as follows:

[0189] The total duration is set to 400ms.

[0190] The time interval between each trajectory point is ms

[0191] The motion execution module controls the dexterous hand:

[0192] 1. Track point 1-2: Approach the registration button while hovering in the air, with your fingertip approximately 10mm from the screen.

[0193] 2. Track point 3: Fingertip touches the screen, applying an initial pressure of 0.5N.

[0194] 3. Trajectory points 4-6: Maintain contact and apply a peak pressure of 1.2N to simulate a clicking action.

[0195] 4. Trajectory point 7: Gradually reduce the pressure to 0.8N.

[0196] 5. Track point 8: When the fingertip lifts off the screen, the pressure drops to 0.3N.

[0197] 6. Track points 9-20: The finger returns to a safe position.

[0198] After execution, the system waits for the interface response and then enters the verification phase of the error correction and recovery module, forming a closed loop from trajectory prediction to execution and then to verification.

[0199] V. Implementation of the Error Correction and Recovery Mechanism

[0200] After the action of clicking the registration button is executed, the system process enters the error correction and recovery module. This module is responsible for identifying the execution result by comparing the interface state before and after execution, forming a closed loop from action execution to verification.

[0201] First, the system captures the interface image after execution and compares it with the interface image before the action was executed. The system calculates the magnitude of the interface state change based on pixel differences or feature differences. Let the difference value be... The system's preset threshold is (Based on pixel differences). When When the value exceeds this threshold, the interface changes as expected, and the execution is considered successful; when... If the value is below this threshold, the action is considered to have failed because it did not cause the expected interface change.

[0202] Assuming the calculation yields greater than the threshold The interface was determined to have changed as expected, and the click operation was successful.

[0203] like Less than the threshold, for example, after clicking, it is found that... If the error occurs, the execution is deemed to have failed, and the system will initiate an error correction mechanism. The system scan interface may encounter the following abnormal scenarios:

[0204] Detect whether a network error message icon (such as a red exclamation mark) appears, and identify it through image matching or the model of the VLA decision module;

[0205] Detect whether a permission request dialog box (such as "Allow camera access") pops up, and confirm it by recognizing it using OCR;

[0206] Detect whether there are ad overlays (such as "Download Now" buttons) and confirm through image matching;

[0207] The system maintains a library of error correction strategies, each strategy containing the identification conditions for exception types and the corresponding recovery operations.

[0208] Suppose a closable pop-up ad is detected above the registration button, and the "close" icon is identified. The system matches the corresponding error correction strategy, restoring the action to clicking the "close" icon. The system sends the restoration instruction "click the close icon" to the model in the VLA decision module, the model generates the corresponding click trajectory, and the action execution module executes the trajectory.

[0209] After performing the recovery operation, the system re-executes the steps to obtain the new interface and recalculates the interface differences. If the test is successful, the process continues to the next testing step, forming a complete "prediction-execution-verification-recovery" closed loop.

[0210] VI. Implementation of Text Input Operations

[0211] When the test case requires "enter mobile number 13800138000", the system predicts the complete input trajectory based on the current fusion features. The input text operation consists of multiple independent click trajectory sequences, each trajectory corresponding to a click action of a virtual keyboard key. The system directly outputs a continuous trajectory sequence through end-to-end trajectory prediction. The trajectory itself implicitly contains information such as operation type, key position, and click order.

[0212] Trajectory structure of input text:

[0213] Entering an 11-digit phone number requires 11 independent click trajectories, each containing approximately 10-15 trajectory points. The trajectory sequence can be represented as:

[0214]

[0215] in Indicates the first The click trajectory of each number key, Include A trajectory point.

[0216] Trajectory prediction process:

[0217] For scenarios involving inputting a mobile phone number, the trajectory prediction network directly outputs a complete trajectory sequence containing 11 sub-trajectories. To ensure the correspondence between trajectory points and virtual keyboard keys, the system employs a two-stage prediction method:

[0218] First, extract the text sequence of the phone number to be entered from the test case language description:

[0219]

[0220] The VLA decision module's model, based on the current interface image and the instruction "Enter mobile number 13800138000", outputs a sequence of click trajectories for the corresponding keys on the virtual keyboard. The model can recognize the layout of the virtual keyboard, understand the position of each number key, and generate the correct click trajectory.

[0221] Example of a trajectory for entering a mobile phone number:

[0222] Assume the virtual keyboard layout is as follows (coordinate values ​​are for example):

[0223] Button "1": Coordinates (480, 850)

[0224] Button "3": Coordinates (720, 850)

[0225] Button "8": Coordinates (720, 950)

[0226] Button "0": Coordinates (600, 1050)

[0227] The target coordinates of the 11 click trajectories predicted by the system are as follows (partial trajectory points):

[0228]

[0229] The execution process of each trajectory is similar to clicking the registration button: 1. Hover the finger close to the target button; 2. Gradually increase the touch pressure; 3. Maintain contact for about 100-200ms; 4. Gradually decrease the pressure and lift the finger.

[0230] Trajectory execution:

[0231] The action execution module receives the predicted trajectory sequence. The program executes 11 click trajectories in sequence, with a total duration of approximately 3-4 seconds. The execution parameters are as follows:

[0232] Duration of a single click trajectory: approximately 300ms

[0233] Inter-track interval: approximately 50ms

[0234] Touch pressure: 1.0N peak

[0235] The system pre-builds a coordinate mapping table for the virtual keyboard, but this is no longer used because the trajectory prediction network directly controls the dexterity hand movement based on the generated coordinates. The trajectory prediction network implicitly learns the key layout and can accurately predict the position of each number key.

[0236] After the input is completed, the system verifies the input result through the error correction and recovery module, checks whether "13800138000" is correctly displayed in the input box, and then performs the next operation (such as clicking the "Get Verification Code" button).

[0237] VII. Implementation of Cross-Application Coordination Mechanism

[0238] When a test case requires "redirecting to the SMS app, obtaining a verification code, and returning to the e-commerce application," the system flow enters the cross-application coordination module. This module works closely with the language understanding module, visual perception module, action execution module, and error correction and recovery module to automate the execution of the cross-application workflow. The cross-application coordination module operates according to the mechanisms designed for cross-application coordination.

[0239] Step 1: Identify the redirection requirement

[0240] The cross-application coordination module obtains semantic information about the test cases from the language understanding module and detects that the test cases contain the task of "getting a verification code". Simultaneously, the visual perception module captures the current e-commerce application interface, recognizes the text content through OCR, and detects the prompt message "Verification code has been sent to your mobile phone, please check your SMS". Combining the results of language understanding and visual perception, the system confirms that a cross-application verification code needs to be obtained and initiates the cross-application coordination process.

[0241] Step 2: Plan the redirection path

[0242] Cross-application coordination module queries application switching graph , where the node set Application, edge set This indicates the navigation relationship between applications. Calculate the probability of navigation from the e-commerce application to the SMS application using the transition probability formula:

[0243]

[0244] in This indicates the current application (e-commerce application). Indicates the target application (SMS APP). For application Jump to application Historical frequency. Query the application switching graph to obtain the transition probability. (Historical data shows that 72% of verification code retrieval operations require redirection to the SMS application). After confirming the redirection, the system performs a redirect-return strategy planning.

[0245] According to the definition of the jump-return strategy, the jump-return strategy Indicating in application and mission objectives The following sequence of operations should be executed:

[0246]

[0247] in The length of the operation sequence. For the first Step-by-step operation. For scenarios involving obtaining verification codes, the strategy is as follows: Click the Home button → swipe to the application list → click the SMS app icon → read the latest SMS message → extract the verification code → click the back button → return to the e-commerce application.

[0248] Step 3: Perform the redirect

[0249] The action execution module is based on the strategy. Perform the first step. The action execution module controls the dexterous hand to tap the phone's "Home" button, swipe to enter the application list, identify the "Messages" application icon through the visual perception module, calculate the icon's coordinates, and tap to launch it. The cross-application coordination module records the current application stack state changes and updates the application stack according to the application stack definition. In this embodiment ,in For the application at the top of the stack, This refers to the application stack depth.

[0250] Step 4: Find the verification code

[0251] After the action execution module reaches the SMS application interface via the verification code, the visual perception module refreshes the SMS collection interface and extracts the latest received SMS content using OCR. Regular expressions are used to match the verification code pattern (usually 4-6 digits). Assuming the recognized SMS content is "[E-commerce Application] Your verification code is 8629, please enter it within 10 minutes," the cross-application coordination module extracts the verification code "8629" and temporarily stores this data for subsequent input operations upon return.

[0252] Step 5: Return to the original application

[0253] The cross-application coordination module controls the action execution module's return operation in the execution strategy. Clicking the "back" button returns to the top of the application stack (e-commerce application). The visual perception module refreshes the e-commerce application interface again, confirming a successful return. The cross-application coordination module updates the application stack state according to the application stack definition. It contains only one application. , .

[0254] Step Six: Enter the verification code

[0255] Returning to the e-commerce application, the system flow reverts to a loop of language understanding, visual perception, multimodal fusion, action prediction, and action execution. The language understanding module parses the "input verification code" step in the test case, while the visual perception module captures the current interface and extracts its features. Following the aforementioned methods, the system performs multimodal fusion and action prediction, identifies the position of the verification code input box, and predicts the input text action. The action execution module executes the input operation, inputting the verification code "8629" temporarily stored by the cross-application coordination module.

[0256] The entire process is automatically coordinated by the cross-application coordination module, involving close collaboration among multiple modules such as language understanding, visual perception, action execution, and error correction and recovery, without requiring manual intervention. The cross-application coordination module continuously records changes in the application stack state during execution. Statistical application switching chart Continuously optimize the jump-return strategy by reducing the number of transitions. This will improve the efficiency of subsequent execution.

[0257] VIII. Training Data Collection and Model Training Implementation

[0258] Training data collection:

[0259] The system uses a human-computer collaborative demonstration learning approach to collect training data. Human experts record the actual operation process using screen recording software and device operation logs. For each operation step, the following information is recorded:

[0260] Visual data: Screenshots before and after the operation, with 5 consecutive frames captured for each operation;

[0261] Language data: Natural language descriptions of test cases;

[0262] Track annotation: Complete motion trajectory sequence, including There are 1 trajectory point, and each trajectory point contains coordinates. ,pressure and contact state ;

[0263] For input text operations: the character sequence to be input is labeled to guide the trajectory prediction network to generate multiple click trajectories;

[0264] A total of 100,000 operation samples were collected, covering applications on four platforms: iOS, Android, HarmonyOS, and Web, including 20 application types such as e-commerce, social networking, games, and tools. The distribution of trajectory types in the dataset is as follows: click (45%), swipe (20%), text input (18%), long press (8%), drag (6%), and double-tap (3%).

[0265] Model training implementation:

[0266] A three-stage training strategy is adopted:

[0267] Training process:

[0268] This embodiment employs a supervised fine-tuning method based on a pre-trained multimodal large language model, which simplifies the traditional multi-stage training process.

[0269] Data preparation:

[0270] Approximately 100,000 GUI operation trajectory data were collected, covering common operation types in mobile applications, including clicking, swiping, dragging, and text input. Each training sample contains:

[0271] Screenshot: RGB image of the current user interface;

[0272] Operation instructions: Text describing the operation, such as "Click the register button", "Scroll down the list", etc.

[0273] Target trajectory: The actual motion trajectory executed, presented in JSON format;

[0274] Monitor and fine-tune the configuration:

[0275] Base model: Qwen-VL-Chat (7B parameters)

[0276] Optimizer: AdamW

[0277] Learning rate: (Full parameter fine-tuning) or (LoRA fine-tuning)

[0278] Batch size: 4 (per GPU)

[0279] Training cycles: 3 epochs

[0280] Weight decay: 0.01

[0281] Gradient clipping: Maximum gradient norm 1.0

[0282] Learning rate scheduling: cosine annealing, warm-up ratio is 0.1.

[0283] Loss function:

[0284] Using standard language model cross-entropy loss:

[0285]

[0286] in To output the sequence length, For the first One token, For the model in a given input image and instructions Output under the condition The probability of.

[0287] This simple training method makes full use of the visual and language understanding capabilities of the pre-trained model, and can achieve excellent GUI operation capabilities with only a small amount of fine-tuning data.

[0288] Training and validation:

[0289] Monitor the following metrics on the validation set:

[0290] Mean Absolute Error (MAE) of Trajectory Coordinates

[0291] Contact status accuracy

[0292] Pressure regression error

[0293] Track integrity (correct track successfully generated)

[0294] After 3 epochs of training, the validation set performance is as follows:

[0295] Mean absolute error of trajectory coordinates: 8.7 pixels

[0296] Contact status accuracy: 96.3%

[0297] Average error of pressure regression: 0.28N

[0298] Track integrity: 94.2%

[0299] The model achieved performance comparable to that on the validation set on the test set, demonstrating good generalization ability.

[0300] IX. Inference Deployment and Performance Evaluation

[0301] The trained model is deployed to the control computer and interacts with the device under test via real-time video streaming. The complete execution flow of the system on a single registration test case is as follows:

[0302] 1. Receive test case natural language description (approximately 50ms)

[0303] 2. Visual perception and feature extraction (approximately 100ms)

[0304] 3. Multimodal fusion and temporal modeling (approximately 80ms)

[0305] 4. Trajectory prediction (approximately 40ms)

[0306] 5. Trajectory execution (dexterous hand movement along the trajectory + touch control, approximately 200-400ms)

[0307] 6. Error correction and recovery (if needed, approximately 50-100ms)

[0308] The end-to-end latency for a single-step operation is approximately 480-730ms, meeting real-time requirements. The entire registration process comprises 12 steps, with a total execution time of approximately 7-9 seconds, comparable to human manual operation speed, but with better stability and repeatability.

[0309] In real-world testing, the system was tested on the registration processes of 100 different applications, achieving a success rate of 96.8%. Failures were primarily due to rare anomalies in a very small number of applications (such as forced restarts during system updates). Compared to traditional coordinate- and UI-tree-based automation tools, the success rate is improved by 18 percentage points, and maintenance costs are reduced by 60%. The trajectory prediction method allows the system to more naturally simulate human operating characteristics, reducing deviations caused by hard-coded coordinates, making it particularly suitable for complex gesture operations and cross-device scenarios.

[0310] In summary, this invention constructs a unified end-to-end vision-language-trajectory model. It simultaneously inputs multimodal information such as current interface screenshots, historical trajectory sequences, and test case text descriptions. Through a multimodal Transformer encoder, it fuses visual and linguistic representations to directly generate a complete motion trajectory that includes spatial location, touch pressure, and contact state. This design avoids the error accumulation caused by independent predictions at multiple stages, achieving integration from perception to decision-making.

[0311] This invention overcomes the limitations of traditional methods that rely solely on single-frame static screenshots. It introduces continuous multi-frame video streams as input and utilizes a spatiotemporal Transformer to model the dynamic changes of the UI interface (such as automatic ad sliding, loading animations, and pop-up windows). By aggregating multi-frame information through a temporal attention mechanism, it integrates temporal context information during action decision-making, predicts possible state transitions of the interface, and plans corresponding strategies in advance.

[0312] This invention innovatively employs external hardware devices such as dexterous hands to directly operate actual terminal devices, rather than through command-line tools, adb, emulators, or other virtual interfaces. This completely bypasses software-level restrictions and detections such as device security checks, keyboard locks, and black screens. The physical operation realistically simulates human finger and mouse operations, ensuring consistency between the operating actions and real human usage behavior.

[0313] This invention establishes a global application stack monitoring mechanism to track the order and status of activated applications in real time. It models cross-application navigation patterns through an application switching graph, learns navigation-return strategies, and supports cross-application workflow execution (e.g., navigating from the main application to a messaging app to obtain a verification code, and then returning to the original application). The system remembers the interface state before the navigation, enabling automatic navigation and backtracking.

[0314] This invention identifies whether execution was successful by comparing the magnitude of changes in the interface state before and after execution (L2 norm). For execution failures or interface anomalies, a built-in error correction strategy library attempts recovery operations such as retrying, clicking cancel, and skipping. Simultaneously, an anomaly scenario classifier is constructed to identify different error types such as network timeouts, permission pop-ups, and forced updates, dynamically adjusting the recovery strategy.

[0315] This invention collects operational data from multiple platforms, including iOS, Android, HarmonyOS, Windows, and Web, during model training. It uses platform-independent feature representations (based on visual elements rather than system-specific UI trees) and learns cross-platform shared underlying operation strategies through multi-task learning. Simultaneously, platform identifier inputs are added to the model as auxiliary features to learn platform-specific adaptation parameters, enabling a single model to adapt to multiple platforms.

[0316] This invention introduces imitation learning, recording real-world demonstrations by human experts (screen recording + device recording), extracting the visual-language-action triplet, and learning human operational strategies through behavioral cloning. For complex dragging, multi-touch, and other difficult-to-define actions, human demonstration data is used to improve model performance. Simultaneously, reinforcement learning is used to explore optimal strategies in a simulated environment, with reward functions including task completion rate, number of operation steps, and success rate.

[0317] This invention inputs high-level test case descriptions (natural language) into the VLA decision module model. The model utilizes its built-in language understanding capabilities to extract information such as task objectives, operation objects, and operation parameters. For vague or uncertain descriptions (such as "click the login button"), the model's multimodal reasoning capabilities are used for fuzzy localization, supporting flexible descriptions in natural language rather than precise coordinates, thus lowering the barrier to test case writing.

[0318] The above embodiments are merely typical illustrative methods of the present invention, and the scope of protection of the present invention is not limited thereto. All equivalent substitutions and improvements made under the concept of the present invention should fall within the scope of protection. It should be emphasized that any modifications or minor adjustments made by those skilled in the art without departing from the basic principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An automated GUI interactive execution system based on multimodal perception and end-to-end VLA, characterized in that, include: It consists of seven core modules: visual perception module, language understanding module, VLA decision-making module, action execution module, temporal modeling module, cross-application coordination module, and error correction and recovery module. The visual perception module captures the dynamic changes and spatial layout of the interface through multi-frame video streams; the language understanding module parses the natural language descriptions of test cases and extracts task objectives and operational constraints; the VLA decision-making module adopts a unified multimodal large language model, takes interface images and task instructions as input, and directly outputs structured motion trajectories through tool invocation capabilities. The trajectory implicitly encodes the action type, operation coordinates, and touch pressure information, realizing end-to-end learning from perception to decision-making. The action execution module controls the external physical device of the dexterous hand, which directly contacts the touch screen of the real terminal device; the timing modeling module models the dynamic changes and timing context of the UI interface and predicts the interface state transition; the cross-application coordination module monitors the global application stack state and realizes the automatic execution of cross-application workflows. The error correction and recovery module identifies the execution result by comparing the interface state before and after execution, and performs anomaly recovery using a built-in strategy library.

2. The automatic GUI interactive execution system based on multimodal perception and end-to-end VLA according to claim 1, characterized in that, The VLA decision module adopts a unified multimodal large language model architecture, which can directly process image and text input and output structured motion trajectories through built-in tool calling capabilities; it uses a pre-trained multimodal base model, which is fine-tuned to enable it to understand and plan GUI operations; Suppose the system is at the th The input received at any given time includes: the current interface image. ,in and These are the height and width of the image, respectively; the natural language description of the test cases. Historical interface feature sequence Multimodal large language models encode these inputs uniformly into an internal representation: in Represents the encoding function of a multimodal large language model. For a unified representation within the model; After fine-tuning, the model can correctly understand the position, function, and interaction method of interface elements based on the input interface image and task instructions, and convert them into executable motion trajectories.

3. The automatic GUI interactive execution system based on multimodal perception and end-to-end VLA according to claim 1, characterized in that, The VLA decision module outputs a structured motion trajectory through a tool invocation mechanism; the system defines a trajectory generation tool, execute_trajectory, for the model, which accepts structured trajectory parameters as input. The trajectory output by the model is directly parsed and executed by the action execution module. The trajectory itself implicitly encodes the operation type information: click operation; swipe operation; drag operation; text input operation.

4. The automatic GUI interactive execution system based on multimodal perception and end-to-end VLA according to claim 1, characterized in that, The system is designed with trajectory buffering and real-time control mechanisms to ensure that the dexterous hand can smoothly and accurately follow the trajectory. Assuming the video stream frame rate is The frame interval is The system continuously captures video frames. When the first When the model is inference with a frame input, the model outputs the trajectory. However, at this point the video had already progressed to the [number]th [stage / section]. Frame, in which The number of frames corresponding to the model inference latency: in For model inference time; if fps and ms, then frame; Using the trajectory output by the model directly may result in: (1) Time base offset; (2) Execution is discontinuous; (3) Cumulative position error; The system employs a trajectory buffer queue combined with real-time interpolation. Trajectory timestamp mapping: Each trajectory point output by the model is accompanied by an absolute timestamp, rather than a relative time proportion; assuming the model is at the [missing information]th ... If the trajectory is output at time 1, then the 1st... The expected execution time for each trajectory point is: in Due to trajectory execution delay, Relative time, This represents the total duration of the trajectory. Track Buffer Queue: The system maintains a track point buffer queue. Each buffer item contains a timestamp of a trajectory point. ,coordinate ,pressure Contact status : When the model outputs a new trajectory, the trajectory points are inserted into the buffer queue in timestamp order; Real-time trajectory query: Dexterous hand controller with fixed control frequency Query current time The status of the trajectory points to be executed; the query method is: like Equal to the timestamp of a certain trajectory point Then use that point directly. like lie in and Between these points, linear interpolation is performed: Trajectory smoothing and path planning: To avoid abrupt changes in dexterity hand movements, the interpolated coordinates are smoothed using a filtering process. in For smoothing coefficients, For the first Smoothing position of each control cycle; Trajectory overlap handling: When a new trajectory arrives, if there are still unexecuted points in the buffer queue, the system determines whether to discard the old trajectory or perform a smooth transition; for continuous operations on input text, the endpoint of the old trajectory is retained. As the starting point of a new trajectory : Timeout handling and safety mechanism: If the buffer queue does not have a new trajectory point within a preset time, the dexterous hand will automatically lift the finger and return to a safe position to avoid accidental operation caused by prolonged contact with the screen.

5. The automatic GUI interactive execution system based on multimodal perception and end-to-end VLA according to claim 1, characterized in that, The temporal modeling module aims to capture the dynamic changes of the GUI interface in the time dimension; the system takes multiple consecutive frames of images as input, enabling the VLA decision module's model to understand and predict the temporal evolution of the interface state. Let the input video frame sequence be ,in This refers to the timing window length; in actual deployment, the system chooses one of the following two methods to provide timing information: Multi-frame image input: The multi-frame image sequence is directly input into the VLA decision module model. The model captures the inter-frame dependencies through its built-in attention mechanism and identifies the timing of the completion of the loading animation, the update rhythm of the ad carousel, and the dynamic changes in the disappearance process of the pop-up. Text description assistance: Describe the interface state change information in the task instructions for the model's reference; The VLA decision module's model, through its pre-trained visual understanding capabilities, can identify dynamic elements in the interface; combined with the timing constraints in natural language instructions, the model can intelligently determine the best time to perform an operation, avoiding execution when the interface is unstable.

6. The automatic GUI interactive execution system based on multimodal perception and end-to-end VLA according to claim 1, characterized in that, The cross-application coordination module works closely with the language understanding module, visual perception module, and action execution module to monitor the global application stack status and enable automatic execution and navigation backtracking of cross-application workflows; the system maintains an application stack. ,in Indicates the first in the application stack One application, The top of the stack, This represents the application stack depth. When an application switching event is detected, the system records the current application stack state and a snapshot of the interface to provide a basis for subsequent navigation. To support intelligent decision-making across application scenarios, the system constructs an application switching graph. , where the node set Application, edge set This represents the navigation relationship between applications; each edge Associate a transition probability , indicating in application Jump to application The probability of; It was obtained through statistical analysis of historical execution data: in For application Jump to application The historical frequency, where the denominator represents the number of times the application has been used. Total number of jumps to all apps in history; When navigation is required across application scenarios, the system plans the optimal jump path based on the current application stack and task objectives; for scenarios that require jumping to other applications to obtain information and then returning, the system learns the "jump-operation-return" pattern. Specifically, define the jump-back strategy. , indicating in application and mission objectives The following sequence of operations should be executed: in The length of the operation sequence. For the first Step-by-step operation For the mission objective; When a verification code is detected, the system executes the following cross-application coordination process: First, the language understanding module combined with OCR recognizes the verification code prompt information. Then, the action execution module launches the SMS APP, the visual perception module reads the SMS interface, extracts the verification code using regular expressions, and the action execution module returns to the original application, enters the verification code in the corresponding input box, and completes the cross-application verification.

7. The automatic GUI interactive execution system based on multimodal perception and end-to-end VLA according to claim 1, characterized in that, The error correction and recovery module forms a closed loop with the visual perception module and the action execution module. It identifies the execution result by comparing the interface state before and after execution, and performs anomaly recovery using a built-in strategy library. The motion execution module follows the predicted motion trajectory. After the operation is executed, the visual perception module captures the resulting interface image; the system determines whether the operation was successful using the following two methods: Interface similarity detection: Calculate the pixel or feature differences between the interface images before and after execution. When the difference exceeds the threshold, it is considered that the operation has caused a change in the interface and is judged as successful. Model result verification based on VLA decision module: Input the interface image after execution and the description of the expected result into the model of VLA decision module, and the model judges whether the current state meets the expectations; Meanwhile, the system detects the characteristics of common abnormal scenarios; for network timeouts, it checks whether the loading indicator has been spinning for an extended period; for permission pop-ups, it checks whether a "Allow access" or "Deny" dialog box has appeared. For forced updates, check if "Update Now" and "Update Later" buttons appear; The system maintains an error correction strategy library. ,in For the number of strategies, A single error correction strategy includes the criteria for identifying the type of exception and the corresponding recovery action: in For anomaly identification conditions, The recovery action is either a new trajectory or a recovery command sent to the model in the VLA decision module to generate a new trajectory. When an interface anomaly is detected, the system executes the corresponding recovery strategy. After the recovery operation is executed, the interface status is checked again to determine whether it has returned to normal. If it has returned to normal, the subsequent tasks continue. If multiple retries still fail, an error log is recorded and manual intervention is notified.

8. The automatic GUI interactive execution system based on multimodal perception and end-to-end VLA according to claim 1, characterized in that, The VLA decision module model is based on a pre-trained multimodal large language model, which is then fine-tuned under supervision to enable it to master GUI operation capabilities. The training data is collected from the operation trajectory records of real users. Each training sample consists of an interface image, task instructions, and target trajectory. Each training sample contains three parts:

1. Interface image: a screenshot of the current moment; 2. Task Instructions: Text describing the current operation; 3. Target Trajectory: The actual operation trajectory, presented in a tool call format. During training, the model receives interface images and task commands as input, and the target output is the correct tool call. The training process employs supervised fine-tuning, using a standard language model loss function to optimize model parameters; the loss function calculates the difference between the model's output tokens and the target trajectory token sequence. in To output the sequence length, For the first One token, For the model in a given input image and instructions Output under the condition The probability of; The following augmentation strategies were used during training:

1. Data augmentation: Randomly rotate, scale, and adjust the colors of the interface images; 2. Trajectory perturbation: Add slight noise to the real trajectory; 3. Multi-task learning: Train data of different operation types simultaneously to improve generalization ability.

Images