Gui automation manipulation method and system fusing visual language reasoning

Abstract

This application discloses a GUI automated manipulation method and system integrating visual language reasoning, comprising: generating text prompts based on user input and system configuration information; determining target GUI elements based on the stage of the target operation; identifying the coordinate information of the target GUI elements from the current screen image of the client interface based on the target GUI elements and text prompts using a multimodal large language model to complete the target operation; after executing the target operation, taking the dialogue area of ​​the target operation as the target GUI element, capturing a screenshot of the dialogue area based on its coordinate information, extracting test information from it, testing it on the terminal, and determining whether to terminate the operation based on the test results. This solves the technical problem that existing computer-based intelligent agent implementation methods are difficult to apply to complex scenarios and have limited versatility and generalization ability. It can improve the versatility and generalization ability of computer-based intelligent agents in complex scenarios and effectively cope with various software interfaces and complex tasks.

Description

Technical Field

[0001] This application relates to the field of graphical user interface manipulation technology, and in particular to a GUI automated manipulation method and system that integrates visual language reasoning. Background Technology

[0002] Computer Use Agents (CUAs) are software entities built upon large language models. Their core objective is to simulate human interaction with computer systems, autonomously performing user-specified tasks such as operating system management, application invocation, and file management. By recognizing and manipulating system interfaces or graphical user interface (GUI) elements, they improve the efficiency of repetitive tasks and expand the boundaries of human-computer collaboration. The Vision Language Model (VLM), a key technology supporting CUA's visual interaction capabilities, is a multimodal artificial intelligence system integrating visual and language understanding. It relies on a deep learning architecture to simultaneously process visual inputs such as images and videos, along with textual information. Its core lies in a unified representation learning framework, aligning visual features with the semantic space to achieve complex tasks such as visual question answering, image description, and cross-modal retrieval, providing core technical support for the understanding and localization of GUI elements such as desktop icons. Graphical User Interface Agents (GUI Agents) are an important subclass of CUA, specifically adapted for graphical user interface environments. Its technical logic is to capture screen pixels in real time through computer vision, combine optical character recognition (OCR) or element detection algorithms to analyze the interface state, and then generate operation sequences through predefined rules or autonomous decision-making to drive peripherals such as mice and keyboards to complete interactions such as clicking, input, and navigation. It is widely used in software testing, process automation and auxiliary tool development, and its core value lies in replicating human peripheral operation modes to achieve efficient human-machine collaboration.

[0003] Currently, the mainstream implementation paradigms of CUA fall into two categories. One is the interaction mode based on Application Programming Interfaces (APIs). This interaction mode relies on APIs to execute tasks, completing instructions by calling the target software's open interfaces (such as file operations and data interaction). It achieves precise control through standardized interfaces, showing significant efficiency in scenarios with complete interfaces. The other is the GUI direct control mode based on visual perception. This direct control mode relies on technologies such as computer vision and OCR (Optical Character Recognition) to identify interface elements and analyze interface states at the pixel level. It does not rely on APIs and drives peripherals to complete interactions by generating operation sequences. Its technological advantage lies in simulating human visual cognition and operational logic, theoretically adaptable to various visible software interfaces. Currently, it is mainly used for simple, repetitive tasks such as information retrieval, form filling, and button clicking.

[0004] While both approaches have their own characteristics, their limitations significantly restrict the versatility and potential of CUA: The interaction mode based on application programming interfaces (APIs) is highly dependent on whether the target software provides an open, stable, and fully functional API. When facing legacy systems, closed commercial software, or desktop applications without automation support, CUA struggles to function due to the lack of available interfaces, severely limiting its versatility and deployment scope. The GUI-based direct manipulation mode, based on visual perception, is limited to simple, repetitive tasks. Its theoretically high degree of "operational freedom" (the ability to manipulate any visible software interface) is not effectively utilized, failing to address the complex task planning and execution needs of closed software and applications without automation design, thus severely compressing the technology's application scenarios. Summary of the Invention

[0005] This application provides a GUI automated manipulation method and system device that integrates visual language reasoning, solving the technical problem that existing computer-based intelligent agent implementation methods are difficult to apply to complex scenarios and have limited versatility and generalization capabilities.

[0006] In a first aspect, embodiments of this application provide a GUI automation manipulation method integrating visual language reasoning, comprising: generating text prompts based on user input and system configuration information, and executing task processing steps based on them; wherein, the task processing steps include: determining a target GUI element based on the stage of the target operation, and identifying the coordinate information of the target GUI element from the current screen image of the client interface using a multimodal large language model based on the target GUI element and the text prompts, in order to complete the target operation, comprising: before executing the target operation, taking the operation object as the target GUI element, and using the multimodal large language model in conjunction with the text prompts to identify the coordinate information of the target GUI element from the current screen image of the client interface, and executing the target operation based on the coordinate information. The process involves several steps: During the execution of a target operation, using a timestamp as the target GUI element and combining it with the text prompts, a multimodal large language model is used to identify the coordinate information of the target GUI element from the current screen image of the client interface. Based on this coordinate information, an image of the element's region is periodically captured, and the timestamp information within it is identified to determine whether the target operation has been completed. After the target operation is executed, using the dialogue area of ​​the target operation as the target GUI element and combining it with the text prompts, a multimodal large language model is used to identify the coordinate information of the target GUI element from the current screen image of the client interface. Based on this coordinate information, a screenshot of the dialogue area is captured, and test information is extracted from the screenshot. The test information is then tested on the terminal, and the operation is terminated based on the test results.

[0007] In conjunction with the first aspect, in one possible implementation, before the task processing step is executed, the method further includes: initializing the client of the multimodal large language model based on the system's API configuration and verifying the validity of the key; and / or setting the scaling ratio of the client interface and fixing the interface position of the client interface; and / or setting a delay time to ensure the stability of the window layout of the client and the terminal.

[0008] In conjunction with the first aspect, in one possible implementation, generating text prompts based on user input and system configuration information includes: generating text prompts based on user input and system configuration information, and adding a summary and test commands to the end of the text prompts, so that the multimodal large language model generates a progress summary and executable test information after each round of tasks.

[0009] In conjunction with the first aspect, in one possible implementation, the coordinate information of the target GUI element includes the position coordinates and element size of the target GUI element.

[0010] In conjunction with the first aspect, in one possible implementation, the step of periodically capturing an element region image based on its coordinate information, using a timestamp as the target GUI element, and identifying the timestamp information therein to determine whether the target operation is completed includes: acquiring the current screen image once every first time interval and performing an identification and judgment step; wherein, the identification and judgment step includes: converting the current screen image into a region image code; inputting the region image code and the corresponding text prompt word into a multimodal large language model to identify the coordinate information of the target GUI element in the current screen image; cropping the element region image containing the timestamp based on the coordinate information; inputting the element region image into a visual language inference model to identify the timestamp information therein, and obtaining an identification result; if the identification result includes timestamp information, the target operation is completed; if the identification result does not include timestamp information, then waiting until the first time interval is reached to recapture the current screen image and perform the identification and judgment step.

[0011] In conjunction with the first aspect, in one possible implementation, testing the test information on the terminal includes: acquiring the current screen image of the terminal, determining the position information of the terminal interface through a multimodal large language model, and recording the current position of the mouse; and / or, clearing the current input in the terminal interface after waiting for a delay time, and waiting for the delay time again; and / or, formatting the test information and inputting it into the terminal interface for testing.

[0012] In conjunction with the first aspect, in one possible implementation, the step of testing the test information on the terminal and determining whether to terminate the operation based on the test results further includes: generating a decision using a large language module based on task information, test information, and the test results, including: if the test result indicates that the operation is normal but the task is not completed, then optimizing the text prompt words using the large language module based on the test information and updating the target operation according to the task information; executing the task processing steps based on the optimized text prompt words and the updated target operation until the task is completed; if the test result indicates that the operation is abnormal, the termination condition is met, or the task is completed, then the operation is terminated, and the current result is output.

[0013] Secondly, embodiments of this application provide a GUI automated manipulation system integrating visual language reasoning, comprising: a task module, used to generate text prompts based on user input and system configuration information, and execute task processing steps based on them; wherein, the task processing steps include: a recognition module, used to determine a target GUI element according to the stage of the target operation, and based on the target GUI element and the text prompts, use a multimodal large language model to recognize the coordinate information of the target GUI element from the current screen image of the client interface to complete the target operation, comprising: a first execution module, used to, before executing the target operation, take the operation object as the target GUI element, combine the text prompts, use a multimodal large language model to recognize the coordinate information of the target GUI element from the current screen image of the client interface, and execute the target operation based on the coordinate information. The system comprises the following modules: a second execution module, used to identify the coordinate information of the target GUI element from the current screen image of the client interface using a multimodal large language model, with the timestamp as the target GUI element and the text prompts, periodically capturing the element region image based on the coordinate information and identifying the timestamp information therein to determine whether the target operation is completed; a third execution module, used to identify the coordinate information of the target GUI element from the current screen image of the client interface after the target operation is executed, with the dialog area of ​​the target operation as the target GUI element and the text prompts, using a multimodal large language model, capturing a screenshot of the dialog area based on the coordinate information and extracting test information from the screenshot; and a testing module, used to test the test information on the terminal and determine whether to terminate the operation based on the test results.

[0014] Thirdly, embodiments of this application provide an apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method as described in the first aspect or any possible implementation of the first aspect.

[0015] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium including storage for storing a computer program or instructions that, when executed, cause the method described in the first aspect or any possible implementation of the first aspect to be implemented.

[0016] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This application's embodiments generate text prompts based on user input and system configuration information, enabling the generation of text prompts according to different task requirements and system environments. Task processing steps are executed based on these text prompts. The coordinate information of target GUI elements is identified from the current screen image of the client interface, achieving precise positioning and operation of GUI elements. At different stages of the target operation, the operation object, timestamp, and dialogue area are used as target GUI elements for processing, allowing for comprehensive monitoring of the operation process and acquisition of key information. Before executing the target operation, the coordinate information of the operation object is identified in advance, ensuring the accuracy and efficiency of the operation. During the execution of the target operation, the completion of the operation is determined by periodically capturing timestamp information, solving the problem of dynamically judging the operation stage based on the state of interface elements in traditional methods, and achieving real-time monitoring of the operation process. After executing the target operation, a screenshot of the dialogue area is captured and test information is extracted for testing, enabling timely detection of problems during the operation and ensuring the efficiency and stability of the operation process. This application integrates visual language reasoning, solving the technical problem that existing methods for implementing computer-based intelligent agents are difficult to apply to complex scenarios and have limited versatility and generalization ability. It can improve the versatility and generalization ability of computer-based intelligent agents in complex scenarios, effectively cope with various software interfaces and complex tasks, and provide a more powerful solution for the automated manipulation of graphical user interfaces. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the GUI automated manipulation method integrating visual language reasoning provided in this application embodiment; Figure 2 A schematic diagram of the structure of the GUI automated manipulation device integrating visual language reasoning provided in the embodiments of this application; Figure 3The current screen image of a first type of client window or terminal window provided in the embodiments of this application; Figure 4 The current screen image of the second type of client window or terminal window provided in the embodiments of this application; Figure 5 The current screen image of a third type of client window or terminal window provided in the embodiments of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0021] Figure 1 This is a flowchart of a GUI automated manipulation method integrating visual language reasoning provided in an embodiment of this application, including steps 101 to 105. Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order of the GUI automation manipulation method that integrates visual language reasoning. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0022] Step 101: Generate text prompts based on user input and system configuration information, and determine the target GUI element based on the stage of the target operation. In this embodiment, text prompts are generated based on user input and system configuration information, and a summary and test commands are added to the end of the text prompts so that the multimodal large language model generates a progress summary and executable test information after each round of tasks. The test information includes test scripts and code summaries.

[0023] Specifically, based on the user-input task description and system configuration information, such as system rules and API configuration information, a text prompt adapted to the IDE is generated. Here, the Large Language Model (GPT-4o-mini, a lightweight, low-cost, high-performance AI model from OpenAI) is used as input for the user-input task description, combined with system rules and API configuration information, to output a complete text prompt. The text prompt should instruct the Large Language Model to add a summary and test commands to the end of the returned information.

[0024] Furthermore, text prompt generation and optimization dynamically optimize text prompts based on test results and test information to ensure the correct task execution direction. The initial text prompt generation function `generate_init_prompt` receives a string input by the user as a parameter. First, it retrieves the system rules (SYSTEM_PROMPT), which include several instructions: the first is "no human interaction," ensuring that the IDE does not wait for user input when executing tasks; the second is "provide a code summary after each round," requiring the IDE to provide a progress summary in the IDE dialog box after each round of tasks; the third is "provide a test script," requiring the IDE to provide a test script (usually by running a file, such as `python file_path.py`); and the fourth is "test results must be output to the console," ensuring that test results can be captured by the system through terminal output. Then, the API configuration information is retrieved, and the API configuration prompt is generated using the `api_config_template` function. This function reads the API key information from environment variables (such as `ALTERNATIVE_OPENAI_API_KEY`, `ALTERNATIVE_OPENAI_API_BASE`, `ALTERNATIVE_WECHAT_APP_ID`, `ALTERNATIVE_WECHAT_APP_SECRET`, etc.) and generates the configuration prompt in the format "Here are the API configuration information you need to follow WHEN YOUNEED THEM. OpenAI API key: {key}, OpenAI API base: {base}, ...". Finally, the system rules, user input (formatted as "User input: {user_input}"), and API configuration information are concatenated into a complete text prompt, logged as "Generating initial prompt...", and returned.

[0025] Step 102: Before executing the target operation, taking the operation object as the target GUI element, and combining the text prompts, a multimodal large language model is used to identify the coordinate information of the target GUI element from the current screen image of the client interface, and the target operation is executed based on the coordinate information. In this embodiment, the coordinate information of the target GUI element includes the position coordinates and element size of the target GUI element. The client is exemplarily an IDE (an abbreviation for Integrated Development Environment, a software tool that provides developers with one-stop code writing, debugging, testing, and deployment functions).

[0026] like Figures 3 to 5 The image shows the current screen images of three different client or terminal windows. Specifically, the client window contains various GUI elements, such as input boxes, buttons, and terminal windows. By recognizing the coordinate information of the target GUI element, operations such as mouse clicks, keyboard input, and window navigation are performed on the target GUI element, achieving automated control of the IDE. This is implemented based on GUI automation libraries such as PyAutoGUI, providing mouse operation functions including click, double-click, drag, and move; keyboard operation functions including text input, shortcut key combinations, and special key operations; and a screenshot function that can capture a specified area of ​​the current screen.

[0027] Furthermore, the system leverages the visual understanding capabilities of a multimodal large language model to analyze the current screen image and identify and locate target GUI elements. First, a screenshot of the current screen is captured. Then, the current screen image, along with a text prompt describing the target GUI element, is input into the multimodal large language model. The model analyzes the GUI elements in the current screen image using visual understanding, identifying the target GUI element's location information (including its X and Y coordinates, width, and height), and returning structured location information. For example, when the target GUI element is an IDE dialog box, the system captures the current screen image and provides the multimodal large language model with the text prompt "Please identify and locate the location information of the IDE dialog box." The model analyzes the current screen image and returns the IDE dialog box's location information. When the target GUI element is a terminal window, the system provides the prompt "Please identify and locate the location coordinates of the terminal window," and the model returns the terminal window's location information. This dynamic positioning method based on visual understanding allows the system to adapt to different window layouts, resolution settings, and multi-monitor configurations without requiring pre-configured coordinate parameters, achieving intelligent target GUI element positioning.

[0028] For example, the multimodal large language model exemplifies the use of Google Gemini-2.5-flash (an AI model launched by Google on April 10, 2025, a hybrid inference model balancing quality, cost, and latency). Through Google Gemini-2.5-flash, visual understanding of the current screen image is achieved, extracting key information such as task execution status, code summary, test scripts, terminal output, and screen timestamps, while simultaneously enabling intelligent location of target GUI elements. First, the current screen image (or region image) is converted to Base64 (an encoding scheme based on 64 printable characters, primarily used to convert binary data into plain text format) encoding and sent to the multimodal large language model along with text prompts. The multimodal large language model returns structured text information or location information. The system uses Google Gemini-2.5-flash for visual understanding, and the specific process includes two stages: image preprocessing and API calls. In the image preprocessing stage, the system uses the pyautogui.screenshot function to capture the current screen image, saves it as a PNG file, and the filename includes a timestamp to ensure uniqueness, with the format statics / img_ <timestamp>The system first takes a .png file and uses Python's `open` function to open it in binary mode ('rb'). Then, it uses the `base64.b64encode` function to encode the PNG file into Base64 format, and finally uses the `decode("utf-8")` method to convert the Base64 byte data to a UTF-8 string, resulting in a Base64 encoded string. During the API call phase, the system creates a `GenerateContentConfig` object, setting `response_mime_type` to "application / json" to ensure the multimodal large language model returns a JSON response for easier parsing later. Then, it calls the `gemini_client.models.generate_content` method, setting the model parameter to `GoogleGemini-2.5-flash`. The `contents` parameter contains two parts: the first part uses the `types.Part.from_bytes` method to create the image portion, the `data` parameter is a Base64 encoded string, and the `mime_type` parameter is 'image / png', specifying the image type.The second part consists of text prompts. Specific text prompts are designed for different target operations. For example, the prompt for locating a target GUI element is "Please identify and locate the position coordinates of [target element description], and return JSON format: {'x': x-coordinate, 'y': y-coordinate, 'width': width, 'height': height}". The prompt for extracting a timestamp is "Extract the text in this image. Return only the text, and do not include any other information." The prompt for extracting a test script is "Extract the unit test script from the screenshot. Common unit test scripts are usually by running a file like this: python file_path.py. Return only the script without quotation marks." The prompt for extracting a code summary is "Extract the cursor summary information from the screenshot. Return only the summary information, and do not include any other information." The prompt for extracting terminal information is "Extract the terminal information from the screenshot. Return only the terminal information, and do not include any other information." After Google Gemini-2.5-flash returns a response, the system uses the `response.text` property to retrieve the text result (i.e., text information). For text results requiring further processing (such as test scripts), the `strip()` method is used to remove leading and trailing whitespace. The response format configuration model returns JSON format for easy subsequent parsing and processing. The system records detailed log information, including the type of extraction operation, the text prompts used, and the returned results, facilitating troubleshooting and system improvement.

[0029] In addition, before executing step 201, the client of the multimodal large language model can be initialized based on the system's API configuration, and the validity of the key can be verified; and / or, the scaling ratio of the client interface can be set, and the interface position of the client interface can be fixed; and / or, a delay time can be set to ensure the stability of the window layout of the client and the terminal.

[0030] Specifically, verifying the system API configuration before starting the operation is fundamental to ensuring the smooth progress of subsequent processes. The client for the multimodal large language model is initialized based on the system's API configuration, including initializing the LLM client (OpenAI compatible API and Gemini API), setting up a logger, and waiting for a user-specified delay to ensure stable window layouts on the client and terminal.

[0031] Furthermore, the system configuration requirements include hardware requirements, software requirements, API configuration, and runtime environment requirements. Hardware requirements include support for Windows / macOS / Linux operating systems, at least one monitor (multi-monitor configuration supported), and a stable network connection (for calling the LLM API). Software requirements include Python 3.9 or later, an installed IDE (such as Cursor IDE), and necessary Python dependencies such as pyautogui, openai, and google-genai. Regarding API configuration, the system supports multiple LLM service providers, including OpenAI-compatible APIs (including the official OpenAI and OpenRouter APIs) and the Google Gemini API. Configuration is achieved by setting the API key and base URL through a .env file, including OPENAI_API_KEY, OPENAI_API_BASE (optional), GEMINI_API_KEY, and GEMINI_API_BASE (optional). Runtime environment requirements include fixed positions for the IDE and terminal windows, screen scaling set to 100% or a recorded scaling factor, and no mouse movement or window repositioning during runtime.

[0032] Error handling and fault tolerance mechanisms include multi-layered fault tolerance strategies. Regarding timeout handling, a reasonable detection interval (20 seconds) is set in the waiting mode to avoid frequent checks that waste system resources and cause excessive API calls, while ensuring timely detection of task progress. If the waiting time is too long (e.g., exceeding the preset maximum waiting time), the system can log warnings and take appropriate measures. For visual positioning fault tolerance, when the location information returned by the multimodal large language model is inaccurate, the system can re-capture the current screen image and re-position, or improve accuracy by averaging multiple positioning attempts. If positioning fails, the system can try alternative strategies, such as expanding the search range or using different text prompts. Regarding model (multimodal large language model or large language model) call failure handling, when an API call fails (network error, API rate limiting, service unavailability, etc.), the system records detailed error logs, including error type, error message, number of retries, etc., and saves the current screenshot for manual analysis, implementing an exponential backoff retry mechanism to prevent frequent retries from exacerbating problems. If the system fails after multiple retries, it can log the error and continue with subsequent steps, or pause execution and wait for manual intervention. Regarding window layout changes, the system dynamically adapts through visual understanding, automatically recognizing window position changes and repositioning GUI elements without manual intervention. If a window is minimized or obscured, the system can attempt to activate the window or wait for it to become visible again. For data validation, the system validates the execution results returned by the multimodal large language model, such as checking if the position information is within the screen range, whether the extracted text conforms to the expected format, and whether the JSON format is correct. If validation fails, the system logs the error and takes appropriate measures, such as re-extracting or using default values. For exception handling, the system uses try-except blocks to catch various exceptions, including file operation exceptions, network exceptions, and GUI operation exceptions, ensuring that the failure of a single operation does not interrupt the entire process. Detailed exception information is logged for troubleshooting and system improvement.

[0033] Step 103: In executing the target operation, using the timestamp as the target GUI element, and combining it with text prompts, a multimodal large language model is used to identify the coordinate information of the target GUI element from the current screen image of the client interface. Based on the coordinate information, the element region image is periodically captured, and the timestamp information within it is identified to determine whether the target operation is completed. In this embodiment, the current screen image is acquired once every first time interval, and the identification and judgment step is executed. The identification and judgment step includes: converting the current screen image into a region image code; inputting the region image code and the corresponding text prompts into the multimodal large language model to identify the coordinate information of the target GUI element in the current screen image; cropping the element region image containing the timestamp based on the coordinate information; inputting the element region image into a visual language inference model to identify the timestamp information within it, and obtaining the identification result; if the identification result includes timestamp information, the target operation is completed; if the identification result does not include timestamp information, the current screen image is recaptured when the first time interval is reached, and the identification and judgment step is executed again.

[0034] Specifically, the system periodically captures the current screen image, dynamically identifies the timestamp display area using a multimodal large language model, and extracts the timestamp information to determine whether the IDE has completed the current target operation (or task). Timestamp detection is crucial for determining the completion status of an operation (or task), and the system achieves this by: entering a waiting loop, capturing the current screen image every 20 seconds (i.e., the first time), first locating the timestamp display area using the multimodal large language model, analyzing the current screen image, identifying the timestamp's location information with the timestamp as the target GUI element, and then retrieving the image of the element region containing the timestamp from the current screen image, saving the element region image as a PNG file (the filename includes the timestamp, and the format is statics / img_). <timestamp>The system reads the image file containing the element region and converts it to Base64 encoded image data. It then creates a `GenerateContentConfig` object, sets the response format to `application / json`, and sends the Base64 encoded image data along with the timestamp extraction prompt "Extract the text in this image. Return only the text, and do not include any other information." to the Gemini-2.5-flash model. The Gemini-2.5-flash model returns the recognition result. The system checks if the returned result contains "AM" or "PM" time stamps, or non-empty text. If timestamp information is detected in the recognition result, it indicates that the IDE interface has been updated and the task may be complete. The system logs "Cursor has finished the task" and exits the waiting loop; otherwise, it logs "Cursor is still generating..." and continues the waiting loop.

[0035] Step 104: After performing the target operation, take the dialog area of ​​the target operation as the target GUI element, and use a multimodal large language model to identify the coordinate information of the target GUI element from the current screen image of the client interface in combination with the text prompt words. Based on the coordinate information, capture a screenshot of the dialog area and extract the test information from the screenshot.

[0036] In this embodiment, after extracting the test information, it is tested on the terminal. Based on the code summary and test script in the test information, the execution result of the target operation is checked to see if it meets expectations. If the test results show problems, such as some functions failing the test, the causes of the problems can be further analyzed, and corresponding solutions can be taken, such as adjusting operation parameters or modifying the code. If the test results show that all indicators of the target operation meet the requirements, the remaining target operations in the task can be executed, thereby realizing the complete process of GUI automated operation integrating visual language reasoning.

[0037] Specifically, after the target operation is completed, the area where the IDE dialog box is located is located using a multimodal large language model, and then a screenshot of that area is captured. The code summary and test script are then extracted using the multimodal large language model. The code summary extraction process includes: based on the current screen image, locating the position information of the area where the IDE dialog box is located (i.e., the dialog area) using the multimodal large language model; using the pyautogui.screenshot function to capture a screenshot of the dialog area in the current screen image, saving it as a PNG file (the filename includes a timestamp); reading the image file of the area screenshot and converting it to Base64 encoded image data; creating a GenerateContentConfig configuration object, setting the response format to application / json; and sending the Base64 encoded image data and the text prompt "Extract the cursor summary information from the screenshot. Return only the summary information, and do not include any other information." together to the Gemini-2.5-flash model. The Gemini-2.5-flash model returns the text of the code summary and logs "Successfully extracted cursor output". The process of extracting the test script includes: taking another screenshot of the area where the IDE dialog box is located, saving it as a PNG file, reading the image file of the area screenshot and converting it into Base64 encoded image data, sending the Base64 encoded image data and the text prompt "Extract the unit test script from the screenshot. Common unit test scripts are usually by running a file like this: python file_path.py. Return only the script without quotationmarks." to the Gemini-2.5-flash model, which returns the text of the test script, removes leading and trailing whitespace characters using the strip() method, logs "Successfully extracted unit testscript", and returns the extracted test script.

[0038] Step 105: Test the test information on the terminal and determine whether to terminate the operation based on the test results. In this embodiment, testing the test information on the terminal includes: acquiring the current screen image of the terminal, determining the position information of the terminal interface through a multimodal large language model, and recording the current position of the mouse; and / or, clearing the current input in the terminal interface after waiting for a delay time, and waiting for the delay time again; and / or, formatting the test information and inputting it into the terminal interface for testing.

[0039] Specifically, the extracted test script is input into the terminal, the test is executed, and the test results are awaited. The test process includes: capturing the current screen image of the terminal, locating the position coordinates of the terminal window using multimodal large language, calculating the coordinates of the lower right corner of the terminal window based on the returned position coordinates (X coordinate is the X coordinate of the terminal area + the width of the terminal window, Y coordinate is the Y coordinate of the terminal area + the height of the terminal window), moving the mouse to that position and clicking, recording the current mouse position in the log, waiting for 1 second (i.e., the delay time), pressing Ctrl+C to clear the current input (implemented using pyautogui.hotkey("ctrl", "c")), waiting for 1 second, checking whether the test script is enclosed in quotes (starting and ending with double quotes), if so, removing the quotes from the beginning and end of the test script, using the pyautogui.typewrite function to input the test script, setting the input interval to 0.1 seconds, waiting for 3 seconds, pressing Enter to execute the test script, waiting 60 seconds for the test to complete, and recording "Unit testexecution completed" in the log.

[0040] In addition, decisions can be generated using the large language module based on task information, test information, and test results. These decisions include: if the test result indicates that the operation is normal but the task is not completed, then the large language module is used to optimize the text prompts based on the test information and update the target operation according to the task information; the task processing steps are executed based on the optimized text prompts and the updated target operation until the task is completed; if the test result indicates that the operation is abnormal, the termination condition is met, or the task is completed, then the operation is terminated and the current result is output.

[0041] Specifically, during the iterative process of text prompt optimization, based on the test results and optimization suggestions from the previous round, optimized text prompts are generated using a large language model (such as GPT-4o-mini). The text prompt optimization employs a two-stage approach: The first stage generates optimization suggestions. The function `generate_refine_advice` receives the user's input of the original task, the code summary of the current round, and the terminal's test results as parameters. The function `generate_refine_advice_template` generates a prompt template. The template includes the user input, the code summary of the current round, terminal information, and instructions for the prompt model to generate continue / stop decisions and optimization suggestions. The prompt template is: "Generate 'continue / stop' and refine advice based on the user input, cursor main output, and terminal information. User input: {user_input} Cursor main output in this round: {cursor_output} Terminal information in this round: {terminal_info} Refineadvice focuses on: what else I can do to make the project better. It should be a paragraph. Return your response in the following JSON format: {'action': 'continue' or 'stop', 'advice': 'your detailed refine advice'}" The input is given to the large language model, with the response_format parameter set to {"type": "json_object"} to ensure a JSON format is returned. The returned JSON result is parsed, and the action field (continue or stop) and advice field (optimization suggestions) are extracted. The log entry "Generating refine advice..." is then logged.The first stage involves generating the original task and "Generated refine advice," returning the action and advice. The second stage generates optimized text prompts. The function `generate_refine_prompt` takes the original task and optimization advice as parameters and uses the `refine_prompt_template` function to generate a prompt template: "Refine the user input based on the refine advice. User input at the start of this round: {user_input} Refine advice in this round: {refine_advice} Return only the refined user input, and do not include any other information." This template is then input into the large language model to obtain the optimized text prompts. This two-stage approach ensures the quality of the optimization advice and the relevance of the text prompts, allowing the system to gradually improve task performance based on the execution results.

[0042] Specifically, after optimizing the text prompt and updating the target operation, steps 101 to 105 are executed again to perform a multi-round self-iterative execution process, where the termination condition is reaching the maximum number of iteration rounds (exemplarily set to 5 rounds). The workflow main function receives user input as a parameter, initializes the round counter round_count to 1, calls the generate_init_prompt function to generate the initial text prompt, calls the init_request function to send the initial request, and then enters a while True loop. Inside the loop, it first checks if `round_count` exceeds the maximum number of iterations. If it does, it exits the loop and outputs the current result. Otherwise, it records the current iteration log, increments `round_count` by 1, and calls the `enter_wait_mode` function to enter wait mode, waiting for the Cursor to complete the task. After the task is completed, it calls the `extract_cursor_output` function to extract the code summary, the `extract_unit_test_script` function to extract the test script, the `conduct_unit_test` function to execute the unit test, and the `extract_terminal_info` function to extract the terminal output information. Then, it calls the `generate_refine_advice` function to make a decision. This function receives the original task, code summary, and terminal information from the user as input, uses the `generate_refine_advice_template` template to generate text prompts for analysis, sets the response format to a JSON object, parses the returned JSON result, and extracts the `action` field (continue or stop) and the `advice` field (optimization suggestions). If the action is "stop", the loop exits, the operation terminates, the log is logged, and the process returns. If the action is "continue", the `generate_refine_prompt` function is called to generate an optimized text prompt, the `request_in_the_loop` function is called to send an optimization request, and then the next iteration continues based on the optimized text prompt and the updated target operation. Simultaneously, the current execution status and iteration information of the target operation can be tracked, and exception handling, timeouts, errors, and other abnormal situations can be monitored to ensure system stability and reliability.

[0043] Furthermore, if the action is "continue", the optimized text prompt is generated based on the optimization suggestions, and the process returns to step 101 for the next round of iterative optimization. If the action is "stop", the operation is terminated and the process ends. Iterative optimization includes the following detailed steps: generating optimized text prompts, calling the `generate_refine_prompt` function, inputting the original task and optimization suggestions from the user, generating text prompts using the `refine_prompt_template` template, generating optimized text prompts using the GPT-4o-mini model, and returning the optimized text prompts; copying terminal information, calling the `copy_terminal_info` function, capturing the current screen image, locating the terminal window's position coordinates using multimodal large language positioning, calculating the position coordinates of the lower right corner of the terminal window based on these coordinates, moving the mouse to that position and clicking, recording the mouse's current position in the log, waiting 3 seconds, and calculating the position coordinates of the upper left corner of the terminal window. The system uses coordinates (X coordinate is the X coordinate of the terminal window, Y coordinate is the y coordinate of the terminal window) to drag the mouse from the bottom right corner to the top left corner using the pyautogui.dragTo function, selects the entire terminal content, records the mouse position after dragging, waits 3 seconds, calculates the right-click position (X coordinate is the X coordinate of the terminal window + 200, Y coordinate is the Y coordinate of the terminal window + 200), uses the pyautogui.doubleClick function to double-click at this position, opens the context menu, waits 3 seconds, calculates the position of the "Copy" option (X coordinate is the X coordinate of the right-click position + 100, Y coordinate is the Y coordinate of the right-click position + 165), clicks the "Copy" option, waits 3 seconds, and logs "Successfully copied terminal info to clipboard".Send an optimization request, capture the current screen image, locate the position coordinates of the IDE dialog box using a multimodal large language model, move the mouse to the IDE dialog box based on the position coordinates and click, record the current mouse position, wait 3 seconds, enter the test result prefix (e.g., "This is partresult of the test scripts in the terminal:"), use the pyautogui.typewrite function to input, set the interval to 0.1 seconds, wait 1 second, press Ctrl+V to paste the terminal information (implemented using pyautogui.hotkey("ctrl", "v")), wait 3 seconds, enter the optimized text prompt, use the pyautogui.typewrite function to input, set the interval to 0.1 seconds, wait 3 seconds, press Enter to send the request, wait 3 seconds, log "Request in the loop completed". Repeat steps 101 to 105 until the maximum number of iterations (5 rounds) is reached or the action is stop. Track the current iteration round number using the round_count variable, and log "Round {round_count}starts" at the beginning of each round.

[0044] The log records include operation steps, current screen images, extraction results, and decision-making basis, for later reproduction and auditing. The log records all target operation steps, including mouse position, click coordinates, and input content; saves all current screen images (and screenshots based on the current screen images) for subsequent analysis; information extracted from the current screen images; and the basis and results of decision-making. The log is named with a timestamp and saved as a text file for easy analysis and reproduction.

[0045] This application breaks through API limitations, enabling the manipulation of any human-operable GUI element through native GUI operation, free from the constraints of tool API interfaces and significantly expanding the scope of automation applications. It achieves human-machine parity, allowing the system to perform clicks, inputs, drags, and other operations like a human operator, realizing true GUI automation. It fully leverages the visual understanding capabilities of a multimodal large language model, extracting structured information from the current screen image, achieving an organic combination of visual perception and language understanding. Adaptive iterative optimization, through a multi-round self-iterative mechanism, allows the system to automatically adjust strategies based on execution results, optimize text prompts, and gradually improve task execution performance. It is configurable and scalable, supporting multi-monitor layouts and adaptation to different resolutions, exhibiting excellent environmental adaptability. Full traceability, through structured log recording, completely records the execution process, enabling reproducibility and auditing, thus improving the system's reliability and trustworthiness.

[0046] While this application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the methods shown in this embodiment or the accompanying drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0047] like Figure 2 As shown in the figure, this application embodiment also provides a GUI automated manipulation system 200 that integrates visual language reasoning. The system includes: a task module 201, a recognition module 202, and a testing module 206, wherein the recognition module 202 includes a first execution module 203, a second execution module 204, and a third execution module 205, as detailed below.

[0048] Task module 201 is used to generate text prompts based on user input and system configuration information, and to execute task processing steps based on these prompts. The task processing steps include: The recognition module 202 is used to determine the target GUI element according to the stage of the target operation. Based on the target GUI element and the text prompt, it uses a multimodal large language model to recognize the coordinate information of the target GUI element from the current screen image of the client interface to complete the target operation, including: The first execution module 203 is used to identify the coordinate information of the target GUI element from the current screen image of the client interface before executing the target operation, taking the operation object as the target GUI element and combining the text prompt words with a multimodal large language model, and then execute the target operation based on the coordinate information.

[0049] The second execution module 204 is used to identify the coordinate information of the target GUI element from the current screen image of the client interface by using a multimodal large language model, taking the timestamp as the target GUI element and combining it with text prompts, during the execution of the target operation. Based on the coordinate information, it periodically captures the image of the element area and identifies the timestamp information therein to determine whether the target operation has been completed.

[0050] The third execution module 205 is used to, after executing the target operation, take the dialog area of ​​the target operation as the target GUI element, combine the text prompt words and use a multimodal large language model to identify the coordinate information of the target GUI element from the current screen image of the client interface, capture a screenshot of the dialog area based on the coordinate information, and extract test information from the screenshot.

[0051] The test module 206 is used to test the test information on the terminal and determine whether to terminate the operation based on the test results.

[0052] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0053] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0054] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0055] This application also provides an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method described in this application.

[0056] This application also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon, which, when executed, enables the method described in this application embodiment to be implemented.

[0057] Furthermore, in the various embodiments of the present invention, each functional module can be integrated into a processing module, or each module can exist independently, or two or more modules can be integrated into a single module.

[0058] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0059] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0060] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0061] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.< / timestamp> < / timestamp>

Claims

1. A GUI automated manipulation method integrating visual language reasoning, characterized in that, include: Text prompts are generated based on user input and system configuration information, and task processing steps are executed accordingly; wherein, the task processing steps include: The target GUI element is determined based on the stage of the target operation. Based on the target GUI element and the text prompt, a multimodal large language model is used to identify the coordinate information of the target GUI element from the current screen image of the client interface to complete the target operation, including: Before performing the target operation, the target GUI element is selected as the operation object. The coordinate information of the target GUI element is identified from the current screen image of the client interface using a multimodal large language model in combination with the text prompt words. The target operation is then performed based on the coordinate information. In performing the target operation, the timestamp is used as the target GUI element. Combined with the text prompt words, the coordinate information of the target GUI element is identified from the current screen image of the client interface using a multimodal large language model. Based on the coordinate information, the element area image is periodically captured and the timestamp information in it is identified to determine whether the target operation has been completed. After the target operation is performed, the dialog area of ​​the target operation is taken as the target GUI element. The coordinate information of the target GUI element is identified from the current screen image of the client interface by combining the text prompt words with the multimodal large language model. Based on the coordinate information, a screenshot of the dialog area is captured, and the test information in the screenshot is extracted. The terminal tests the test information and determines whether to terminate the operation based on the test results.

2. The method according to claim 1, characterized in that, Before the task processing steps are executed, the following steps are also included: The client for the multimodal large language model is initialized based on the system's API configuration, and the validity of the key is verified; and / or, Set the scaling ratio of the client interface and fix the interface position; and / or, Set a delay time to ensure stable window layout on the client and terminal.

3. The method according to claim 2, characterized in that, The step of generating text prompts based on user input and system configuration information includes: Text prompts are generated based on user input and system configuration information, and a summary and test commands are added to the end of the text prompts so that the multimodal large language model can generate a progress summary and executable test information after each round of tasks.

4. The method according to claim 1, characterized in that, The coordinate information of the target GUI element includes the position coordinates and size of the target GUI element.

5. The method according to claim 1, characterized in that, The method of periodically capturing an image of a GUI element's region based on its coordinate information, using a timestamp as the target GUI element, and identifying the timestamp information within it to determine whether the target operation has been completed includes: The current screen image is acquired once every first time interval, and the recognition and judgment steps are performed. The identification and judgment step includes: The current screen image is converted into a region image code, and the region image code and the corresponding text prompt are input into a multimodal large language model to identify the coordinate information of the target GUI element in the current screen image; Based on the coordinate information, extract the image of the element region containing the timestamp from the current screen image; The element region image is input into the visual language inference model to identify the timestamp information and obtain the recognition result; If the recognition result includes timestamp information, then the target operation is complete; If the recognition result does not include timestamp information, then wait until the first time is reached to recapture the current screen image and execute the recognition judgment step.

6. The method according to claim 1, characterized in that, The step of testing the test information on the terminal includes: Acquire the current screen image of the terminal, determine the position information of the terminal interface through a multimodal large language model, and record the current position of the mouse; and / or, After the delay period, clear the current input on the terminal interface, and wait for the delay period again; and / or, The test information is formatted and then entered into the terminal interface for testing.

7. The method according to claim 3, characterized in that, The step of testing the test information on the terminal and determining whether to terminate the operation based on the test results also includes: Based on task information, test information, and the test results, a decision is generated using the large language module, including: If the test result indicates that the operation is normal but the task is not completed, then the text prompt word is optimized using the large language module based on the test information, and the target operation is updated according to the task information; the task processing steps are executed based on the optimized text prompt word and the updated target operation until the task is completed. If the test result indicates an operational error, the termination condition has been met, or the task has been completed, the operation will terminate and the current result will be output.

8. A GUI automated manipulation system for implementing the method according to any one of claims 1-7, characterized in that, include: The task module is used to generate text prompts based on user input and system configuration information, and to execute task processing steps based on these prompts; wherein, the task processing steps include: The recognition module is used to determine the target GUI element according to the stage of the target operation, and based on the target GUI element and the text prompt, to recognize the coordinate information of the target GUI element from the current screen image of the client interface using a multimodal large language model, so as to complete the target operation, including: The first execution module is used to identify the coordinate information of the target GUI element from the current screen image of the client interface using a multimodal large language model, taking the operation object as the target GUI element, in conjunction with the text prompt words, and execute the target operation based on the coordinate information before executing the target operation. The second execution module is used to identify the coordinate information of the target GUI element from the current screen image of the client interface by using the timestamp as the target GUI element and combining the text prompt words with the multimodal large language model during the execution of the target operation. Based on the coordinate information, the module periodically captures the element area image and identifies the timestamp information therein to determine whether the target operation has been completed. The third execution module is used to, after executing the target operation, take the dialog area of ​​the target operation as the target GUI element, combine the text prompt words and use a multimodal large language model to identify the coordinate information of the target GUI element from the current screen image of the client interface, capture a screenshot of the dialog area based on the coordinate information, and extract test information from the screenshot. The testing module is used to test the test information on the terminal and determine whether to terminate the operation based on the test results.

9. An apparatus for performing a GUI automated manipulation method that integrates visual-language reasoning, characterized in that, include: processor; Memory used to store processor-executable instructions; When the processor executes the executable instructions, it implements the method as described in any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that, Includes storage of computer programs or instructions that, when executed, cause the method as described in any one of claims 1 to 7 to be implemented.

Images