A graphical user interface automation method, apparatus and device

Abstract

This application provides a graphical user interface automation method, apparatus, and device. In this application embodiment, a current screenshot of a target application interface is acquired; the current screenshot and user instructions are input into a visual big data model; based on a visual understanding of the current screenshot and the semantics of the user instructions, the visual big data model plans a structured operation sequence required to complete the user instructions. The structured operation sequence includes at least one atomic operation and the normalized coordinates of the interactive elements associated with each atomic operation in the screenshot; the structured operation sequence is parsed, and the normalized coordinates of the interactive elements associated with each atomic operation are mapped to the absolute coordinates of the current physical screen; according to the type of each atomic operation, the operating system's input device is driven to perform an operation at the corresponding absolute coordinates.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a graphical user interface automation method, apparatus, and device. Background Technology

[0002] As software applications become increasingly complex and diverse, automating the operation of graphical user interfaces (GUIs) has become a key technological requirement for improving work efficiency, ensuring software quality, and building intelligent workflows. This technology is widely used in software automated testing, robotic process automation (RPA), and assisted operation scenarios, with its core objective being to replace or assist manual processes in recognizing and interacting with various software interface elements.

[0003] Currently, mainstream GUI automation technologies mainly follow two paths: one is automation based on the program's internal structure, which obtains the logical tree structure of interface controls (such as the DOM tree or UI Automation tree) by accessing the application programming interfaces (APIs) provided by the operating system or browser, and performs positioning and manipulation based on the unique attributes of elements; representative tools include Selenium and various RPA components. The other is automation based on external visual features, which uses image matching technology to find pre-stored visual templates in screenshots and performs interactions based on matching coordinates; representative tools include SikuliX. Both methods rely on predefined, static positioning rules or visual features.

[0004] However, the effectiveness of related automation technologies heavily relies on access to the application's internal structure (such as control trees and document object models) or precise matching with static visual features of the user interface (such as image templates and fixed coordinates). This makes it difficult for existing technologies to adapt to application interfaces that lack a standard internal structure or whose visual presentation changes dynamically, and their automation processes lack the ability to naturally simulate the interactions of actual users based on visual cognition and semantic understanding. Summary of the Invention

[0005] This application provides a graphical user interface automation method, apparatus, and device to address the problem in related technologies where the automation process of graphical user interfaces lacks the ability to naturally simulate the interaction of actual users based on visual cognition and semantic understanding.

[0006] This application provides a graphical user interface automation method, including: The acquisition module is used to receive prediction request messages from the request node; The planning module is used to input the current screenshot and user command into the visual big model. Based on the visual understanding of the current screenshot and the semantics of the user command, the visual big model plans a structured operation sequence required to complete the user command. The structured operation sequence includes at least one atomic operation and the normalized coordinates of the interactive element associated with each atomic operation in the screenshot. The operation module is used to parse the structured operation sequence, map the normalized coordinates of the interactive elements associated with each atomic operation to the absolute coordinates of the current physical screen, and drive the operating system's input device to perform the operation at the corresponding absolute coordinates according to the type of each atomic operation.

[0007] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. The processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform steps in the graphical user interface automation method provided in this application.

[0008] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps in the graphical user interface automation method provided in this application.

[0009] This application also provides a computer program product that stores instructions that, when executed by a computer, cause the computer to perform the steps in the graphical user interface automation method provided in this application.

[0010] The graphical user interface automation method provided in this application's embodiments inputs screenshots and user commands into a large visual model. The model then directly outputs a structured operation sequence containing normalized coordinates based on multimodal understanding. This process simulates the human cognitive interaction logic of "viewing the screen - understanding - planning," thereby achieving truly universal automation that does not rely on control trees, document object models, or pre-stored image templates. Furthermore, by dynamically mapping the normalized coordinates output by the model to the absolute coordinates of the current physical screen and driving the operating system input device to execute, the system achieves decoupling from specific screen resolutions, interface layouts, and visual themes, significantly improving cross-platform and cross-device adaptability and robustness. Finally, through the above-mentioned full visual path automation process, this method completely eliminates dependence on application implementation details while providing a graphical user interface automation solution that can naturally respond to user commands, has dynamic interface adaptability, and low maintenance costs. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a graphical user interface automation method provided for an exemplary embodiment of this application; Figure 2 A schematic diagram of the core process of the graphical user interface automation method provided in the exemplary embodiments of this application; Figure 3 A timing flowchart illustrating a graphical user interface automation method provided for an exemplary embodiment of this application; Figure 4 A schematic diagram of the structure of a graphical user interface automation device provided for an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The following is a description of the terms used in this application: A graphical user interface (GUI) is a visual interface through which a user interacts with a computer program. It typically consists of visual elements such as windows, buttons, text boxes, and icons.

[0014] Visual large models (VLMs) refer to large-scale artificial intelligence models that can simultaneously process visual (image) and linguistic (text) inputs and perform multimodal understanding and reasoning.

[0015] Multimodal understanding refers to the ability of a model to simultaneously process and correlate information from different modalities (such as images, text, and speech).

[0016] Normalized coordinates, a representation method that maps absolute pixel coordinates in a screen or image to a standard range (such as [0,1000] or [0, 1]), are used in this application to achieve resolution-independent positioning.

[0017] A bounding box is a rectangular box used to identify the location of a target in an image. It is usually represented by the coordinates of the top left and bottom right corners [x1, y1, x2, y2].

[0018] Structured output: The model outputs data organized according to a predefined format (such as JSON, XML), which facilitates subsequent program parsing and processing.

[0019] Atomic operations are indivisible basic interactive actions, such as clicking, entering text, scrolling, and waiting.

[0020] Screenshot: Captures a static image of the content displayed on the current monitor or in a specified window.

[0021] The client area is the actual content display area in the application window, excluding system decorative areas such as the title bar, borders, and menu bar.

[0022] A window handle is a unique identifier assigned to each window by the operating system, used by programs to identify and control windows.

[0023] Chain of Thought (CoT) is a prompting method that guides models to demonstrate their step-by-step reasoning process and is often used to improve the accuracy of complex tasks.

[0024] Prompts are text instructions given to large language or visual models to guide them in generating the desired output.

[0025] Hallucination is the phenomenon where a model generates output that does not match the input or is actually incorrect.

[0026] Natural language instructions are task descriptions that users express in everyday language, such as "Log in to the system and check the order".

[0027] The Document Object Model (DOM) is a programming interface through which browsers represent web pages as a tree structure, allowing programs to dynamically access and manipulate page content.

[0028] An execution agent is a program module responsible for receiving instructions and driving external devices (such as mice and keyboards) to perform operations.

[0029] Closed-loop feedback is a control mechanism in which the system obtains changes in the environmental state after performing an operation and adjusts its subsequent behavior accordingly.

[0030] As described in the background section, the automation capabilities of related technologies heavily rely on the implementation details of applications or the static visual features of interfaces, lacking the ability to simulate the cognitive and interactive process of actual users "viewing the screen - understanding - operating". Specifically, this manifests as: (1) inability to handle interfaces without standard control structures or with frequently changing visual features; (2) inability to understand the semantic functions of interface elements, making it difficult to respond to high-level natural language task instructions; and (3) system fragility, where any minor change to the interface may lead to automation failure.

[0031] To address the aforementioned problems in related technologies, this application provides a graphical user interface automation method, apparatus, and device, effectively solving the problem of strong dependence on the internal structure of applications or static visual features in existing technologies. Specifically, the system inputs screenshots and user commands into a large visual model, which directly outputs a structured operation sequence containing normalized coordinates based on multimodal understanding. This process simulates the human cognitive interaction logic of "viewing the screen - understanding - planning," thereby achieving truly universal automation that does not rely on control trees, document object models, or pre-stored image templates. Furthermore, by dynamically mapping the normalized coordinates output by the model to the absolute coordinates of the current physical screen and driving the operating system input device to execute, the system achieves decoupling from specific screen resolutions, interface layouts, and visual themes, significantly improving cross-platform and cross-device adaptability and robustness. Finally, through the closed-loop automation process of the entire visual path described above, this method completely eliminates dependence on application implementation details while providing a graphical user interface automation solution that can naturally respond to user semantic commands, has dynamic interface adaptability, and low maintenance costs.

[0032] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0033] Figure 1 This is a flowchart illustrating a graphical user interface automation method provided for an exemplary embodiment of this application. Figure 1 As shown, the method includes: Step 110: Obtain a screenshot of the current screen of the target application interface.

[0034] In some exemplary embodiments, obtaining a current screenshot of the target application interface includes: Locate the target application window using semantic fuzzy matching; Call the operating system interface to obtain the handle, position, and size information of the target application window; Based on the window's position and size information, dynamically determine the coordinate range of its client area; Capture the image within the coordinate range of the client area as the current screen screenshot.

[0035] Specifically, firstly, semantic fuzzy matching is used to locate the target application window. This is achieved by executing a proxy to call the operating system interface and performing semantic fuzzy matching on the window title based on pre-configured or dynamically input keywords, thereby accurately locating the target application window. Because this method no longer relies on volatile fixed window handles or screen coordinates, it ensures that even if the target application window's position changes or multiple similar windows exist, the target application window can still be accurately located, thus laying a reliable foundation for subsequent precise interaction.

[0036] Secondly, the system calls the operating system interface to obtain the handle, position, and size information of the target application window. Based on the window's position and size information, it dynamically determines the coordinate range of its client area. Specifically, it calls the operating system interface to obtain the precise geometric attributes of the target application window, including its handle (representing its structural information), position, and size information. Based on the window's position and size information, it dynamically determines the coordinate range of its client area, effectively eliminating visual noise from non-interactive elements such as the title bar and borders, and dynamically distinguishing between the window's "non-client area" (such as the title bar, borders, and system menus) and "client area." Through real-time calculation, it only captures the client area image containing actual interactive elements (such as buttons, input boxes, and list views), automatically eliminating visual interference from non-functional interfaces.

[0037] Finally, capturing the image within the client area coordinates as the current screenshot ensures the quality of the data input to the large-scale visual model. Furthermore, state normalization preprocessing can be performed before the final capture, such as automatically bringing the target window to the foreground and potentially standardizing its resolution or scaling. Subsequently, image capture is performed within the dynamically calculated client area coordinates to generate a clean current screenshot. This provides a highly relevant and accurate visual snapshot of the interactive interface to the upper-level large-scale visual model without relying on the application's internal control tree or accessibility interfaces, significantly improving the accuracy and robustness of subsequent element recognition and localization, and fundamentally overcoming the shortcomings of traditional image template methods that fail due to changes in interface layout or display settings.

[0038] For example, obtaining a current screenshot of the target application's interface may include: Step 1: Smart Window Locking and Capture: First, the operating system's underlying API is called through the Executor Agent to fuzzily match the target window based on the configured keywords.

[0039] Step 2: Dynamic Region Determination: Based on this, dynamic region determination is further performed to intelligently distinguish between the "non-client area" (title bar, border) and the "client area" of the window, and prioritizes capturing the client area image containing the actual interactive content, thereby improving the accuracy of operation.

[0040] Step 3: State normalization processing: In order to ensure the consistency of the input model data, state normalization processing will be performed before taking the screenshot, including automatically bringing the target window to the front and adjusting it to the standard resolution.

[0041] Step 120: Input the current screenshot and user command into the visual big model. Based on the visual understanding of the current screenshot and the semantics of the user command, the visual big model plans the structured operation sequence required to complete the user command.

[0042] The structured operation sequence includes at least one atomic operation and the normalized coordinates of the interactive element associated with each atomic operation in the screenshot.

[0043] For example, the acquired current screenshot and user commands can be input into a Visual Model (VLM). VLM first performs a comprehensive thought chain analysis: based on multimodal visual understanding of the current screenshot, it combines the semantic recognition of the user commands to identify the corresponding interface type, core functions, and interactive elements related to the commands, and understands their functional semantics (e.g., recognizing the "login button" and understanding that clicking it submits credentials). Subsequently, VLM performs coordinate localization and task decomposition: it calculates the normalized bounding box coordinates of each identified key interactive element in the current screenshot, while parsing the user commands into a series of predefined atomic operations, and associating each operation with the normalized coordinates and functional semantics of the corresponding interactive element. Finally, VLM outputs a structured JSON sequence (i.e., a structured operation sequence), explicitly including the action type, associated normalized coordinates, and necessary parameters. This completes the planning from "what to see" and "what to do" to "how to do it specifically (i.e., where to click, what to input)," achieving automated decision-making based entirely on visual semantic understanding of the screenshot content and semantic understanding of the user commands, independent of the program's internal structure.

[0044] One possible example of mind chain analysis is as follows: Taking "Add product to cart" as an example, the intent is understood to be that the current page is the search results page of an e-commerce platform, and the user's goal should be to filter and locate specific products. Operation status: The page has loaded stably, there are multiple product items in the list, and there are currently no pop-ups or error messages. Target location: The key interactive elements are the 'Brand Filter' sidebar on the left, the 'Sort' dropdown menu in the upper right corner, and the 'Add to Cart' button for the first product item.

[0045] For example, the structured operation sequence is JSON formatted data. Predefined atomic operations include at least: click, input text, scroll, and wait.

[0046] In some exemplary embodiments, the visual big data model plans a structured sequence of operations required to complete the user command based on its visual understanding of the current screenshot and the semantics of the user command, including: The visual big data model identifies at least one interactive element in the current screenshot that is related to completing the user's command and understands its functional semantics. Determine the position of each identified interactive element in the current screenshot and generate the corresponding normalized coordinates; Based on the semantics of user instructions and the functional semantics of interactive elements, the task is decomposed into at least one predefined type of atomic operation, and each atomic operation is associated with the corresponding interactive element and its normalized coordinates.

[0047] Specifically, the visual big model is first instructed to analyze the current interface state, including: a. Interface structure and intent understanding: The visual big model first analyzes the current screenshot as a whole, identifying the corresponding interface type and core functions. b. Key interaction element localization: After clarifying the interface scenario, the visual big model needs to further identify and locate all UI components, i.e., interaction elements, related to user commands. This includes not only visually circling key elements such as "username input box," "password input box," and "login button," but also understanding their functional attributes to provide precise coordinate targets for generating specific operation commands in the next stage. c. Operation state and transition judgment: The visual big model needs to dynamically evaluate the results caused by the previous operation to confirm whether the task is progressing as expected.

[0048] Next, after understanding the screen image, the model first performs a visual localization subtask. The key to this step is generating normalized bounding boxes (in the format [x1, y1, x2, y2]) of the target interactive elements in the image, represented using a normalized coordinate system of 0–1000. By converting absolute pixel coordinates to normalized values, the model decouples itself from the specific screen resolution. Based on accurate target localization, the task execution module parses user commands into a series of standardized atomic operations, primarily including click, type, scroll, and wait. This decomposition mechanism constrains the model output through a predefined action vocabulary, ensuring that each operation is semantically clear and highly executable. Finally, it generates structured JSON instructions, specifically a JSON-formatted action list that follows a template structure, returned by the LLM.

[0049] Clearly, by employing the above method, the task execution module, based on accurate target location, parses user instructions into a series of standardized atomic operations, primarily including click, type, scroll, and wait. This decomposition mechanism constrains the model output through a predefined action vocabulary, ensuring that each operation is semantically clear and highly executable. c. Structured JSON Instruction Generation: LLM returns a JSON-formatted action list that follows a template structure.

[0050] For example, the large visual model is a visual language model.

[0051] As an example, a multimodal prompt word construction mechanism organically integrates visual perception with task instructions: First, the acquired screenshots are encoded into Base64 format in real time to form visual input, providing complete visual information of the interface corresponding to the current screenshot; second, the user's natural language instructions (i.e., user instructions) constitute the task instruction input in Base64 format; finally, the maintained operation history queue forms contextual input, using previous operation results as contextual input to ensure continuous decision-making capabilities.

[0052] Step 130: Parse the structured operation sequence, map the normalized coordinates of the interactive elements associated with each atomic operation to the absolute coordinates of the current physical screen, and drive the operating system's input device to perform the operation at the corresponding absolute coordinates according to the type of each atomic operation.

[0053] Specifically, after the visual large model outputs the original instructions, they are not executed directly. Instead, the output content undergoes rigorous post-processing. Specific technical means include: format cleaning and parsing, i.e., removing irrelevant tags (such as Markdown code block symbols) that may be mixed in the model response and extracting valid JSON content. The parser has fault tolerance capabilities, such as attempting automatic completion when encountering incomplete structures; and security and logic verification, i.e., checking whether the normalized coordinate values ​​exceed a preset range and making reasonable corrections, while verifying the operation type according to a preset action whitelist to filter out invalid or hallucinatory instructions that the model may generate. Specifically, before the parsing step, the method provided in this application embodiment further includes: The structured operation sequence output by the large visual model is cleaned and parsed, and the normalized coordinates are corrected for range and validated logically. The operation types in the structured operation sequence are verified based on a pre-defined whitelist of atomic operation types.

[0054] First, format cleaning and fault-tolerant parsing significantly improve overall robustness, effectively handling common non-standard and incomplete issues in large visual model output, ensuring that automated processes are not interrupted by non-fatal format errors. Second, coordinate range correction and logic verification ensure the safety and accuracy of action execution, preventing mouse clicks on non-target areas or even off-screen due to model coordinate output errors (such as numerical overflow or logical inversion), thus avoiding uncontrollable operational risks. Finally, whitelisting of operation types constrains behavioral boundaries, ensuring that only predefined trusted atomic operations are executed, fundamentally eliminating the possibility of harmful or ambiguous instructions generated due to the "illusion" of large visual models.

[0055] In some exemplary embodiments, the operating system's input device is driven to perform an operation at the corresponding absolute coordinates based on the type of each atomic operation, including: When the atomic operation type is click, the final execution coordinates are generated based on a random algorithm within the screen area determined by the mapped absolute coordinates. Control the cursor to move along a trajectory that includes random delay and jitter until the final execution coordinates are reached, then execute the click event.

[0056] The final execution coordinates are generated using a random algorithm, which can employ a Gaussian distribution probability model to sample within the screen area, ensuring that the sampled points have a high probability of falling in the center of the area. Specifically, if a sampled point falls outside the screen area, it is resampled until it falls within the area.

[0057] For example, firstly, within the physical screen area determined by the mapped absolute coordinates, random coordinate sampling can be performed based on a Gaussian distribution probability model. Specifically, within the mapped physical bounding box, a random click coordinate can be generated based on a Gaussian distribution algorithm. This random coordinate has the characteristic that the sampled point has a high probability of falling in the center of the area and a low probability of falling on the edge, thus simulating the natural dispersion of human operations. If the sampled point accidentally falls outside the area, resampling is performed to ensure validity. Secondly, the cursor can be controlled to move to this coordinate with a trajectory including random delay and jitter; for example, a small random delay and physical jitter can be injected into the mouse movement trajectory. Finally, a click event is executed at the coordinate.

[0058] Clearly, by replacing fixed-center clicks with Gaussian random sampling, and combining random delays with trajectory jitter, the spatiotemporal characteristics of automated operations can be made highly approximate to those of real human users. Furthermore, randomized clicks reduce the risk of click failures caused by minor shifts or dynamic changes in interface elements, while biomimetic trajectories make the automation process smoother and more natural, avoiding the user experience discomfort that may arise from mechanical operations.

[0059] In some exemplary embodiments, the method provided in this application can, after a structured operation sequence is completed, re-acquire the current screenshot of the target application interface and input this new screenshot, along with incomplete user instructions or historical context, into the Visual Large Model (VLM) to plan subsequent operation sequences. This process continues until the user instruction is determined to be completed, thereby achieving continuous and adaptive task progression. Specifically, the method provided in this application can be executed in a closed-loop manner: After the input device performs a structured operation sequence, a new screenshot of the target application interface is obtained. The updated current screenshot, along with incomplete user commands or historical context, is input into the visual big model to plan the sequence of subsequent operations until the user command is determined to be completed. If the previous operation did not achieve the expected state based on the updated current screenshot, the visual big model will autonomously decide on the subsequent action according to the injected exception handling strategy prompts. The subsequent action includes at least one of the following: retrying the same interactive element, finding and operating on a substitute interactive element, or performing a swipe operation to refresh the interface.

[0060] If, based on the updated screenshot, the previous operation is determined not to have achieved the expected state (e.g., the interface did not redirect after clicking), the visual big data model autonomously decides on subsequent actions according to the injected exception handling strategy. Specifically, the visual big data model may make decisions based on prompts (e.g., "If there has been no change in the history for a long time, look for obstructing elements or try to swipe the page to update the page"). For example, if the page freezes after clicking the "Login" button, the visual big data model may retry the same interactive element (login button); if the failure is caused by a pop-up window obstructing the view, the model may identify and click the "Close" button on the pop-up window as a replacement interactive element to find and operate; if the page is not fully loaded, the model may perform a swipe operation to refresh the interface content. The entire process does not require real-time human intervention and only terminates when the model's consecutive decision failures reach a preset step limit, thus ensuring the high robustness and continuity of the automated process in the face of dynamic interfaces and unexpected states.

[0061] In some exemplary embodiments, to address the positioning drift problem caused by the inconsistency between logical pixels and physical pixels due to display scaling (DPI scaling), and to achieve accurate clicks across devices, the normalized coordinates of the interactive elements associated with each atomic operation are mapped to the absolute coordinates of the current physical screen, including: Convert the normalized coordinates to the logical pixel coordinates corresponding to the current screenshot; Call the operating system interface to obtain the scaling ratio of the current display device, and convert the logical pixel coordinates into physical pixel coordinates based on the scaling ratio; The final absolute coordinates are calculated by combining the target application window's position offset on the screen.

[0062] For example, firstly, the normalized coordinates (e.g., values ​​ranging from 0 to 1000) output by the Visual Large Model (VLM) are linearly converted back to the logical pixel coordinates of the screenshot itself, based on the screenshot size. However, in most modern operating systems, the user interface is often DPI scaled (e.g., scaled to 150%), where logical pixels do not correspond one-to-one with the actual physical pixels on the screen. Therefore, it is necessary to call the operating system interface to obtain the scaling ratio of the current display device (e.g., 150% corresponds to a scaling factor of 1.5). Subsequently, based on this scaling ratio, the logical pixel coordinates are converted to physical pixel coordinates; for example, logical coordinates (100, 100) are converted to physical coordinates (150, 150) at a scaling factor of 1.5. Finally, since the screenshot may only target the client area of ​​a window, it is also necessary to combine the target application window's position offset on the screen (i.e., the physical pixel distance between the top-left corner of the window and the top-left corner of the screen), add the corrected client area physical coordinates to the window offset, and thus calculate the final absolute coordinates. For example, if the corrected coordinates are (150, 150) and the window offset is (200, 200), then the final absolute coordinates sent to the mouse control API will be (350, 350). Through this series of mappings and corrections, it is ensured that regardless of changes in screen resolution, scaling, or window position, the operations planned by the model can accurately fall on the correct physical position of the target interactive element.

[0063] To decouple the coordinate system from the specific physical parameters of the screen, the normalized coordinates adopt a unified scaling system independent of the current screen capture resolution. Specifically, this means that when the Visual Large Model (VLM) processes the current screen capture and locates interactive elements, it does not output absolute coordinates based on the actual pixels of the capture (e.g., (500, 300) on a 1920x1080 screen), but rather outputs a normalized bounding box coordinate based on a fixed range (0–1000) (e.g., [x1, y1, x2, y2] are all values ​​within this range). This scaling system is unified; that is, regardless of whether the original resolution of the input screenshot is 1920x1080, 2560x1440, or any other size, the Visual Large Model uses the same set of numerical values ​​to describe the target location. For example, a button in the center of the screen might have normalized coordinates of [450, 520, 550, 560] in a low-resolution screenshot, while in a high-resolution screenshot, since the button's relative position in the image remains almost unchanged, its normalized coordinates will still be close to this range. Therefore, the model only needs to learn and output positional information based on this normalized scale, while the execution agent is responsible for dynamically and accurately mapping it back to any real-time physical screen, thus ensuring true cross-resolution and cross-device adaptability of the automation solution.

[0064] Figure 2 This is a schematic diagram of the core flow of the graphical user interface automation method provided for an exemplary embodiment of this application. The process begins with acquiring a screenshot of the current screen (corresponding to the interface perception step). It then proceeds to stage one: visual perception, where the system first locks onto the target application window. Next, it enters stage two: VLM inference. In this stage, the visual big model performs multimodal analysis on the input screenshot and task context. Its internal processes are as follows: generating thought chain analysis to identify interactive elements and understand their functional semantics; generating specific atomic operations and associated normalized coordinates based on this; and determining the specific type of action. After inference, the process makes a decision: if the task is determined to be completed, the process terminates; if not, it proceeds to stage three: biomimetic execution. In this stage, the system first performs denormalization mapping on the normalized coordinates output by the VLM, converting them into absolute coordinates of the current physical screen; for click-type operations, the system randomly generates the final execution coordinates within the coordinate area based on a Gaussian distribution model to simulate human operation; finally, the system calls the operating system (OS) API to drive the input device to perform the corresponding operation. Once completed, the process clearly indicates the screenshot step that returns to the beginning via an arrow, thus intuitively demonstrating the closed-loop feedback mechanism and preparing for the start of the next cycle.

[0065] Figure 3 A timing flowchart illustrating a graphical user interface automation method provided for an exemplary embodiment of this application. Figure 3 This presentation uses a time-series interactive format to illustrate the collaborative relationships and data flow between the user, the local execution agent, the operating system / target application, and the Visual Model (VLM) during the execution of a user command. The entire process is encapsulated within a loop aimed at completing the task. Each loop consists of three phases: Phase 1: Visual Perception, initiated by the local execution agent, which uses the operating system API to locate the target window, obtain its handle, and request a screenshot, ultimately retrieving the current screen capture data from the operating system. Phase 2: Inference and Planning, the local execution agent sends the screenshot and task context to the Visual Model (VLM). After internal semantic understanding and action planning, the VLM returns a structured sequence of operations (JSON format) containing atomic operation types and normalized coordinates. Phase 3: Action Execution, the local execution agent parses the command, completes coordinate mapping, and generates biomimetic random coordinates. It then calls the operating system API to simulate mouse or keyboard events to execute the operation. After the target application's interface updates, the execution agent waits for the interface to stabilize. After a single action is executed, if the task is not completed, the sequence will explicitly jump back to the starting point of Phase 1, re-acquire the updated screenshot, and thus start the next loop until the task is completed. Finally, the local execution agent sends task completion feedback to the user.

[0066] The graphical user interface automation method provided in this application's embodiments inputs screenshots and user commands into a large visual model. The model then directly outputs a structured operation sequence containing normalized coordinates based on multimodal understanding. This process simulates the human cognitive interaction logic of "viewing the screen - understanding - planning," thereby achieving truly universal automation that does not rely on control trees, document object models, or pre-stored image templates. Furthermore, by dynamically mapping the normalized coordinates output by the model to the absolute coordinates of the current physical screen and driving the operating system input device to execute, decoupling from specific screen resolutions, interface layouts, and visual themes is achieved, significantly improving cross-platform and cross-device adaptability and robustness. Finally, through the above-mentioned full visual path automation process, this method completely eliminates dependence on application implementation details while providing a graphical user interface automation solution that can naturally respond to user commands, has dynamic interface adaptability, and low maintenance costs.

[0067] Figure 4 This is a schematic diagram of the structure of a graphical user interface automation device 400 provided for an exemplary embodiment of this application. Figure 4 As shown, the device 400 includes: an acquisition module 410, a planning module 420, and an operation module 430, wherein: The acquisition module 410 is used to acquire a current screenshot of the target application interface; The planning module 420 is used to input the current screenshot and user command into the visual big model. Based on the visual understanding of the current screenshot and the semantics of the user command, the visual big model plans a structured operation sequence required to complete the user command. The structured operation sequence includes at least one atomic operation and the normalized coordinates of the interactive elements associated with each atomic operation in the screenshot. The operation module 430 is used to parse the structured operation sequence, map the normalized coordinates of the interactive elements associated with each atomic operation to the absolute coordinates of the current physical screen, and drive the operating system's input device to perform the operation at the corresponding absolute coordinates according to the type of each atomic operation.

[0068] The graphical user interface automation device 400 provided in this application embodiment inputs a screenshot and user commands into a large visual model. The model then directly outputs a structured operation sequence containing normalized coordinates based on multimodal understanding. This process simulates the human cognitive interaction logic of "viewing the screen - understanding - planning," thereby achieving truly universal automation that does not rely on control trees, document object models, or pre-stored image templates. Furthermore, by dynamically mapping the normalized coordinates output by the model to the absolute coordinates of the current physical screen and driving the operating system input device to execute, the system achieves decoupling from specific screen resolutions, interface layouts, and visual themes, significantly improving cross-platform and cross-device adaptability and robustness. Finally, through the above-mentioned full visual path automation process, this method completely eliminates dependence on application implementation details while providing a graphical user interface automation solution that can naturally respond to user commands, has dynamic interface adaptability, and low maintenance costs.

[0069] Optionally, the acquisition module 410 is specifically used for: Locate the target application window using semantic fuzzy matching; Call the operating system interface to obtain the handle, position, and size information of the target application window; Based on the position and size information of the window, the coordinate range of its client area is dynamically determined; The image within the coordinate range of the client area is captured as the current screenshot.

[0070] Optionally, the planning module 420 is specifically used for: The visual big data model identifies at least one interactive element in the current screenshot that is related to completing the user command and understands its functional semantics. Determine the position of each identified interactive element in the current screenshot and generate the corresponding normalized coordinates; Based on the semantics of the user instructions and the functional semantics of the interactive elements, the task is decomposed into at least one predefined type of atomic operation, and each atomic operation is associated with the corresponding interactive element and its normalized coordinates.

[0071] Optionally, the operation module 430 is specifically used for: When the atomic operation type is click, the final execution coordinates are generated based on a random algorithm within the screen area determined by the mapped absolute coordinates. The cursor is controlled to move along a trajectory that includes random delay and jitter to the final execution coordinates before a click event is executed.

[0072] Optionally, the device operates in a closed-loop manner: After the input device executes the structured operation sequence once, a new screenshot of the target application interface is obtained. The updated current screenshot and incomplete user instructions or historical context are input into the visual big model to plan the sequence of subsequent operations until the user instructions are determined to be completed. If the previous operation is determined not to have achieved the expected state based on the updated current screenshot, the visual big model will autonomously decide on the subsequent action according to the injected exception handling strategy prompts. The subsequent action includes at least one of the following: retrying the same interactive element, finding and operating on a substitute interactive element, or performing a swipe operation to refresh the interface.

[0073] Optionally, the operation module 430 is specifically used for: Convert the normalized coordinates into logical pixel coordinates corresponding to the current screenshot; Call the operating system interface to obtain the scaling ratio of the current display device, and convert the logical pixel coordinates into physical pixel coordinates based on the scaling ratio; The final absolute coordinates are calculated by combining the position offset of the target application window on the screen.

[0074] The graphical user interface automation device 400 can achieve Figures 1-3 For details of the method implementation examples, please refer to [link / reference]. Figures 1-3 The graphical user interface automation method shown in the embodiment will not be described in detail.

[0075] Figure 5 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. For example... Figure 5 As shown, the device includes a memory 51 and a processor 52.

[0076] Memory 51 is used to store computer programs and can be configured to store various other data to support operation on the computing device. Examples of this data include instructions for any application or method operating on the computing device, contact data, phone book data, messages, images, videos, etc.

[0077] Processor 52, coupled to memory 51, is used to execute computer programs in memory 51 for: acquiring a current screenshot of the target application interface; inputting the current screenshot and user instructions into a visual big model, whereby the visual big model, based on its visual understanding of the current screenshot and the semantics of the user instructions, plans a structured operation sequence required to complete the user instructions, the structured operation sequence containing at least one atomic operation and the normalized coordinates of the interactive elements associated with each atomic operation in the screenshot; parsing the structured operation sequence, mapping the normalized coordinates of the interactive elements associated with each atomic operation to the absolute coordinates of the current physical screen, and driving the operating system's input device to perform an operation at the corresponding absolute coordinates according to the type of each atomic operation.

[0078] The electronic device provided in this application simulates the human cognitive interaction logic of "viewing the screen-understanding-planning" by inputting screenshots and user commands into a large visual model, and having the model directly output a structured operation sequence containing normalized coordinates based on multimodal understanding. This achieves true universal automation without relying on control trees, document object models, or pre-stored image templates. Furthermore, by dynamically mapping the normalized coordinates output by the model to the absolute coordinates of the current physical screen and driving the operating system input device to execute, the system achieves decoupling from specific screen resolutions, interface layouts, and visual themes, significantly improving cross-platform and cross-device adaptability and robustness. Finally, through the above-mentioned automated process of the entire visual path, this method completely eliminates dependence on application implementation details while providing a graphical user interface automation solution that can naturally respond to user commands, has dynamic interface adaptability, and low maintenance costs.

[0079] Furthermore, such as Figure 5 As shown, the electronic device also includes other components such as a communication component 53, a display 54, a power supply component 55, and an audio component 56. Figure 5 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 5 The components shown. Additionally, depending on the implementation of the traffic playback device, Figure 5 The components within the dashed box are optional, not mandatory. For example, when an electronic device is implemented as a terminal device such as a smartphone, tablet, or desktop computer, it may include... Figure 5The components within the dashed box; when the electronic device is implemented as a server-side device such as a conventional server, cloud server, data center, or server array, it may be excluded. Figure 5 The component within the dashed box.

[0080] The above Figure 5 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may further include a Near Field Communication (NFC) module, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, etc.

[0081] The above Figure 5 The memory in the memory can be implemented by any class of volatile or non-volatile storage devices or combinations thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0082] The above Figure 5 The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action, but also the duration and pressure associated with the touch or swipe operation.

[0083] The above Figure 5 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0084] The above Figure 5The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0085] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments.

[0086] Accordingly, this application also provides a computer program product, which stores instructions that, when executed by a computer, cause the computer to perform the steps in the method embodiments provided in this application.

[0087] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0092] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0093] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other classes of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0094] It should be understood that the training and prediction processes of the AI ​​models involved in the various embodiments of this specification all adhere to multiple legal and compliant principles, including legal data sources, compliant data content, compliant data governance, compliant training objectives and schemes, compliant training processes, compliant training environments and tools, and compliant ethical verification of training results, and comply with the requirements of Article 5 of the Patent Law. Among them: Data Source Legality: All datasets used for AI model training were obtained through legal channels, covering three categories: publicly authorized data, data authorized by partners, and self-collected compliant data. Publicly authorized data originates from compliant data sources following open-source licenses such as Apache 2.0, with complete copyright attribution and authorization scope clearly marked, and no unauthorized open-source code or data reuse exists. Data authorized by partners has been subject to formal data usage agreements, clearly defining the scope, duration, and confidentiality obligations, and possessing a complete authorization chain. For self-collected data involving personal information, strict informed consent procedures have been followed, and anonymization processes (including but not limited to field masking, feature anonymization, and differential privacy technology applications) have been implemented to remove personally identifiable information, fully complying with the requirements of the "Interim Measures for the Administration of Generative Artificial Intelligence Services," the "Personal Information Protection Law," and other relevant laws and regulations.

[0095] Data content compliance: The AI ​​model's dataset undergoes multiple screenings and cleaning processes to remove all content that may violate social morality or harm public interests, and does not involve the illegal acquisition or use of genetic resources. For data in sensitive fields (such as healthcare and finance), an additional privacy-preserving computation module (including federated learning and secure multi-party computation technologies) ensures that the data is "usable but not visible," avoiding compliance risks during the original data transmission process and ensuring that the data application scenarios and uses comply with public order and good morals and industry regulatory requirements.

[0096] Data governance compliance: A complete data traceability system is established during the AI ​​model training process to automatically record the source, collection time, annotation process, cleaning rules, and permission allocation of training data, generating traceable compliance reports to ensure that the data is verifiable throughout its entire lifecycle. The dataset annotation process for AI models is completed by a professional human R&D team, clearly defining the proportion of human creative contributions, avoiding reliance on AI-generated data that has not undergone substantial human modification, and complying with the examination requirements for "human main contributions" in AI patent applications.

[0097] Training objectives and scheme compliance: The training objectives of the AI ​​model focus on visual understanding of screenshots and semantic understanding of user commands. The training scheme and the final output results do not violate any mandatory provisions of laws and administrative regulations, do not harm the public interest or the legitimate rights and interests of others, and do not pose any potential risks of being used for illegal activities, privacy infringement, or public safety disruption. The training scheme and the final output results strictly adhere to the ethical principle of "intelligent for good".

[0098] Compliance of the training process: A closed-loop training framework is adopted to ensure compliance and controllability of the training process. The specific process is as follows: First, training samples are obtained through compliant data sources. After the aforementioned data cleaning and desensitization, they are input into the neural network model to generate preliminary training results. Second, an expert system is introduced to verify the preliminary results. Based on preset rules and human expert experience, the feasibility of the results is evaluated, and outputs that may pose ethical risks or compliance hazards are corrected (such as removing decision logic that violates public order and good morals, and adjusting model parameters that do not comply with safety regulations). Finally, the loss function weights are dynamically optimized based on the feedback from the expert system to strengthen the model's learning of compliant results, avoid overfitting errors or non-compliant labels, and form a closed-loop control of "data input - model training - expert verification - parameter optimization - result feedback" to ensure that the entire training process complies with A5 ethical review requirements.

[0099] Training Environment and Tools Compliance: AI model training is implemented based on nationally licensed chips and a compliant training platform. All open-source frameworks and components used in the training process have obtained their corresponding licenses, and copyright statements and patent citation information are fully retained, with no instances of infringement or reuse. The training environment is constructed using virtual devices (containers / virtual machines) with fixed random seeds and initial parameter configurations to ensure the reproducibility of the training process. Furthermore, through access control and operation log recording, risks such as data leakage and parameter tampering during training are prevented, ensuring the security and compliance of the training process.

[0100] Training results ethical verification and compliance: After the model is trained, it undergoes additional third-party ethical compliance assessment and algorithm filing review to verify that the model output does not violate social morality or harm public interests. For potentially sensitive scenarios (such as public services and intelligent decision-making), a special result verification mechanism is established to ensure that the model always complies with Article 5 of the Patent Law and relevant laws and regulations in practical applications.

[0101] In summary, the data and training process used in the AI ​​model of this specification strictly comply with the relevant provisions of Article 5 of the Patent Law and the Patent Examination Guidelines (2023 Edition), and there are no violations of laws, social ethics, public interests, or illegal use of genetic resources. It fully meets the compliance requirements for patent authorization.

[0102] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A graphical user interface automation method, characterized in that, include: Get a screenshot of the current screen of the target application interface; The current screenshot and user command are input into the visual big model. Based on the visual understanding of the current screenshot and the semantics of the user command, the visual big model plans a structured operation sequence required to complete the user command. The structured operation sequence includes at least one atomic operation and the normalized coordinates of the interactive element associated with each atomic operation in the screenshot. The structured operation sequence is parsed, and the normalized coordinates of the interactive elements associated with each atomic operation are mapped to the absolute coordinates of the current physical screen. Based on the type of each atomic operation, the operating system's input device is driven to perform the operation at the corresponding absolute coordinates.

2. The method as described in claim 1, characterized in that, The step of obtaining a current screenshot of the target application interface includes: Locate the target application window using semantic fuzzy matching; Call the operating system interface to obtain the handle, position, and size information of the target application window; Based on the position and size information of the window, the coordinate range of its client area is dynamically determined; The image within the coordinate range of the client area is captured as the current screenshot.

3. The method as described in claim 2, characterized in that, The visual big data model, based on its visual understanding of the current screenshot and the semantics of the user command, plans a structured sequence of operations required to complete the user command, including: The visual big data model identifies at least one interactive element in the current screenshot that is related to completing the user command and understands its functional semantics. Determine the position of each identified interactive element in the current screenshot and generate the corresponding normalized coordinates; Based on the semantics of the user instructions and the functional semantics of the interactive elements, the task is decomposed into at least one predefined type of atomic operation, and each atomic operation is associated with the corresponding interactive element and its normalized coordinates.

4. The method as described in claim 1, characterized in that, The step of driving the operating system's input device to perform operations based on the type of each atomic operation includes: When the atomic operation type is click, the final execution coordinates are generated based on a random algorithm within the screen area determined by the mapped absolute coordinates. The cursor is controlled to move along a trajectory that includes random delay and jitter to the final execution coordinates before a click event is executed.

5. The method according to any one of claims 1 to 4, characterized in that, The method is executed in a closed-loop manner: After the input device executes the structured operation sequence once, a new screenshot of the target application interface is obtained. The updated current screenshot and incomplete user instructions or historical context are input into the visual big model to plan the sequence of subsequent operations until the user instructions are determined to be completed. If the previous operation is determined not to have achieved the expected state based on the updated current screenshot, the visual big model will autonomously decide on the subsequent action according to the injected exception handling strategy prompts. The subsequent action includes at least one of the following: retrying the same interactive element, finding and operating on a substitute interactive element, or performing a swipe operation to refresh the interface.

6. The method as described in claim 1, characterized in that, The step of mapping the normalized coordinates of the interactive elements associated with each atomic operation to the absolute coordinates of the current physical screen includes: Convert the normalized coordinates into logical pixel coordinates corresponding to the current screenshot; Call the operating system interface to obtain the scaling ratio of the current display device, and convert the logical pixel coordinates into physical pixel coordinates based on the scaling ratio; The final absolute coordinates are calculated by combining the position offset of the target application window on the screen.

7. A graphical user interface automation device, characterized in that, include: The acquisition module is used to capture a screenshot of the current screen of the target application interface; The planning module is used to input the current screenshot and user command into the visual big model. Based on the visual understanding of the current screenshot and the semantics of the user command, the visual big model plans a structured operation sequence required to complete the user command. The structured operation sequence includes at least one atomic operation and the normalized coordinates of the interactive element associated with each atomic operation in the screenshot. The operation module is used to parse the structured operation sequence, map the normalized coordinates of the interactive elements associated with each atomic operation to the absolute coordinates of the current physical screen, and drive the operating system's input device to perform the operation at the corresponding absolute coordinates according to the type of each atomic operation.

8. An electronic device, characterized in that, include: One or more processors, memory, and one or more computer programs; The one or more computer programs are stored in the memory and configured to be executed by the one or more processors, the one or more computer programs including instructions for performing the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the steps of the method as described in any one of claims 1 to 7.

Images