A semantic disambiguation click method and system based on RPA and context-aware visual tagging

Abstract

This invention relates to the field of automation technology and proposes a semantic disambiguation click method and system based on RPA and context-aware visual tags. The method involves receiving user natural language prompts and interface screenshots; identifying UI elements and generating basic visual tags using computer vision technology; injecting contextual features into the tags based on spatial geometric calculations to form enhanced visual tags; semantically expanding the enhanced tags using a first-stage multimodal large model to supplement functional labels and contextual descriptions; then, using a second-stage large language model to perform deep matching and disambiguation between user intent and semantic tags to accurately select target elements; finally, mapping the element position to actual screen coordinates to drive RPA to execute the click operation, and forming an operation loop through visual verification. The corresponding system includes a series of functional modules that perform the above steps.

Description

Technical Field

[0001] This invention relates to the field of automation technology, specifically to a semantic disambiguation click method and system based on RPA and context-aware visual tagging. Background Technology

[0002] In the current wave of digital transformation, Robotic Process Automation (RPA) technology is widely used to replace manual execution of repetitive and rule-based software interaction tasks. Its core lies in accurately identifying and manipulating target elements in a graphical user interface (GUI). However, existing mainstream RPA solutions have significant limitations: methods based on fixed screen coordinates or image template matching are extremely fragile and cannot cope with minor changes in interface layout, window scaling, or theme changes; while methods relying on the selection of underlying software control attributes (such as IDs and class names) are limited by the specific application's technical architecture, resulting in poor versatility and difficulty in cross-platform compatibility. With the development of Large Language Models (LLM) and multimodal models, explorations have emerged that use natural language to directly describe operational intentions. However, these methods often lack a deep understanding of the interface's visual structure and the contextual relationships between elements, leading to serious semantic ambiguity problems in complex interfaces. For example, it is difficult to distinguish the specific functional context of multiple "confirm" buttons, resulting in a mismatch between operational intentions and execution targets, and insufficient reliability.

[0003] Therefore, a semantic disambiguation click method and system based on RPA and context-aware visual tags is proposed. First, computer vision technology is used to generate visual tags for UI elements containing basic geometric information. Then, spatial relationship calculations are used to inject contextual features such as row, column, proximity, and region affiliation, constructing enhanced visual tags. Subsequently, this scheme innovatively adopts a two-stage large-scale model processing flow: the first stage uses a multimodal large-scale model to semantically expand the enhanced tags, giving each element a fine-grained type, functional semantic label, and rich contextual description; the second stage uses a dedicated large language model to perform deep matching and disambiguation between the user's natural language intent and the semantically generated visual tags, ultimately accurately locking the target and driving RPA to perform coordinate mapping and closed-loop verification click operations. Summary of the Invention

[0004] To address the vulnerabilities of existing RPA technologies that rely on fixed coordinates or image templates, as well as the semantic ambiguity issues that arise when directly using natural language commands, this invention provides a semantic disambiguation click method and system based on RPA and context-aware visual tags.

[0005] To achieve the above-mentioned objective, this invention provides the following technical solution: a semantic disambiguation click method based on RPA and context-aware visual tags, comprising the following steps: It receives natural language prompts and screenshots of the user interface.

[0006] The screenshot is subjected to UI element detection, and a basic visual tag containing location and type information is generated for each identified UI element.

[0007] Based on the positional information of UI elements in the basic visual markup, spatial relationship calculations are performed to generate enhanced visual markup containing spatial context features.

[0008] The enhanced visual tags and screenshots are input into the first-stage multimodal large model to obtain semantic visual tags containing semantic labels and contextual information.

[0009] The natural language prompts and semantic visual tags are input into the second-stage large language model for semantic matching, and the target UI elements that match the user's intent and their matching confidence scores are output.

[0010] Based on the location information of the target UI element and the metadata of the screenshot, the click coordinates in the actual screen coordinate system are calculated.

[0011] The RPA click operation is performed based on the click coordinates, and the interface after the operation is visually verified to confirm the operation result.

[0012] Preferably, the step of detecting UI elements in the screenshot includes: using a trained UI object detection model or a multimodal large model to identify various UI elements in the screenshot, and generating a structured basic visual label for each element with a unique number, type, bounding box position and detection confidence.

[0013] Preferably, the first-stage multimodal large model is configured to perform the following tasks: supplement fine-grained types for each UI element, improve its contextual description, and generate standardized semantic tags that describe its functions.

[0014] Preferably, the step of performing spatial relationship calculation includes: performing row or column clustering based on the center coordinates of UI elements, and identifying elements in the same row or column; calculating the Euclidean distance between elements, and identifying several nearest neighbor elements for each element; and identifying the large region type and parent region to which each element belongs based on the enclosing and aggregation relationships between elements.

[0015] Preferably, before inputting the data into the second-stage large language model, the method further includes: filtering out irrelevant UI elements based on the similarity between the natural language prompts and the text or semantic tags of elements in the semantic visual markup.

[0016] Preferably, the step of calculating the click coordinates includes: calculating the center point of the target UI element in the screenshot coordinate system based on the bounding box position of the target UI element; converting the center point into the actual click coordinates in the screen coordinate system by combining the offset value of the upper left corner of the screenshot in the screen coordinate system and the screen scaling parameters; and verifying whether the deviation between the position of the current operation window and the position when the screenshot was captured is within the allowable threshold before clicking.

[0017] Preferably, the step of visually verifying the interface after the operation includes: waiting for a preset time after the click operation and taking a second screenshot of the interface; judging whether the interface has undergone the expected change based on the user's operation intention by visual feature matching; if the expected change is not detected, executing a multi-level retry strategy, the strategy including: changing the click position in the target element area to retry, or re-executing the complete process starting from UI element detection.

[0018] A semantic disambiguation click system based on RPA and context-aware visual tagging, wherein the system is applied to the aforementioned semantic disambiguation click method based on RPA and context-aware visual tagging, and the system includes the following modules: The user input module is used to receive and standardize natural language prompts and screenshots of the interface and their metadata. The CV element detection module is used to identify UI elements in screenshots and generate a list of basic visual markers. The context-aware visual tag processing module is used to perform spatial relationship calculations and feature enhancement on the basic visual tags to generate an enhanced visual tag list; The first-stage large model extension semantic construction module is used to call the multimodal large model to semantically extend the enhanced visual tags and generate a list of semantic visual tags. The second-stage large-scale model semantic disambiguation matching module is used to call the large language model to match the natural language prompt words with the semantic visual tag list and output the target element; The coordinate mapping module is used to calculate the actual click coordinates in the screen coordinate system and perform window position verification based on the position of the target element and the screenshot metadata. The RPA execution and closed-loop verification module is used to execute click operations and verify the results and retry through secondary screenshots and visual analysis.

[0019] Preferably, it also includes a log caching module for structured storage of the semantic visual tag list, as well as the system's full-process operation logs, exception information, and retry records, and provides query and analysis functions.

[0020] Compared with existing technologies, this invention provides a semantic disambiguation click method and system based on RPA and context-aware visual tagging, which has the following beneficial effects: 1. This solution constructs a middleware architecture for context-aware visual tag generation and semantic understanding, fundamentally changing the traditional RPA's mechanical localization mode that relies on fixed coordinates, image templates, or low-level code attributes. By integrating initial detection from computer vision, context enhancement based on geometric relationships, and collaborative reasoning from a two-stage large model (visual understanding and semantic disambiguation), this solution enables automated systems to comprehensively understand the visual layout of graphical user interfaces, the spatial logical relationships between elements, and their functional semantics, just like humans. This achieves accurate interpretation of the user's fuzzy natural language commands and matching of target elements.

[0021] 2. In this solution, a paradigm shift from feature matching to semantic understanding enables the automated process to possess strong fault tolerance and adaptability. The system can not only cope with dynamic changes such as changes in the position of interface elements and updates in control types, but also accurately handle interactive elements that are synonymous but have diverse forms of expression. Furthermore, a closed-loop verification mechanism ensures the reliability of the operation results, ultimately achieving a highly accurate, robust, and easily natural human-computer interaction intelligent automation effect. Attached Figure Description

[0022] Figure 1 This is a flowchart of the semantic disambiguation click method of the present invention; Figure 2 This is a module architecture diagram of the semantic disambiguation click system of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1:

[0024] Please see Figure 1 This embodiment provides a preferred method for implementing the present invention, namely, a semantic disambiguation click method based on RPA and context-aware visual tags. Using user-input natural language prompts and interface screenshots as basic data, basic visual tags for UI elements are generated through computer vision technology. The context-aware visual tags are enhanced by combining spatial relationship preprocessing. Then, the semantic expansion of the visual tags and semantic disambiguation matching based on user intent are realized sequentially through a two-stage large model. Finally, coordinate system mapping and RPA precise click operation are completed, and the effectiveness of the operation is ensured through closed-loop verification and multi-level retry mechanism.

[0025] This method comprises the following seven steps, with unidirectional data flow between each step, and each step supports node-level retry and error correction. If any step fails (such as abnormal data acquisition, model parsing failure, or validation failure), the retry mechanism for that step is triggered. The preset number of retries is 3. If the retry still fails, a manual confirmation and intervention process is triggered, as detailed below: Step 1: Raw data and screenshot metadata collection The system collects and receives two types of core technical data from user input through a human-computer interaction interface, serving as the raw input data for RPA click operations. The first type is natural language prompts that express the intention of the operation, serving as the intent benchmark for semantic disambiguation. These prompts support expressions in mainstream natural languages ​​such as Chinese and English, and are non-empty valid text. The second type is full-screen or regional screenshots of the current operation interface. These screenshots support computer-readable formats such as file paths, binary streams, and image pixel matrices, and are complete, undamaged, and valid image data.

[0026] The metadata of the screenshot material is collected and recorded synchronously. The metadata consists of the spatial and attribute parameters of the screenshot in the computer system. It is the basic data for subsequent coordinate system mapping and window position verification. It includes: the horizontal and vertical offset values ​​of the top left corner of the screenshot in the virtual desktop coordinate system, the pixel width and pixel height of the screenshot, the computer screen DPI scaling parameters, the timestamp of the screenshot collection, and the unique handle identifier of the operation window.

[0027] If a data collection anomaly occurs in this step (such as a corrupted screenshot, empty prompt words, or failure to extract metadata), a collection failure message will be displayed to the user, and the re-collection operation in this step will be triggered. If the collection still fails after 3 retries, a manual confirmation and intervention process will be triggered.

[0028] Step 2: UI Element Detection and Basic Visual Markup Generation A dedicated UI object detection model or a general multimodal large model trained on a UI element sample set is used to analyze the collected screenshot materials based on pixel features, and to identify various interactive / non-interactive UI elements in the screenshots, such as buttons, input boxes, icons, text blocks, pop-ups, menus, and drop-down boxes. The element recognition results and single element detection confidence scores are output. The overall confidence score threshold for UI element recognition is set to 80%. If the overall confidence score is lower than this threshold, the model is re-analyzed or the screenshot is re-collected to correct the error.

[0029] A basic visual tag is generated for each identified UI element. The basic visual tag adopts a computer-readable structured data format, preferably JSON, but XML can also be used. The required content includes: element unique number, element basic type, rectangular boundary range (x1, y1, x2, y2) in the screenshot coordinate system with the top left corner of the screenshot as the origin, element original text, and detection confidence of the model in recognizing the element; the optional content includes the element's color features, pixel size, and whether it is an interactive element.

[0030] All the basic visual tags of all UI elements are integrated into a basic visual tag list according to the element's unique number. The basic visual tag list is temporarily stored in computer memory as the basic data material for subsequent context enhancement and semantic processing.

[0031] Step 3: Spatial Relationship Preprocessing and Enhanced Visual Tag Generation Based on the rectangular boundary range information in the screenshot coordinate system of the basic visual markers, spatial relationship calculation and inference operations are performed on the local terminal to supplement the basic visual markers with spatial context features and generate enhanced context-aware visual markers, so as to reduce the inference calculation of subsequent large models and improve the accuracy of semantic disambiguation.

[0032] Specifically, three types of spatial relationship processing operations based on pixel coordinate geometric calculations are performed. The results of these operations are embedded with basic visual labels using feature fields: 1) Row / column clustering operation: By calculating the center ordinate and abscissa of the element's rectangular boundary, UI elements in the same row or column are identified, and a list field of sibling element numbers is added to them; 2) Nearest neighbor analysis operation: The Euclidean distance of each element to other elements is calculated, the 5 closest UI elements are selected, and a list field of neighboring element numbers is generated; 3) Large region clustering operation: Based on the enclosing relationship between elements and pixel region aggregation features, large regions such as dialog boxes, pop-ups, sidebars, and main windows are roughly identified, and each element is labeled with feature fields such as candidate parent region number, whether it is in a pop-up window, and the type of large region it belongs to.

[0033] The aforementioned spatial context feature fields are embedded into the basic visual tags of the corresponding UI elements to form an enhanced context-aware visual tag list, realizing the feature upgrade of visual tags from single element features to "element + spatial relationship"; if spatial calculation anomalies occur in this step, invalid element features are removed and the calculation is re-executed to ensure the validity of the tag list.

[0034] Step 4: First-stage large-scale model processing and semantic visual tag generation The screenshot material or its partial cropped image reference, the enhanced context-aware visual tag list, and the spatial relationship inference results are input into the first-stage large model for processing through a standardized API interface. The first-stage large model is a multimodal large model with visual understanding and semantic analysis capabilities, supporting mainstream multimodal large models such as domestic multimodal large models and GPT-type visual models, and can be deployed on local terminals or cloud servers.

[0035] The system pre-defined system instructions define the functional roles of the UI layout and semantic analysis assistant for the first-stage large model, clarifying their processing responsibilities as follows: supplementing each UI element with fine-grained types, improving contextual information, and generating standardized English semantic tags; wherein, the fine-grained types are UI element types that are more subdivided than the basic types, including ordinary buttons, icon buttons, text input boxes, title text, menu options, etc.; the contextual information is the element's contextual spatial attributes, including pop-up title, parent area name, relative position in the parent area, neighboring icon types, whether it is a core operation element, etc.; the standardized English semantic tags are unified tags used to describe the element's function, facilitating subsequent cross-platform and cross-model semantic processing, including confirm, cancel, delete_dialog, primary_action, etc.

[0036] In the first stage, the large model outputs semantic visual tagging information according to a preset structured format. The semantic visual tagging information includes six core contents: element-specific number, element display text, element fine type, rectangular boundary range in the screenshot coordinate system, context features, and semantic tags. The context features include information such as whether it is in a pop-up window, pop-up window title, neighboring icons, parent region number, relative position in the parent region, and list of sibling element numbers.

[0037] The output of the first-stage large model is subjected to structured validation. The information format, field integrity, and data validity are checked against preset rules using a preset JSON Schema or structured validation library. If parsing fails, automatic error correction (such as field completion and format conversion) or model re-inference is performed to ensure the structural stability of the semantic visual markup list. After the validation is passed, all semantic visual marks are integrated into a semantic context-aware visual markup list.

[0038] Step 5: Second-stage large model processing and semantic disambiguation matching Before inputting the natural language prompts and the semantic context-aware visual tag list into the second-stage large model, a local coarse screening operation is performed on the semantic visual tag list on the local terminal. Text filtering is performed by cosine similarity algorithm or edit distance algorithm, and tag filtering is performed by precise matching based on semantic tags. UI elements that are irrelevant to the user's operation intention are eliminated, reducing the inference computation of the second-stage large model and improving matching efficiency.

[0039] The coarsely filtered list of semantic visual tags and the user's natural language prompts are input into the second-stage large model through a standardized API interface. The second-stage large model is a large language model with natural language understanding and semantic matching capabilities. It can be the same model instance or a different model instance as the first-stage large model, and supports mainstream large language models such as domestic large language models and GPT-type large language models.

[0040] The second-stage large model is defined by pre-defined system instructions, which define the functional role of the RPA UI element selector and set three rigid constraints: 1) Select target UI elements only from the semantic visual mark list after coarse screening, and do not generate element information outside the list; 2) Output semantic matching results according to a preset fixed format; 3) The matching results must include a confidence score in the range of 0-1 and an explanation of the natural language matching reason.

[0041] In the second stage, the large model combines the keywords, operation scenarios, and intent tendencies in the natural language prompts with the text, contextual features, and semantic tags of UI elements for multi-dimensional deep matching. It performs semantic disambiguation and quantifies the degree of fit between the elements and the user's operation intent, outputting the disambiguation matching result. The disambiguation matching result is computer-recognizable structured data, including the selected element number, matching confidence (a value between 0 and 1), natural language matching reason, and a list of candidate elements. The list of candidate elements is sorted from high to low confidence, and each candidate element is labeled with an number, matching confidence, and reason for not being selected.

[0042] Post-processing verification is performed on the output of the second-stage large model. Verification items include: whether the selected element number exists in the semantic visual tag list, and whether the matching confidence reaches a preset threshold of 90%. If the verification passes, proceed to the next step; if the verification fails, trigger the re-matching operation of this step. If the matching confidence is lower than 90%, the manual confirmation and intervention process is directly triggered to avoid the execution of erroneous click operations.

[0043] Step 6: Coordinate Calculation and Window Position Verification Extract the rectangular boundary range (x1, y1, x2, y2) information under the screenshot coordinate system from the selected semantic visual mark. Using the geometric center of the element rectangular area as the reference, calculate the click reference point in the screenshot coordinate system. The calculation formula is: horizontal coordinate of the click reference point = (x1 + x2) / 2, vertical coordinate of the click reference point = (y1 + y2) / 2.

[0044] Based on the screenshot metadata collected in step 1, the click reference point in the screenshot coordinate system is converted into the actual click coordinates in the computer virtual desktop coordinate system. The calculation formula is: actual click horizontal coordinate = horizontal offset value of the top left corner of the screenshot + horizontal coordinate of the click reference point × screen horizontal scaling parameter; actual click vertical coordinate = vertical offset value of the top left corner of the screenshot + vertical coordinate of the click reference point × screen vertical scaling parameter.

[0045] The actual position parameters of the current operating window are read through the window handle of the computer system. The timeliness of the screenshot is determined by combining it with the screenshot acquisition timestamp in step 1. The window position is compared with the window position parameters at the time of screenshot acquisition, and a window position verification operation is performed. The pixel position deviation threshold is set to ±50. If the deviation between the current window position and the position at the time of screenshot acquisition exceeds the threshold, the click operation is rejected and an instruction to re-acquire the screenshot is output to the user to avoid coordinate offset and click operation errors caused by window movement and interface scaling. If the deviation is within the threshold range, the actual click coordinates are confirmed to be valid, and the process proceeds to the next step.

[0046] Step 7: RPA Click Execution and Closed-Loop Verification The system calls the underlying mouse control interface of RPA at the computer system level and executes pointer movement and click operations sequentially according to the actual click coordinates in the virtual desktop coordinate system. The click operation supports common RPA mouse operations such as single click, double click, right click, and long press of the left button, and is adaptively selected based on the user's natural language prompts.

[0047] After the click operation is completed, wait for a preset time of 300~800 milliseconds (the waiting time can be customized according to the actual application scenario). Then, use computer vision technology to perform a secondary screenshot analysis of the current operation interface. Based on the user's operation intention and preset visual feature matching rules, determine whether the interface has undergone the expected changes. The expected changes include pop-up closing, page jump, button state change, input box activation, etc., and perform a closed-loop verification operation of the click result in this way.

[0048] If the expected change is detected, the click operation is considered successful, the method flow ends, and a success log is recorded. If the expected change is not detected, the following multi-level retry strategy is executed: 1) First-level retry: Generate 5 candidate click points at different locations (corners and edges) within the element's rectangular area, execute the click operation sequentially, and verify the results; 2) Second-level retry: Re-execute the entire process of "Steps 2-7" of this method, and re-complete element detection, semantic matching, and coordinate calculation before executing the click operation; 3) If both levels of retry fail (each retry count is 3 times), an exception log is recorded, and a manual confirmation and intervention process is triggered. The exception log must contain complete information such as the entire process data, failure nodes, and retry records to achieve robust closed-loop operation of the method. Example 2:

[0049] Please see Figure 2 To implement the semantic disambiguation click method based on RPA and context-aware visual tags described in Example 1, this embodiment provides a corresponding semantic disambiguation click system based on RPA and context-aware visual tags. The system is a modular system combining hardware and software. The hardware can be deployed on a terminal computer, server, embedded device, or cloud server cluster. Each functional module interacts and transmits technical data through a system bus, internal network, or standardized API interface. The software adopts a modular and decoupled architecture design. Each functional module is a functional unit combining hardware and software, which can be implemented through hardware circuits and computer-executable program instructions. The program instructions are stored in a computer-readable storage medium. Each unit can be deployed, optimized, and upgraded independently. Modules interact with each other through unified structured context-aware visual tags.

[0050] The system comprises seven core functional modules and one auxiliary functional module. Its overall architecture is an end-to-end structure: "Input Layer → Detection Layer → Preprocessing Layer → Semantic Expansion Layer → Matching Layer → Coordinate Layer → Execution Verification Layer." Each module corresponds to the method steps in Embodiment 1, enabling automated execution of all steps. A log caching module is also configured to achieve full-process data storage and analysis. The hardware composition, core software functions, and input / output data of each module are clearly defined, as follows: Module 1: User Input Module As the input layer unit of the system, the hardware includes: a human-computer interaction input interface, a data acquisition interface, a temporary storage cache unit, and a GPU / CPU basic computing unit; the software consists of computer program instructions stored in a computer-readable storage medium, which realize the functions of raw data acquisition, parsing, and standardization processing; it is used to collect and receive natural language prompts and interface screenshots input by users through the human-computer interaction input interface, complete the standardization processing of raw data, and supports multiple input formats such as text strings, file paths, binary streams, and image pixel matrices.

[0051] The core functions of the software are: 1) Standardizing natural language prompts, including noise reduction, word segmentation, and keyword extraction, to generate structured prompt data; 2) Converting the format and verifying the integrity of screenshot materials, and removing damaged or invalid screenshots; 3) Automatically extracting metadata such as horizontal offset, vertical offset, pixel width and height, and DPI scaling factor from screenshot materials and storing them as structured metadata; 4) Categorizing and storing all collected data by timestamp to provide data support for subsequent processes.

[0052] Input data: Natural language prompts entered by the user through the human-computer interaction interface, and screenshot materials; Output data: Standardized natural language intent data, standardized screenshot data (binary stream / pixel matrix), and a structured list of screenshot metadata.

[0053] Module 2, CV Element Detection Module As the detection layer unit of the system, the hardware includes: GPU / TPU / NPU model inference chip, high-speed memory computing unit, and data interaction interface; the software integrates UI detection model, element tag generation program, and confidence judgment program, which is the basic building block of context-aware visual tags; and realizes the detection of UI elements and the generation of basic visual tags based on computer vision technology.

[0054] The core functions of the software are: 1) Loading a dedicated UI object detection model or a general multimodal large model trained on a UI element sample set, and performing pixel feature-based analysis on standardized screenshot data; 2) Identifying various UI elements in the screenshot, outputting the element identification results and detection confidence, and marking elements with low confidence; 3) Generating a unique number for each identified UI element, extracting element features and generating basic visual tags according to a preset structured format; 4) Integrating all basic visual tags into a basic visual tag list, and outputting it after completing validity verification.

[0055] Input data: Standardized screenshot data, structured list of screenshot metadata; Output data: UI element recognition result set, basic context-aware visual tag list, low-confidence element tag information.

[0056] Module 3: Context-Aware Visual Tag Processing Module As the preprocessing layer unit of the system, the hardware includes: CPU / MCU computing processor, local data cache unit, and data interaction interface; the software integrates spatial relationship geometric calculation algorithm, feature embedding program, and feature verification program, and is the core preprocessing unit of the system; it is used to complete the spatial context enhancement of visual tags and realize bidirectional technical data interaction and transmission with the CV element detection module and the first-stage large model extension semantic construction module.

[0057] The core functions of the software are: 1) Based on the location range information of UI elements, perform spatial relationship calculations such as row / column clustering, nearest neighbor analysis, and large area clustering to generate spatial context features of elements; 2) Embed the spatial context features into the basic visual tags of the corresponding UI elements to generate enhanced context-aware visual tags; 3) Provide feature verification functions to remove invalid spatial relationship features and damaged tag data; 4) Integrate the enhanced visual tags into a list and store them in a structured manner.

[0058] Input data: Basic context-aware visual marker list, UI element recognition result set; Output data: Enhanced context-aware visual marker list, spatial relationship inference result set, invalid element marker information.

[0059] Module 4: First Phase Large Model Extended Semantic Construction Module As the semantic extension layer unit of the system, the hardware includes: a model deployment server (GPU / TPU cluster), a standardized model calling interface, and a high-speed data transmission unit; the software integrates a multimodal large model inference program, a structured verification program, and an automatic error correction program to realize the semantic extension of visual tags and provide full-dimensional semantic information of UI elements for semantic disambiguation matching.

[0060] The core functions of the software are: 1) Inputting screenshots, enhanced visual tag lists, and spatial relationship inference results into the multimodal large model through a standardized model call interface; 2) Loading pre-configured customized system instruction templates to define fixed functional roles and output formats for the large model, guiding it to complete the fine-grained type supplementation of UI elements, the improvement of contextual information, and the generation of standardized semantic tags; 3) Performing structured verification and automatic error correction on the output results of the large model, filling in missing fields and correcting format errors; 4) Integrating the verified semantic tags into a semantic context-aware visual tag list, supporting custom output rule configuration.

[0061] Input data: Standardized screenshot data, enhanced context-aware visual tag list, spatial relationship inference result set; Output data: Semantic context-aware visual tag list (extended semantic list), model inference log, structured validation results.

[0062] Module 5, Second Stage Large Model Semantic Disambiguation Matching Module As the matching layer unit of the system, the hardware includes: a large language model inference server, a local coarse screening calculation unit, and a data comparison and verification unit; the software integrates a natural language understanding program, a semantic matching algorithm, and a result verification program, and is the core semantic processing unit of the system; it is used to realize deep semantic disambiguation matching between user operation intentions and UI elements.

[0063] The core functions of the software are: 1) Built-in local coarse screening engine, which performs text similarity matching and precise filtering based on semantic tags through cosine similarity / edit distance algorithm to coarsely screen the list of semantic visual tags; 2) Input the coarsely screened tag list and natural language prompts into the large language model through a standardized model call interface, and load the exclusive system instruction template to constrain the output format of the model; 3) Receive the disambiguation matching results of the large model and perform post-processing verification, including element number validity check and confidence threshold judgment; 4) Sort the candidate element list by confidence, generate anomaly markers and trigger the corresponding process.

[0064] Input data: Standardized natural language intent data, semantic context-aware visual tag list; Output data: Selected target element number, matching confidence, explanation of matching reason, list of alternative elements, anomaly flags (low confidence / element number does not exist / no matching element).

[0065] Module Six: Coordinate Mapping Module As the coordinate layer unit of the system, the hardware includes: a floating-point numerical calculation processor, a system window interface call unit, and a data cache unit; the software integrates coordinate calculation algorithms, coordinate system transformation programs, and window position verification programs to complete coordinate system transformation and calculation of precise click points, and to achieve technical data connection with the second-stage large model semantic disambiguation matching module, RPA execution and closed-loop verification module.

[0066] The core functions of the software are: 1) Extracting the position range information of the selected element from semantic visual tags and generating the center click point in the screenshot coordinate system through coordinate calculation algorithm; 2) Combining screenshot metadata to complete the mapping transformation from the screenshot coordinate system to the virtual desktop coordinate system and generate the actual click coordinates; 3) Reading the current operation window position through the system window handle interface and comparing it with the position when the screenshot was captured to perform verification; 4) Generating coordinate validity identifiers and triggering a re-capture command for invalid coordinates.

[0067] Input data: semantic visual markers of the selected target element, and a structured list of screenshot metadata; Output data: actual click coordinates in the virtual desktop coordinate system, coordinate validity indicators, and window position verification results.

[0068] Module 7: RPA Execution and Closed-Loop Verification Module As the execution and verification layer unit of the system, the hardware includes: RPA underlying interface calling unit, secondary detection and acquisition unit, and log recording and storage unit; the software integrates mouse control program, closed-loop detection algorithm, multi-level retry program, and exception handling program, and is the core unit of system execution and verification; it is used to call the RPA underlying interface to implement click operation and complete closed-loop verification, multi-level retry and exception handling of click results.

[0069] Core software functions: 1) Integrates system-level RPA underlying mouse interface, executes pointer movement and click operations according to actual click coordinates, and supports multiple types of mouse operations; 2) After the click operation is completed, a second screenshot of the current interface is captured, and visual feature matching is used to determine whether the expected change has occurred; 3) Built-in multi-level retry strategy, performs multi-position retry within the element and full-process retry for operations that fail to be verified; 4) Real-time recording and classification statistics of abnormal logs, generates completion logs for successful operations, and triggers manual confirmation and intervention processes for failed operations.

[0070] Input data: actual click coordinates in the virtual desktop coordinate system, coordinate validity indicators, and user natural language intent data; Output data: click operation result (success / failure), operation process log, exception information, and retry records.

[0071] Module 8, Log Cache Module As the foundational support module of the system, the hardware includes: a database server (non-relational document database such as MongoDB / relational database such as MySQL), a large-capacity data storage array, and a multi-dimensional query and analysis interface; the software integrates data caching programs, log management programs, and query and analysis programs, built on a computer-readable database, to realize the storage of context-aware visual tags and the analysis of the entire process log, thereby improving the traceability and optimization capabilities of the system.

[0072] Core software functions: 1) Supports structured caching and fast retrieval of semantic context-aware visual tag lists, enabling the reuse of historical tag data and reducing redundant computation; 2) Records operation logs for the entire system process, including input data, output data of each module, model inference results, click operation results, exception information, retry records, etc. Log data is stored or expired and cleaned up according to preset strategies; 3) Provides multi-dimensional query, statistical and visualization analysis functions for logs, providing data support for model optimization and system performance improvement; 4) Compatible with mainstream database systems, supports custom configuration of caching strategies such as expiration time, storage capacity, and data backup, and supports the export and secondary analysis of log data.

[0073] Input data: Output data of each module in the system, operation logs, tag lists, and exception information; Output data: Cache data query results, log statistical analysis reports, and model optimization data support sets.

[0074] Finally, it should be noted that the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Any obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A semantic disambiguation click method based on RPA and context-aware visual tagging, characterized in that: Includes the following steps: Receive natural language prompts and screenshots of the user interface from the user input; The screenshot is subjected to UI element detection, and a basic visual tag containing location and type information is generated for each identified UI element; Based on the positional information of UI elements in the basic visual markup, spatial relationship calculations are performed to generate enhanced visual markup containing spatial context features; The enhanced visual tags and screenshots are input into the first-stage multimodal large model to obtain semantic visual tags containing semantic labels and contextual information; The natural language prompts and semantic visual tags are input into the second-stage large language model for semantic matching, and the target UI elements that match the user's intent and their matching confidence scores are output. Based on the location information of the target UI element and the metadata of the screenshot, calculate the click coordinates in the actual screen coordinate system; The RPA click operation is performed based on the click coordinates, and the interface after the operation is visually verified to confirm the operation result.

2. The semantic disambiguation click method based on RPA and context-aware visual tagging according to claim 1, characterized in that: The step of detecting UI elements in the screenshot includes: using a trained UI object detection model or a multimodal large model to identify various UI elements in the screenshot, and generating a structured basic visual label for each element with a unique number, type, bounding box position and detection confidence.

3. The semantic disambiguation click method based on RPA and context-aware visual tagging according to claim 1, characterized in that: The first-stage multimodal large model is configured to perform the following tasks: supplement fine-grained types for each UI element, improve its contextual description, and generate standardized semantic tags that describe its functions.

4. The semantic disambiguation click method based on RPA and context-aware visual tagging according to claim 1, characterized in that: The steps for performing spatial relationship calculations include: Cluster elements by row or column based on their center coordinates, and identify elements in the same row or column. Calculate the Euclidean distance between elements and identify several nearest neighbor elements for each element; Based on the enclosing and aggregation relationships between elements, identify the type of large region to which each element belongs and its parent region.

5. The semantic disambiguation click method based on RPA and context-aware visual tagging according to claim 1, characterized in that: Before inputting the data into the second-stage large language model, the process also includes filtering out irrelevant UI elements based on the similarity between the natural language prompts and the text or semantic tags of elements in the semantic visual markers.

6. The semantic disambiguation click method based on RPA and context-aware visual tagging according to claim 5, characterized in that: The steps for calculating the click coordinates include: Calculate the center point of the target UI element in the screenshot coordinate system based on the bounding box position of the target UI element; By combining the offset value of the top left corner of the screenshot in the screen coordinate system and the screen scaling parameters, the center point is converted into the actual click coordinates in the screen coordinate system; Before clicking, verify whether the deviation between the current operation window position and the position when the screenshot was taken is within the allowable threshold.

7. The semantic disambiguation click method based on RPA and context-aware visual tagging according to claim 1, characterized in that: The step of visually verifying the interface after operation includes: After clicking the action, wait for the preset time and then take a second screenshot of the interface; Based on the user's operational intent, visual feature matching is used to determine whether the interface undergoes the expected changes; If the expected change is not detected, a multi-level retry strategy is executed, which includes: retrying by changing the click position within the target element area, or re-executing the entire process starting from UI element detection.

8. A semantic disambiguation click system based on RPA and context-aware visual tagging, wherein the system is applied to the semantic disambiguation click method based on RPA and context-aware visual tagging as described in any one of claims 1 to 7, characterized in that: The system includes the following modules: The user input module is used to receive and standardize natural language prompts and screenshots of the interface and their metadata. The CV element detection module is used to identify UI elements in screenshots and generate a list of basic visual markers. The context-aware visual tag processing module is used to perform spatial relationship calculations and feature enhancement on the basic visual tags to generate an enhanced visual tag list; The first-stage large model extension semantic construction module is used to call the multimodal large model to semantically extend the enhanced visual tags and generate a list of semantic visual tags. The second-stage large-scale model semantic disambiguation matching module is used to call the large language model to match the natural language prompt words with the semantic visual tag list and output the target element; The coordinate mapping module is used to calculate the actual click coordinates in the screen coordinate system and perform window position verification based on the position of the target element and the screenshot metadata. The RPA execution and closed-loop verification module is used to execute click operations and verify the results and retry through secondary screenshots and visual analysis.

9. The semantic disambiguation click system based on RPA and context-aware visual tagging according to claim 8, characterized in that: It also includes a log caching module, which is used to structure and store the semantic visual tag list as well as the system's operation logs, exception information and retry records throughout the entire process, and provides query and analysis functions.

Images