Element positioning and self-adaptive execution method and system in robot process automation

Abstract

The invention provides an element positioning and self-adaptive execution method and system in robot process automation, and the method comprises the steps: obtaining a to-be-executed operation instruction, and converting the to-be-executed operation instruction into a semantic intention vector through a target semantic analysis model; identifying a plurality of candidate interface elements on a current user interface to be operated, respectively extracting and fusing image visual features, structural topological features and text content features of corresponding areas, and generating a multi-modal feature vector; calculating a similarity score between the semantic intention vector and the multi-modal feature vector of each candidate element, and determining the candidate interface element exceeding a preset similarity matching threshold as a target operation element; after the interaction action for the target operation element is executed, if it is detected that the expected page state does not occur, the similarity matching threshold value is adjusted, the candidate interface elements are matched again after updating, and then self-adaptive positioning retry is carried out. According to the method and the device, the stability and the automatic continuous execution capability of the RPA system in a frequent UI change environment are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of robotic process automation technology, and in particular to a method and system for element localization and adaptive execution in robotic process automation. Background Technology

[0002] In fields such as finance, taxation, and e-commerce, Robotic Process Automation (RPA) technology is widely used for process automation in web or desktop applications. However, existing RPA systems rely on coordinate positioning, element selectors, or image matching for interface element recognition, which has significant limitations. Screen coordinate-based positioning is greatly affected by changes in resolution and interface layout, making it difficult to execute stably; DOM-based selectors become ineffective after changes to the page's HTML hierarchy, ID, or Class attributes; and image matching is sensitive to visual details such as color, font, and borders, with even minor modifications leading to matching failures. All of these methods lack the ability to understand the functional semantics of interface elements, making them unable to adapt to frequent UI redesigns or dynamically generated content. This results in poor script execution stability, high maintenance costs, and difficulty in handling complex scenarios such as A / B testing and random pop-ups. Summary of the Invention

[0003] The purpose of this application is to provide a method and system for element localization and adaptive execution in robotic process automation, so as to alleviate the above-mentioned technical problems existing in the prior art.

[0004] In a first aspect, the present invention provides a method for element localization and adaptive execution in robotic process automation, comprising: Obtain the operation instructions to be executed, and transform the operation instructions into high-dimensional semantic intent vectors through the target semantic parsing model; In the current user interface to be operated, multiple candidate interface elements are identified. For each candidate interface element, the image visual features, structural topology features, and text content features of its corresponding region are extracted. The three types of features are then fused to generate a multimodal feature vector for each candidate interface element. Calculate the similarity score between the semantic intent vector and the multimodal feature vector of each candidate element, and determine the candidate interface elements whose similarity scores exceed the preset similarity matching threshold as the target operation elements; After performing an interactive action on the target element, if the expected page state is not detected, the similarity matching threshold is adjusted, and candidate interface elements are re-matched based on the updated similarity matching threshold for adaptive positioning retry.

[0005] In an optional implementation, the operation instruction to be executed is obtained, and the operation instruction is transformed into a high-dimensional semantic intent vector through a target semantic parsing model, including: Obtain historical human-computer interaction logs, determine the instruction training set based on the historical human-computer interaction logs, and train the target semantic parsing model through the instructions; The operation instructions are semantically decomposed using a target semantic parsing model to extract functional intent information and contextual constraint information. Functional intent information and contextual constraint information are encoded into a first semantic representation and a second semantic representation in a continuous vector space, respectively. The first semantic representation and the second semantic representation are then fused to generate a high-dimensional semantic intent vector. The high-dimensional semantic intent vector is used to indicate the action type and target of the operation instruction.

[0006] In an optional implementation, for each candidate interface element, image visual features, structural topological features, and text content features of its corresponding region are extracted, and the three types of features are fused to generate a multimodal feature vector for each candidate interface element, including: For each candidate interface element, the image region corresponding to the image region and its surrounding context region are sampled at the pixel level, and the image visual features are extracted through a visual coding network. Based on the parent-child node relationship and sibling node relationship of candidate interface elements in the document object model tree, a local structure graph is constructed, and structural topological features used to represent hierarchical positions are extracted through a graph coding network. Optical character recognition is performed on the displayed text content in the area where the candidate interface element is located to obtain the text content features; Image visual features, structural topological features, and text content features are input into the feature fusion module. Cross-modal interaction is performed through nonlinear transformation and attention mechanism to generate multimodal feature vectors for each candidate interface element.

[0007] In an optional implementation, a similarity score is calculated between the semantic intent vector and the multimodal feature vectors of each candidate element. Candidate interface elements whose similarity scores exceed a preset similarity matching threshold are identified as target operation elements, including: The similarity score between the semantic intent vector and the multimodal feature vectors of each candidate interface element in high-dimensional space is calculated using a multimodal fusion localization model. Compare the similarity scores of each candidate interface element with a preset similarity matching threshold. When there is a unique candidate interface element whose score exceeds the preset similarity matching threshold, the candidate interface element is identified as the target operation element, and the corresponding simulated click or input operation is triggered.

[0008] In an optional implementation, after performing an interaction action on the target element, if the expected page state is not detected, the similarity matching threshold is adjusted, and candidate interface elements are re-matched based on the updated similarity matching threshold to perform adaptive positioning retry, including: After performing an interactive action on the target element, monitor the changes in the page's Uniform Resource Locator, the appearance status of key text, and whether the document object model structure hash value is consistent with expectations. If any of the above monitoring results do not meet the preset conditions, the operation is deemed to have failed, and the original preset similarity matching threshold is lowered by one level to expand the acceptable matching range. Based on the lowered similarity matching threshold, the element with the highest similarity score among the remaining candidate interface elements is selected as the replacement target, and a retry operation is performed.

[0009] In an optional implementation, the method further includes: If the operation still fails to be completed based on the lowered similarity matching threshold, and no candidate interface element meets the matching conditions, record the semantic intent vector at the time of the operation failure, the multimodal data snapshot of the current user interface, and the multimodal feature vector of the closest matching candidate interface element. The recorded data is packaged to generate new training samples and sent to a remote model training center for incremental learning and updating of the target semantic parsing model and the multimodal fusion localization model.

[0010] In an optional implementation, the method further includes: After receiving multiple new training samples from different execution instances in the model training center, they are batch aggregated according to time windows. The parameters of the existing model are fine-tuned using the aggregated sample set to generate an updated version of the target semantic parsing model and the multimodal fusion localization model; The updated model is distributed to each RPA execution terminal to replace the original model for continuous system optimization.

[0011] Secondly, the present invention provides an element localization and adaptive execution system for robotic process automation, comprising: The semantic conversion module is used to obtain the operation instructions to be executed and convert the operation instructions into high-dimensional semantic intent vectors through the target semantic parsing model; The feature extraction and fusion module is used to identify multiple candidate interface elements in the current user interface to be operated. For each candidate interface element, it extracts the image visual features, structural topology features and text content features of its corresponding region, and fuses the three types of features to generate a multimodal feature vector for each candidate interface element. The similarity matching module is used to calculate the similarity score between the semantic intent vector and the multimodal feature vector of each candidate element, and to identify the candidate interface elements whose similarity scores exceed the preset similarity matching threshold as the target operation elements. The adaptive adjustment module is used to adjust the similarity matching threshold after performing an interactive action on the target element. If the expected page state is not detected, the module will adjust the similarity matching threshold and rematch candidate interface elements based on the updated similarity matching threshold to perform adaptive positioning retry.

[0012] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the element localization and adaptive execution method in robotic process automation of any of the foregoing embodiments.

[0013] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement the element localization and adaptive execution method in robotic process automation according to any of the foregoing embodiments.

[0014] The element localization and adaptive execution method and system for robotic process automation (RPA) provided in this application transforms operation instructions into high-dimensional semantic intent vectors. This allows the system to no longer rely on specific coordinates or DOM paths, but instead perform target recognition based on the semantic function of the operation, effectively avoiding localization failures caused by changes in interface layout or attributes. By fusing three types of features—image vision, structural topology, and text content—multimodal feature vectors are generated, enhancing the comprehensive representation capability of interface elements and improving recognition robustness in scenarios such as UI style changes and A / B testing. Semantic-driven element localization is achieved based on a similarity matching mechanism, ensuring that elements with consistent functions can still be accurately identified even if their appearance or position changes. When an operation fails, adaptive recovery during runtime is achieved by dynamically adjusting the matching threshold and retrying, avoiding process interruption. Overall, this significantly improves the stability and continuous automated execution capability of the RPA system in environments with frequent UI changes. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating an element localization and adaptive execution method in robotic process automation provided in this application embodiment; Figure 2A flowchart illustrating another method for element localization and adaptive execution in robotic process automation provided in this application embodiment; Figure 3 This application provides a structural diagram of an element localization and adaptive execution system in robotic process automation. Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0018] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0020] This application provides a method for element localization and adaptive execution in robotic process automation. See also... Figure 1 As shown, the method mainly includes the following steps: S110: Obtain the operation instruction to be executed, and transform the operation instruction into a high-dimensional semantic intent vector through the target semantic parsing model.

[0021] Operation instructions are abstract functional operation descriptions in RPA scripts, such as "click the submit button" or "enter username in the login form". The target semantic parsing model is a deep learning-based natural language and structured intent mapping model. Its role is to deconstruct such operation instructions into multi-dimensional semantic information containing operation type, target semantics, functional category, and contextual constraints.

[0022] In practice, the system receives abstract instructions from the original RPA process, uses a semantic parsing module pre-trained on a large number of human-computer interaction logs to perform semantic analysis on the instructions, identifies core verbs (such as "click"), key text (such as "submit"), the corresponding functional area (such as "login form"), and the expected control type (such as "button"), and encodes and merges these semantic components into a high-dimensional continuous vector, namely the semantic intent vector.

[0023] S120: In the current user interface to be operated, identify multiple candidate interface elements. For each candidate interface element, extract the image visual features, structural topology features and text content features of its corresponding region, and fuse the three types of features to generate a multimodal feature vector for each candidate interface element.

[0024] Candidate UI elements refer to all interactive or identifiable UI components on the current page, such as buttons, input boxes, and links. Image visual features are used to characterize the appearance attributes of candidate UI elements, including color, shape, size, border style, and surrounding layout; structural topology features are used to characterize the hierarchical relationship of candidate UI elements in the DOM tree, such as the structural paths of parent nodes, child nodes, and sibling nodes; text content features include the text displayed within the candidate UI elements and their OCR recognition results.

[0025] In practice, the process begins by acquiring a full-screen screenshot and the complete DOM structure of the current page. For each candidate interface element, its corresponding image region is cropped, and visual features are extracted using a VisionTransformer model. Simultaneously, its DOM attributes are parsed, and its structural position is encoded using a graph neural network. Furthermore, the visible text content is identified using an OCR engine, and semantic features are extracted using a TextEncoder. Finally, a fusion layer weights and concatenates the three types of feature vectors, performing a non-linear transformation to generate a unified multimodal feature vector, thus creating a high-dimensional semantic fingerprint of the candidate interface element in the digital space.

[0026] S130, calculate the similarity score between the semantic intent vector and the multimodal feature vector of each candidate element, and determine the candidate interface elements whose similarity scores exceed the preset similarity matching threshold as the target operation elements.

[0027] The model utilizes a multimodal fusion localization model (MMF-Locator) to calculate a similarity score representing the functional semantic matching degree between semantic intent and candidate elements. This MMF-Locator is a trained deep matching network that receives the semantic intent vector and the multimodal feature vectors of all candidate elements as input, comparing and calculating their semantic similarity in a high-dimensional space. The model prioritizes functional semantic consistency; for example, when the intent points to "login," it increases the matching score of elements containing the text "login" and presented as buttons. It also further filters based on contextual structure (such as being located within a form). All candidate elements are sorted by score, and the element with the highest score exceeding a preset threshold is selected as the target operation element, triggering a corresponding simulated interaction action, such as clicking or inputting, thereby achieving accurate localization based on semantic understanding.

[0028] S140: After performing an interactive action on the target element, if the expected page state is not detected, the similarity matching threshold is adjusted, and candidate interface elements are re-matched based on the updated similarity matching threshold to perform adaptive positioning retry.

[0029] Expected page states include quantifiable business feedback signals such as page URL redirection, appearance of specific text, completion of new element loading, or changes in DOM structure. In practice, a status monitoring module is activated after an operation is performed to compare the actual response with the expected result in real time. If no expected status change is detected, the operation is considered a failure, and a self-repair mechanism is initiated: first, the original similarity matching threshold is slightly lowered to expand the matching range, allowing suboptimal but semantically similar candidate elements to participate in the matching; then, the S130 matching process is re-executed on the current page to attempt to locate new potential targets. If a match is successful and the operation is completed, the subsequent process continues; if it still fails, a deeper repair strategy is implemented, such as recording failure samples for subsequent model optimization. This mechanism achieves dynamic fault tolerance and rapid recovery capabilities at runtime.

[0030] To facilitate understanding, the following provides a detailed explanation of the element localization and adaptive execution methods in this robotic process automation.

[0031] The above-mentioned acquisition of the operation instruction to be executed, and the transformation of the operation instruction into a high-dimensional semantic intent vector through the target semantic parsing model, may include the following steps 1.1 to 1,3 in specific implementation: Step 1.1: Obtain historical human-computer interaction logs, determine the instruction training set based on the historical human-computer interaction logs, and train the target semantic parsing model through instruction training.

[0032] Historical human-computer interaction logs refer to data collections recorded during actual user operations, containing operational behaviors and corresponding interface states. These logs cover user actions such as clicks, inputs, and page navigation, along with their contextual information. The instruction training set consists of structured samples extracted and labeled from historical human-computer interaction logs, used to train the semantic parsing model.

[0033] In one implementation, typical interaction segments are extracted from a large number of manual operation records accumulated by the enterprise RPA platform. The mapping relationship between the operation intention and the actual functioning element is identified to construct a labeled training dataset. For example, "a user clicks the button with the text 'Login' in the 'Login Form' area" is labeled as the functional intention "Login" and the contextual constraint "Login Form". Subsequently, a deep neural network model (such as a language model based on the Transformer architecture) is used to supervise the learning of the instruction training set, enabling the model to learn to map natural language descriptions or abstract instructions to a structured semantic space, generating a target semantic parsing model with semantic understanding capabilities.

[0034] Step 1.2: Semantically decompose the operation instructions using the target semantic parsing model to extract functional intent information and contextual constraint information.

[0035] Functional intent information refers to the core action purpose reflected in the operation instructions, such as verbal semantics such as "submit", "confirm", and "query"; "contextual constraint information" refers to environmental clues that limit the scope of the operation, such as the functional module to which it belongs ("reimbursement process"), the interface area where it is located ("bottom of pop-up"), or the type of associated control ("button").

[0036] In one example, the received operation instruction (such as "click the submit button") is input into a pre-trained target semantic parsing model. The model identifies key semantic components by performing word segmentation and syntactic analysis on the instruction text: classifying "click" as the operation type, "submit" as the core functional intent, "button" as a visual preference or control category, and "at the end of the form" as the layout context.

[0037] Step 1.3: Encode the functional intent information and context constraint information into a first semantic representation and a second semantic representation in a continuous vector space, respectively, and fuse the first semantic representation and the second semantic representation to generate a high-dimensional semantic intent vector; wherein, the high-dimensional semantic intent vector is used to indicate the action type and target of the operation instruction.

[0038] The first semantic representation is the mathematical representation of functional intent information in a high-dimensional continuous vector space, used to represent the essential function of the operation. The second semantic representation is the vector expression of contextual constraint information, used to enhance the accuracy of localization and scene adaptability.

[0039] In practical implementation, the functional intent (such as "submit") and contextual constraints (such as "buttons in the login form") are mapped into dense vectors through the embedding layer within the target semantic parsing model. Then, feature fusion is performed through methods such as concatenation, weighted summation, or attention mechanisms to generate a unified high-dimensional semantic intent vector. This vector comprehensively expresses the complete semantics of "what operation to perform" and "in what environment to perform," becoming the basis for subsequent multimodal matching queries and achieving semantic-driven element localization independent of physical location.

[0040] The above method trains a semantic parsing model based on historical interaction logs and performs functional and contextual deconstruction and vector fusion on operation instructions. This achieves a precise transformation from abstract instructions to computable semantic representations, improves the RPA system's ability to understand operation intentions, and enables it to accurately identify target elements even when the interface layout changes or element attributes are updated. This significantly enhances the adaptability and robustness of the script.

[0041] Furthermore, for each candidate interface element, the image visual features, structural topological features, and text content features of its corresponding region are extracted, and the three types of features are fused to generate a multimodal feature vector for each candidate interface element. In specific implementation, this may include the following steps 2.1 to 2.4: Step 2.1: Perform pixel-level sampling on the image region corresponding to each candidate interface element and its surrounding context region, and extract the image visual features through a visual coding network.

[0042] The image region refers to the rectangular pixel area occupied by a candidate interface element in a screenshot, while the surrounding context region includes the interface blocks adjacent to the element, used to preserve its layout environment information. Visual encoding networks are deep neural network models used to automatically learn high-level semantic features from images.

[0043] In one implementation, the corresponding image block is first cropped from the full-screen screenshot based on the coordinate information of the DOM element, and then extended outward to include the surrounding visual context. The image block is then input into a visual encoding network (such as VisionTransformer), which extracts visual attributes such as color, shape, texture, border style and spatial layout through multi-layer self-attention mechanism and convolution operation, and outputs a fixed-dimensional vector representation, that is, the image visual features.

[0044] Step 2.2: Based on the parent-child node relationship and sibling node relationship of the candidate interface elements in the document object model tree, construct a local structure graph and extract the structural topology features used to represent the hierarchical position through a graph coding network.

[0045] Parent-child node relationships are used to characterize the containment relationship between a candidate interface element and its direct container or child component, while sibling node relationships are used to characterize the relationship between parallel elements under the same parent node. Graph coding networks refer to graph neural networks (GNNs) used to process data with topological structures.

[0046] In one specific implementation, the complete DOM tree of the current page is first parsed, and for each candidate element, its ancestor path several levels above, its list of child elements below, and its sibling nodes at the same level are extracted to construct a local structural subgraph centered on the element. Then, the graph is input into a graph encoding network, and by performing message passing and aggregation on the connection relationships between nodes, the relative position semantics of each node in the structure are learned, and finally, the structural topology feature vector of the candidate interface element is output.

[0047] Step 2.3: Perform optical character recognition on the displayed text content in the area where the candidate interface element is located to obtain text content features.

[0048] Display text content refers to text information visible to the user, which may originate from HTML text nodes or be embedded in images (such as text drawn on a button). Optical Character Recognition (OCR) is used to extract readable text from images.

[0049] The system uses image regions of candidate elements as a basis, and calls a high-precision OCR engine to perform text detection and recognition on those regions, obtaining the character sequences contained therein and their spatial positions in the image. For standard HTML elements, the system also extracts their innerText or textContent attributes as supplementary information. After word segmentation and cleaning, the resulting text content is input into a text encoder (such as BERT or TextEncoder) to be transformed into a semantic vector, forming text content features. These features not only contain keyword information (such as "submit" and "confirm"), but also represent the functional intent at the language level, improving the accuracy of semantic matching.

[0050] Step 2.4: Input the image visual features, structural topology features, and text content features into the feature fusion module, and generate multimodal feature vectors for each candidate interface element through nonlinear transformation and attention mechanism to perform cross-modal interaction.

[0051] The feature fusion module is a multi-layer neural network structure used to integrate information from different modalities. Nonlinear transformation refers to the complex mapping of features through activation functions, while the attention mechanism allows the model to dynamically focus on the most relevant modalities or feature dimensions.

[0052] In practical implementation, the three independently extracted feature vectors are concatenated or input into the fusion module in parallel. A multi-head attention mechanism is used to calculate the correlation weights between the modalities, enabling visual, structural, and textual information to mutually enhance each other. For example, when the text contains "login" and the visual presentation is a button, the attention mechanism strengthens the joint expression of these two types of features. After multiple rounds of cross-modal interaction and nonlinear transformation, the fusion module outputs a unified, high-dimensional multimodal feature vector, serving as the unique semantic fingerprint of the candidate element in the digital space. This comprehensively integrates its visual appearance features, textual semantic information, and structural positional relationships within the user interface, generating a unified feature representation of the candidate interface element's multi-dimensional and high-dimensional representation.

[0053] The above approach extracts the visual features, structural topology features, and text content features of candidate elements respectively, and uses an attention mechanism to achieve deep cross-modal fusion, thus constructing a richer and more robust element representation method. This effectively improves the element recognition capability in the case of UI redesign, style change, or partial information loss, and significantly enhances the adaptability and positioning stability of the RPA system to complex dynamic interfaces.

[0054] Furthermore, the above calculation of the similarity score between the semantic intent vector and the multimodal feature vectors of each candidate element, and the determination of candidate interface elements whose similarity scores exceed a preset similarity matching threshold as target operation elements, may include the following steps 3.1 and 3.2 in specific implementation: Step 3.1: Calculate the similarity score between the semantic intent vector and the multimodal feature vectors of each candidate interface element in high-dimensional space using a multimodal fusion localization model.

[0055] The Multimodal Fusion Localization Model (MMF-Locator) is a trained deep learning matching network used to determine the consistency between an operation's intent and page elements at the functional semantic level. Specifically, the semantic intent vector is used as the query vector and input in parallel with the multimodal feature vectors generated from all candidate interface elements into the MMF-Locator model. This MMF-Locator model, through an internal cross-attention mechanism or a dual-tower structure, compares the correlation between the semantic intent and the features of each element one by one in a high-dimensional embedding space, outputting a scalar similarity score to characterize the comprehensive matching degree between the two across visual, textual, and structural dimensions. A higher similarity score indicates that the element better matches the functional expectation of the current operation.

[0056] Step 3.2: Compare the similarity scores of each candidate interface element with the preset similarity matching threshold. When there is a unique candidate interface element whose score exceeds the preset similarity matching threshold, the candidate interface element is determined as the target operation element, and the corresponding simulated click or input operation is triggered.

[0057] The preset similarity matching threshold is a safety boundary used to filter low-confidence matches, ensuring that only highly matched elements are selected. A score exceeding the threshold means that only one element on the current page possesses sufficiently high semantic consistency.

[0058] In one implementation, all candidate elements can be sorted by similarity score to filter out a set of elements with a score higher than a threshold. If the set contains only one element, the location is considered successful, its screen coordinates or DOM reference are obtained, and the automation engine is called to execute the corresponding interactive action, such as mouse click or keyboard input. If no element meets the condition or more than one element meets the condition, the subsequent self-repair process is entered to avoid process abnormalities caused by misoperation.

[0059] Furthermore, after performing an interactive action on the target element, if the expected page state is not detected, the similarity matching threshold is adjusted, and candidate interface elements are re-matched based on the updated similarity matching threshold for adaptive positioning retry. In specific implementation, this may include the following steps 4.1 to 4.3: Step 4.1: After performing the interactive action on the target element, monitor the changes in the page's Uniform Resource Locator, the appearance status of key text, and whether the document object model structure hash value is consistent with expectations.

[0060] The expected page state is a predefined operation feedback signal based on business logic. Immediately after a click or input action is completed, the system activates the state monitoring module to collect state data from multiple dimensions in real time: checking if the browser's current URL has redirected to the target path; scanning the page content for the presence of expected key prompt text (such as "submission successful"); and simultaneously calculating the structural hash value of the current page's DOM tree and comparing it with the standard hash value from previous successful executions.

[0061] Step 4.2: If any of the above monitoring results does not meet the preset conditions, the operation is deemed to have failed, and the original preset similarity matching threshold is lowered by one level to expand the acceptable matching range.

[0062] Operation failure indicates that the system failed to observe any expected state response, which may be caused by incorrect element positioning or dynamic UI changes. The failure assessment process is triggered once situations such as URL not redirecting, missing key text, or DOM structure mismatch are detected. At this point, the system does not immediately terminate the script but instead initiates a self-healing mechanism: it appropriately lowers the previously high similarity matching threshold (e.g., from 0.95 to 0.85), allowing more semantically similar but not optimal candidate elements to participate in the matching, thereby improving fault tolerance in cases of UI tweaking or temporary loading anomalies.

[0063] Step 4.3: Based on the lowered similarity matching threshold, select the element with the highest similarity score from the remaining candidate interface elements as the replacement target and perform a retry operation.

[0064] The remaining candidate interface elements refer to the options other than the target that failed the first time. In practice, the system re-searches the similarity scores of all candidate elements, selects the element with the highest score that meets the new threshold requirement as the new target operation element, and executes the same interaction action again. This process enables rapid failover during runtime, allowing the process to resume without manual intervention, thus improving the continuity and stability of the RPA system.

[0065] Furthermore, when the operation still fails to be completed successfully based on the lowered similarity matching threshold, and no candidate interface element meets the matching conditions, the semantic intent vector at the time of the operation failure, the multimodal data snapshot of the current user interface, and the multimodal feature vector of the closest matching candidate interface element are recorded. The recorded data is packaged to generate new training samples and sent to the remote model training center for incremental learning and updating of the target semantic parsing model and the multimodal fusion localization model.

[0066] The aforementioned multimodal data snapshot refers to the complete page information captured at the moment of operation, including full-screen images, DOM structure, and OCR text content. When both matching attempts fail (i.e., no effective match even after lowering the threshold), it is considered a severe localization error. In this case, the system automatically saves the context of this failure: including the original semantic intent vector (representing what the user wants to do), the current page's multimodal snapshot (representing the latest UI state), and the multimodal feature vector of the candidate element with the highest score but not meeting the threshold (representing the result closest to the correct answer). This data constitutes a valuable negative sample or correction sample.

[0067] The new training samples construct positive training instances in supervised learning based on the correspondence between the original user intent and the actual optimal matching response elements. The system encapsulates the above three parts of data into standardized data packets and uploads them to the central model training platform through a secure communication channel. These training samples can be used to enhance the model's understanding of new UI styles, layout changes, or feature migrations, especially in scenarios such as A / B testing and system upgrades, accumulating real-world adaptation experience and providing high-quality feedback for subsequent model iterations.

[0068] Furthermore, after the model training center receives multiple new training samples from different execution instances, it performs batch aggregation according to time windows. The aggregated sample set is used to fine-tune the existing model parameters, generating updated versions of the target semantic parsing model and multimodal fusion localization model. The updated models are then distributed to each RPA execution terminal to replace the original models for continuous system optimization.

[0069] An execution instance is an RPA robot running node deployed in different clients or business processes. The model training center continuously receives failure repair samples from all RPA execution ends across the network, collects and organizes them according to fixed time periods (such as hourly or daily), and generates a batched incremental dataset.

[0070] During fine-tuning, the parameters are optimized a few times using new samples on top of the existing pre-trained model. The training platform employs a transfer learning strategy, using the original model weights as initial values ​​and combining them with new samples for supervised training, focusing on enhancing the model's ability to recognize emerging UI patterns and cross-version control evolution. After training, a new version of the target semantic parsing model and multimodal fusion localization model are generated. The training center pushes the updated model files to all registered RPA clients through a secure channel, completing the local model replacement during off-peak business hours.

[0071] The above aggregation method ensures both the timeliness of data updates and training efficiency, avoiding the computational overhead caused by frequent small-sample updates. After the update, each execution terminal uses the latest model for semantic parsing and element localization when the next task starts, thereby achieving the collaborative evolution of the entire RPA asset system and improving global robustness.

[0072] The above approach, by introducing a high-precision matching mechanism based on a multimodal fusion positioning model, a retry strategy with dynamically adjusted thresholds, a failed sample feedback and incremental learning process, and a centralized model update and distribution mechanism, constructs a closed-loop adaptive and self-healing system. This not only achieves immediate response and fault tolerance to UI changes but also continuously absorbs new knowledge through continuous learning, enabling the RPA system to self-optimize over time. This fundamentally reduces maintenance costs and improves the long-term stability and intelligence level of automated processes.

[0073] See Figure 2 As shown, this application provides a method for element localization and adaptive execution in robotic process automation, which includes the following steps one through five: Step 1: UI State Awareness and Multimodal Feature Acquisition Before performing any operation, the system first performs a comprehensive scan and perception of the current page. This process collects three types of raw data: 1) a pixel-level screenshot of the entire page to capture all visual information; 2) the complete HTML DOM tree and its element attributes and hierarchical relationships to preserve structural context; and 3) all text content and its coordinates extracted from the screenshot using OCR technology. This data is synchronized over time and integrated into a unified atomic snapshot of the page state. This step provides a redundant, multimodal data foundation for subsequent localization, surpassing traditional RPA (which relies solely on the DOM or selectors), ensuring that even if the DOM structure changes, visual and textual information can still support element localization.

[0074] Step 2: Generating the semantic intent vector of the target element The system receives abstract operation instructions (e.g., Click_Submit_Button) from the original RPA script and transforms them into a high-dimensional semantic intent vector using a pre-trained semantic parsing model. This model, trained on large-scale human-computer interaction logs, deconstructs instructions into semantic components such as operation type, core target semantics, function category, contextual constraints, and visual preferences, and encodes and fuses these components. The generated intent vector represents the user's operational intent and is the core driver of subsequent localization, replacing traditional fixed coordinates or static paths and decoupling the intent from the specific UI implementation.

[0075] Step 3: Multimodal fusion representation of page element features For each candidate page element in the snapshot from step one, the system extracts and fuses its multimodal features to generate a unique feature vector. Specifically: visual features (such as color, shape, and layout) are extracted using a VisionTransformer; associated OCR text and DOM attribute text features are extracted using a text encoder; and its structural relationship in the DOM tree is encoded using a graph neural network. Finally, a fusion layer deeply interacts and concatenates the visual, textual, and structural features to generate a unified multimodal representation of each element, thus comprehensively depicting the element's visual appearance, textual meaning, and structural position.

[0076] Step 4: Similarity matching of multimodal fusion localization models The system employs a pre-trained multimodal fusion localization model (MMF-Locator) to perform matching decisions. This model receives the semantic intent vector generated in step two and the feature vectors of all candidate elements generated in step three. It then uses deep learning to calculate a high-dimensional semantic similarity score between the intent and each element's features. The matching process prioritizes semantic consistency and incorporates visual and structural features for refined selection. Finally, the system selects the element with the highest similarity score exceeding a preset threshold (e.g., 0.95) as the target element and performs corresponding automated operations (such as clicking or inputting) on ​​it.

[0077] Step 5: Operation Result Monitoring and Script Self-Repair Mechanism After the operation is executed, the system monitors the page status and business feedback in real time (such as URL changes, expected text appearance, error pop-ups, etc.). If an operation failure is detected, the system automatically initiates a two-stage self-repair process: the first stage (rapid relocation) dynamically relaxes the matching threshold and expands the search range, attempting to retry the operation using suboptimal matching elements; if multiple retries still fail, the system enters the second stage (incremental learning and reconstruction), where the system constructs new training samples from the context information of this failure and the features of the closest new element on the current page, and feeds them back to the central model for incremental learning. This mechanism not only bypasses single failures immediately, but also continuously learns new interface features, enabling continuous optimization of the RPA model and semi-automatic script reconstruction, thereby improving the system's long-term adaptability and robustness.

[0078] Through the coordinated work of the above five steps, this invention constructs an intelligent RPA system that can understand user intent, adapt to interface changes, and autonomously recover from failures.

[0079] In summary, this application constructs an RPA robot script execution method with adaptive and self-healing capabilities by integrating computer vision and deep semantic understanding technologies. The system transforms operation instructions into semantic intent vectors and combines the visual, textual, and DOM structure multimodal features of page elements with a deep matching model to achieve semantic-driven target localization, eliminating reliance on fixed coordinates or selectors. When UI redesign causes traditional localization to fail, the system can automatically initiate a multi-stage relocalization and threshold adjustment mechanism to complete fault recovery within milliseconds; and through incremental learning, it continuously optimizes the model, enabling the RPA script to evolve in response to future changes. Overall, this significantly improves the stability and business continuity of automated processes, greatly reduces maintenance costs, and achieves continuous adaptation of the RPA script to dynamic UI evolution through semantic-driven multimodal matching and incremental model update mechanisms, realizing a closed-loop technology for one-time deployment and long-term stable operation.

[0080] Based on the above method embodiments, this application also provides an element localization and adaptive execution system for robotic process automation, see [link to relevant documentation]. Figure 3 As shown, the system includes the following components: The semantic conversion module 310 is used to obtain the operation instruction to be executed and convert the operation instruction into a high-dimensional semantic intent vector through the target semantic parsing model. The feature extraction and fusion module 320 is used to identify multiple candidate interface elements in the current user interface to be operated. For each candidate interface element, it extracts the image visual features, structural topology features and text content features of its corresponding region, and fuses the three types of features to generate a multimodal feature vector for each candidate interface element. The similarity matching module 330 is used to calculate the similarity score between the semantic intent vector and the multimodal feature vector of each candidate element, and to determine the candidate interface elements whose similarity scores exceed the preset similarity matching threshold as the target operation elements. The adaptive adjustment module 340 is used to adjust the similarity matching threshold and rematch candidate interface elements based on the updated similarity matching threshold after performing an interactive action on the target operation element, if the expected page state is not detected, in order to perform adaptive positioning retry.

[0081] In one feasible implementation, the semantic conversion module 310 is specifically used for: Obtain historical human-computer interaction logs, determine the instruction training set based on the historical human-computer interaction logs, and train the target semantic parsing model through the instructions; The operation instructions are semantically decomposed using a target semantic parsing model to extract functional intent information and contextual constraint information. Functional intent information and contextual constraint information are encoded into a first semantic representation and a second semantic representation in a continuous vector space, respectively. The first semantic representation and the second semantic representation are then fused to generate a high-dimensional semantic intent vector. The high-dimensional semantic intent vector is used to indicate the action type and target of the operation instruction.

[0082] In one feasible implementation, the feature extraction and fusion module 320 is specifically used for: For each candidate interface element, the image region corresponding to the image region and its surrounding context region are sampled at the pixel level, and the image visual features are extracted through a visual coding network. Based on the parent-child node relationship and sibling node relationship of candidate interface elements in the document object model tree, a local structure graph is constructed, and structural topological features used to represent hierarchical positions are extracted through a graph coding network. Optical character recognition is performed on the displayed text content in the area where the candidate interface element is located to obtain the text content features; Image visual features, structural topological features, and text content features are input into the feature fusion module. Cross-modal interaction is performed through nonlinear transformation and attention mechanism to generate multimodal feature vectors for each candidate interface element.

[0083] In one feasible implementation, the similarity matching module 330 is specifically used for: The similarity score between the semantic intent vector and the multimodal feature vectors of each candidate interface element in high-dimensional space is calculated using a multimodal fusion localization model. Compare the similarity scores of each candidate interface element with a preset similarity matching threshold. When there is a unique candidate interface element whose score exceeds the preset similarity matching threshold, the candidate interface element is identified as the target operation element, and the corresponding simulated click or input operation is triggered.

[0084] In one feasible implementation, the adaptive adjustment module 340 is specifically used for: After performing an interactive action on the target element, monitor the changes in the page's Uniform Resource Locator, the appearance status of key text, and whether the document object model structure hash value is consistent with expectations. If any of the above monitoring results do not meet the preset conditions, the operation is deemed to have failed, and the original preset similarity matching threshold is lowered by one level to expand the acceptable matching range. Based on the lowered similarity matching threshold, the element with the highest similarity score among the remaining candidate interface elements is selected as the replacement target, and a retry operation is performed.

[0085] In one feasible implementation, the system further includes an update learning module for: If the operation still fails to be completed based on the lowered similarity matching threshold, and no candidate interface element meets the matching conditions, record the semantic intent vector at the time of the operation failure, the multimodal data snapshot of the current user interface, and the multimodal feature vector of the closest matching candidate interface element. The recorded data is packaged to generate new training samples and sent to a remote model training center for incremental learning and updating of the target semantic parsing model and the multimodal fusion localization model.

[0086] In one feasible implementation, the system further includes a model fine-tuning module for: After receiving multiple new training samples from different execution instances in the model training center, they are batch aggregated according to time windows. The parameters of the existing model are fine-tuned using the aggregated sample set to generate an updated version of the target semantic parsing model and the multimodal fusion localization model; The updated model is distributed to each RPA execution terminal to replace the original model for continuous system optimization.

[0087] The element localization and adaptive execution system in robotic process automation provided in this application has the same implementation principle and technical effects as the aforementioned method embodiments. For the sake of brevity, any parts not mentioned in the embodiments of the element localization and adaptive execution system in robotic process automation can be referred to the corresponding content in the aforementioned embodiments of the element localization and adaptive execution method in robotic process automation.

[0088] This application also provides an electronic device, such as... Figure 4 The diagram shows the structure of the electronic device 100, which includes a processor 41 and a memory 40. The memory 40 stores computer-executable instructions that can be executed by the processor 41. The processor 41 executes the computer-executable instructions to implement any of the above-mentioned element localization and adaptive execution methods in robotic process automation.

[0089] exist Figure 4 In the illustrated embodiment, the electronic device further includes a bus 42 and a communication interface 43, wherein the processor 41, the communication interface 43, and the memory 40 are connected via the bus 42.

[0090] The memory 40 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 42 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0091] Processor 41 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 41 or by software instructions. Processor 41 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory. The processor 41 reads the information in the memory and, in conjunction with its hardware, completes the steps of the element localization and adaptive execution method in the robotic process automation of the aforementioned embodiment.

[0092] This application also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the element localization and adaptive execution method in the above-mentioned robotic process automation. For specific implementation details, please refer to the foregoing method embodiments, which will not be repeated here.

[0093] The computer program product of the element localization and adaptive execution method and system in robotic process automation provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0094] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0095] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] In the description of this application, it should be noted that the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for element localization and adaptive execution in robotic process automation, comprising: Obtain the operation instruction to be executed, and transform the operation instruction into a high-dimensional semantic intent vector through the target semantic parsing model; In the current user interface to be operated, multiple candidate interface elements are identified. For each candidate interface element, the image visual features, structural topology features, and text content features of its corresponding region are extracted. The three types of features are then fused to generate a multimodal feature vector for each candidate interface element. Calculate the similarity score between the semantic intent vector and the multimodal feature vector of each candidate element, and determine the candidate interface elements whose similarity scores exceed a preset similarity matching threshold as target operation elements; After performing an interactive action on the target element, if the expected page state is not detected, the similarity matching threshold is adjusted, and candidate interface elements are re-matched based on the updated similarity matching threshold to perform adaptive positioning retry.

2. The element localization and adaptive execution method in robotic process automation according to claim 1, characterized in that, Obtain the operation instruction to be executed, and transform the operation instruction into a high-dimensional semantic intent vector through a target semantic parsing model, including: Obtain historical human-computer interaction logs, determine the instruction training set based on the historical human-computer interaction logs, and train the target semantic parsing model using the instructions; The operation instructions are semantically decomposed using the target semantic parsing model to extract functional intent information and contextual constraint information. The functional intent information and context constraint information are encoded into a first semantic representation and a second semantic representation in a continuous vector space, respectively. The first semantic representation and the second semantic representation are then fused to generate a high-dimensional semantic intent vector. The high-dimensional semantic intent vector is used to indicate the action type and target of the operation instruction.

3. The element localization and adaptive execution method in robotic process automation according to claim 1, characterized in that, For each candidate interface element, the image visual features, structural topological features, and text content features of its corresponding region are extracted. These three types of features are then fused to generate a multimodal feature vector for each candidate interface element, including: Pixel-level sampling is performed on the image region corresponding to each candidate interface element and its surrounding context region, and visual features of the image are extracted through a visual coding network. Based on the parent-child node relationship and sibling node relationship of the candidate interface elements in the document object model tree, a local structure graph is constructed, and structural topological features for representing hierarchical positions are extracted through a graph coding network. Optical character recognition is performed on the displayed text content in the area where the candidate interface element is located to obtain text content features; The image visual features, the structural topology features, and the text content features are input into the feature fusion module. Cross-modal interaction is performed through nonlinear transformation and attention mechanism to generate multimodal feature vectors for each candidate interface element.

4. The element localization and adaptive execution method in robotic process automation according to claim 3, characterized in that, Calculate the similarity score between the semantic intent vector and the multimodal feature vectors of each candidate element, and determine the candidate interface elements whose similarity scores exceed a preset similarity matching threshold as target operation elements, including: The similarity score between the semantic intent vector and the multimodal feature vectors of each candidate interface element in high-dimensional space is calculated using a multimodal fusion localization model. The similarity scores of each candidate interface element are compared with a preset similarity matching threshold. When there is a unique candidate interface element whose score exceeds the preset similarity matching threshold, the candidate interface element is determined as the target operation element, and the corresponding simulated click or input operation is triggered.

5. The element localization and adaptive execution method in robotic process automation according to claim 4, characterized in that, After performing an interaction action on the target element, if the expected page state is not detected, the similarity matching threshold is adjusted, and candidate interface elements are re-matched based on the updated similarity matching threshold to perform adaptive positioning retry, including: After performing an interactive action on the target operation element, monitor the changes in the page's Uniform Resource Locator, the appearance status of key texts, and whether the document object model structure hash value is consistent with expectations. If any of the above monitoring results do not meet the preset conditions, the operation is deemed to have failed, and the original preset similarity matching threshold is lowered by one level to expand the acceptable matching range. Based on the lowered similarity matching threshold, the element with the highest similarity score among the remaining candidate interface elements is selected as the replacement target, and a retry operation is performed.

6. The element localization and adaptive execution method in robotic process automation according to claim 5, characterized in that, The method further includes: If the operation still fails to be completed based on the lowered similarity matching threshold, and no candidate interface element meets the matching conditions, record the semantic intent vector at the time of the operation failure, the multimodal data snapshot of the current user interface, and the multimodal feature vector of the closest matching candidate interface element. The recorded data is packaged to generate new training samples and sent to a remote model training center for incremental learning and updating of the target semantic parsing model and the multimodal fusion localization model.

7. The element localization and adaptive execution method in robotic process automation according to claim 6, characterized in that, The method further includes: After receiving multiple new training samples from different execution instances in the model training center, they are batch aggregated according to time windows. The parameters of the existing model are fine-tuned using the aggregated sample set to generate an updated version of the target semantic parsing model and the multimodal fusion localization model; The updated model is distributed to each RPA execution terminal to replace the original model for continuous system optimization.

8. An element localization and adaptive execution system for robotic process automation, comprising: The semantic conversion module is used to obtain the operation instructions to be executed and convert the operation instructions into high-dimensional semantic intent vectors through the target semantic parsing model; The feature extraction and fusion module is used to identify multiple candidate interface elements in the current user interface to be operated. For each candidate interface element, it extracts the image visual features, structural topology features and text content features of its corresponding region, and fuses the three types of features to generate a multimodal feature vector for each candidate interface element. The similarity matching module is used to calculate the similarity score between the semantic intent vector and the multimodal feature vector of each candidate element, and to determine the candidate interface elements whose similarity scores exceed a preset similarity matching threshold as target operation elements. The adaptive adjustment module is used to adjust the similarity matching threshold and re-match candidate interface elements based on the updated similarity matching threshold after performing an interactive action on the target operation element, if the expected page state is not detected, in order to perform adaptive positioning retry.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the element localization and adaptive execution method in robotic process automation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the element localization and adaptive execution method in robotic process automation as described in any one of claims 1 to 7.

Images