Ui element positioning method and system for cross-terminal full-scene rpa based on functional semantics

Abstract

This invention discloses a method and system for UI element localization in cross-platform, full-scenario RPA based on functional semantics, belonging to the field of robotic process automation technology. It includes a pre-construction step, an element recording step, and an element playback step. The pre-construction step involves pre-constructing a UI element functional semantic ontology library specific to the RPA business scenario. This ontology library includes a standardized element functional semantic tag system independent of the client side, business context rules corresponding to each semantic tag, and interactive behavior features. By pre-constructing a client-side independent functional semantic ontology library specific to the RPA business scenario, and using business functional semantics as the core, this invention constructs a cross-platform generalized feature vector for target elements, freeing them from the binding limitations of recording client features. After the same business process is recorded on the Web client, it can be directly run on PC desktop clients, mobile mini-programs, domestic terminals, and closed systems without interfaces without modification, effectively solving the problem of high repetitive development costs in cross-platform scenarios.

Description

Technical Field

[0001] This invention belongs to the field of robotic process automation technology, specifically, it relates to a method and system for UI element localization in cross-platform, full-scenario RPA based on functional semantics. Background Technology

[0002] Robotic Process Automation (RPA) is a technology that uses software robots to simulate human computer operations. It can replace manual labor in completing repetitive and rule-based office tasks such as form entry, data verification, process approval, and report export, and is one of the core tools for enterprise digital transformation. UI element localization is the core underlying capability of RPA technology. Its core function is to accurately identify and locate controls such as buttons, input boxes, and drop-down lists in the interface. It is the core foundation for the stable and accurate execution of RPA processes and directly determines the development efficiency, operational stability, and full lifecycle maintenance cost of RPA processes. With the diversified development of enterprise digital systems, the same business function is often launched simultaneously on multiple end-user versions such as web, PC desktop client, mobile app, and domestic terminal. At the same time, there are also a large number of closed systems without interfaces within the enterprise, such as industrial control, confidential intranets, and remote desktop virtual machines. This places extremely high demands on the cross-end adaptability, full-scenario coverage, and dynamic anti-interference capabilities of RPA UI element localization.

[0003] The UI element localization technology of current mainstream RPA products has mainly evolved through four stages: The first stage is the core localization solution based on attribute paths. This solution obtains the hierarchical attribute chain of the target element through the UI automation framework, recording information such as control type, name, class name, automation ID, and hierarchical index. During playback, it matches layer by layer according to the attribute chain, which is the core basic capability of current mainstream RPA tools. The second stage is the auxiliary localization solution based on image recognition. This solution uses computer vision technologies such as template matching and feature point matching to capture screenshots of elements during recording and match corresponding areas on the screen during playback, serving as a fallback solution when attribute paths fail. The first stage addresses the basic positioning needs in scenarios without UI automation interfaces. The second stage is a relative positioning scheme based on anchor points. This scheme selects stable elements in the interface as anchor points, records the relative position and hierarchical relationship between the target element and the anchor points, and locates the anchor points first and then calculates the position of the target element during playback, which improves the positioning stability to a certain extent in scenarios with slight attribute changes. The third stage is a semantic-assisted positioning scheme based on large language models. With the popularization of large language model technology, some RPA tools have introduced LLM to parse interface text information and match target elements through text semantic similarity, which solves some positioning problems in scenarios with changes in interface display text and internationalization switching.

[0004] However, while the aforementioned existing technical solutions can achieve basic UI element positioning functions in specific scenarios, they still have certain shortcomings in actual operation and use:

[0005] All existing positioning solutions heavily rely on the unique characteristics of the recording end, including DOM structure, control attributes, interface layout, and pixel-level visual features. However, the same business function has completely different UI frameworks, control attributes, and layout designs on different platforms such as web, PC clients, mobile, domestic terminals, and mini-programs. Existing technical solutions cannot achieve cross-platform reuse. The same business process needs to be recorded and maintained separately for different platforms, which may increase development costs by 3-5 times. Maintenance costs increase exponentially with the number of platforms. Existing solutions all use static logic that fixes features during recording and performs single-point matching during playback. They only fix features for a single target element and cannot perceive the correlation of the global business scenario. For SPA single-page applications and dynamically rendered interfaces on low-code platforms, automation IDs, control levels, and sibling indexes are randomly generated every time the application is opened. For enterprise applications with 1-2 version iterations per month, control attributes and interface layouts will undergo regular changes. In the above scenarios, the positioning success rate of existing mainstream solutions may be low, which can easily lead to batch interruptions of RPA processes, requiring enterprises to invest a lot of manpower for monitoring and repair.

[0006] Furthermore, existing multimodal fusion solutions all employ a serial stitching mode that prioritizes attributes and uses images as a fallback. Each modality operates independently and without coordination, only switching image matching schemes after attribute matching completely fails, thus failing to achieve deep fusion and mutual verification of multimodal information. Existing semantic assistance solutions only perform general similarity matching of interface text, without considering the topological relationships of business scenarios. When faced with elements with multiple functional labels on the interface (such as multiple "submit" buttons), the mismatch rate may be high. Finally, existing solutions all adopt a passive logic of remediation after location failure. Only when the process has been interrupted and location has failed will alternative strategies be switched and matching parameters adjusted, making it impossible to anticipate interface change risks in advance and avoid business process interruptions. Existing weight adjustment mechanisms are only optimized based on historical failure data and cannot adapt to future application version iterations. This is an ex-post optimization and cannot achieve ex-ante prevention, so the long-term operational stability of the RPA process may not be guaranteed. Summary of the Invention

[0007] To address the aforementioned problems and technical deficiencies, this invention employs the following technical solution: a cross-platform, full-scenario RPA UI element localization method based on functional semantics, comprising a pre-construction step, an element recording step, and an element playback step:

[0008] The pre-construction step involves pre-constructing a UI element functional semantic ontology library specific to RPA business scenarios. The semantic ontology library includes a standardized element functional semantic tag system independent of the client side, business context rules and interaction behavior features corresponding to each semantic tag;

[0009] The element recording steps include:

[0010] S1. Perform full analysis of the target business interface, extract the functional semantic tags of each UI element in the interface based on the functional semantic ontology library, and construct a business scenario semantic topology graph that is independent of the terminal side, with UI elements as nodes and business semantic relationships between elements as edges.

[0011] S2. Based on the functional semantic tags of the target element and its relative position features in the semantic topology map of the business scenario, generate a cross-end generalized feature vector of the target element to complete element recording and persistent storage.

[0012] The element playback step includes:

[0013] S3. Perform full semantic parsing on the target interface of the playback end, construct a real-time semantic topology graph, and perform global topology alignment between the real-time semantic topology graph and the recorded business scenario semantic topology graph to lock the business semantic subgraph to which the target element belongs.

[0014] S4. Within the business semantic subgraph, functional semantic matching is performed based on the cross-end generalized feature vector of the target element to filter out the candidate element set. After multimodal cross-validation, the final positioning result is output to complete the element playback positioning.

[0015] Preferably, in the pre-construction step, the functional semantic tag system is hierarchically divided according to high-frequency RPA business scenarios, including first-level scenario tags, second-level function tags, and third-level element tags. The first-level scenario tags correspond to business scenarios such as form entry, approval process, data query, and report export. The second-level function tags correspond to business operation links within the scenario. The third-level element tags correspond to the specific business functions of UI elements.

[0016] Furthermore, in step S1, when constructing the semantic topology graph of the business scenario, the visual shape invariants and interaction timing features of each UI element are extracted to generate the spatiotemporal feature fingerprints of the elements, and the spatiotemporal feature fingerprints are written into the semantic topology graph of the business scenario as additional attributes of the nodes.

[0017] Preferably, in step S2, when generating the cross-platform generalized feature vector, the business context association features and interaction behavior pattern features of the target element are also integrated, and the final cross-platform generalized feature vector is generated by weighted summation, wherein the weight of the functional semantic tag is not less than 50%.

[0018] Furthermore, in step S3, global topology alignment specifically involves: using a graph structure matching algorithm to calculate the similarity between the real-time semantic topology graph and the recorded business scenario semantic topology graph; completing node matching based on the invariants of business semantic association; locking the business semantic subgraph consistent with the recorded scenario; and filtering irrelevant interface elements.

[0019] Preferably, in step S4, the multimodal cross-validation specifically involves: for each element in the candidate element set, calculating the structural similarity score, visual similarity score, and semantic similarity score respectively, obtaining the final matching score through weighted summation, and selecting the element with the highest matching score that exceeds a preset threshold as the final localization result; for closed scenarios without UI automation interfaces, extracting the functional semantic tags and topological relationships of the playback interface elements through a visual and semantic joint perception model to complete the matching and validation.

[0020] Preferably, after the element playback step is completed, the self-evolution update step is also included: recording the execution result of this element positioning, constructing an interface change risk assessment model based on the version iteration information of the target application and the historical change pattern of interface elements, scoring the risk of the element positioning strategy, generating alternative positioning schemes in advance for high-risk strategies, and incrementally updating the semantic topology map of the business scenario and the cross-terminal generalized feature vector.

[0021] Furthermore, in the self-evolutionary update step, an exponentially weighted moving average algorithm is used to continuously adjust the weight coefficients of functional semantic labels, topological features, and visual features based on historical execution results. The weight of features that are continuously successfully located is increased, while the weight of features that are continuously unsuccessful in location is decreased.

[0022] Furthermore, the playback end includes a web application, a PC desktop client, a mobile terminal, a domestically produced terminal, an interface-less closed system, and a remote desktop virtual machine.

[0023] A cross-platform, full-scenario RPA UI element localization system based on a native semantic system for business functions includes a semantic ontology library construction module, an element recording module, and an element playback module, wherein:

[0024] The semantic ontology library construction module is used to pre-build a UI element function semantic ontology library specific to RPA business scenarios. The semantic ontology library includes a standardized element function semantic tag system that is independent of the client side, business context rules and interaction behavior features corresponding to each semantic tag;

[0025] The element recording module is used to perform full parsing of the target business interface, extract the functional semantic tags of each UI element in the interface based on the functional semantic ontology library, construct a business scenario semantic topology map that is independent of the terminal side, and generate cross-terminal generalized feature vectors of the target elements to complete element recording and storage.

[0026] The element playback module is used to perform full semantic parsing on the target interface of the playback end and construct a real-time semantic topology graph. After completing global topology alignment, it locks the target business semantic subgraph, completes functional semantic matching and multimodal cross-validation based on cross-end generalized feature vectors, and outputs the final element location result.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] (1) This invention pre-builds a terminal-independent functional semantic ontology library for RPA business scenarios. With business function semantics as the core, it constructs cross-terminal generalized feature vectors of target elements, freeing them from the binding restrictions of recording terminal features. After the same business process is recorded on the Web terminal, it can be run directly on PC desktop clients, mobile mini-programs, domestic terminals, and interface-free closed systems without modification, effectively solving the problem of repeated development and exponential increase in maintenance costs in cross-terminal scenarios.

[0029] (2) This invention uses UI elements as nodes and business semantic associations between elements as edges to construct a business scenario semantic topology graph that is independent of the terminal side. During playback, the target business semantic subgraph is locked by global topology alignment first, and then the element is accurately matched within the subgraph. The positioning logic is upgraded from finding a single element to locking the business scenario first and then matching the target element. The positioning success rate is effectively improved.

[0030] (3) This invention deeply integrates four types of information: functional semantics, topological structure, visual features, and interactive behavior. It achieves the synergy of multi-dimensional information in the feature vector generation stage and realizes the mutual verification of multi-dimensional information through multi-modal cross-validation in the matching stage. At the same time, combined with the global constraints of the business semantic topology graph, it effectively solves the problem of mismatch of elements with the same functional label, and the mismatch rate in cross-terminal scenarios is effectively reduced.

[0031] (4) Based on the application version iteration history and the changing patterns of element features, this invention can predict the risk of feature failure in advance and generate alternative positioning strategies in advance; at the same time, it adopts an exponential weighted moving average algorithm to continuously optimize the weight coefficients of each feature based on historical execution data, so as to realize the continuous self-evolution of the positioning strategy and upgrade the adaptation logic from post-failure repair to pre-prediction and seamless adaptation. Thus, it effectively solves the problem of high long-term maintenance costs of RPA processes. Attached Figure Description

[0032] In the attached diagram:

[0033] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0035] Example 1:

[0036] like Figure 1 As shown, the UI element localization method for cross-platform full-scenario RPA based on functional semantics includes a pre-construction step: pre-constructing a UI element functional semantic ontology library specific to RPA business scenarios. In this embodiment, the semantic ontology library is persistently stored using the graph database Neo4j, and includes a standardized element functional semantic tag system independent of the client side, business context rules and interaction behavior features corresponding to each semantic tag.

[0037] Specifically, the functional semantic tagging system is divided into three levels according to high-frequency RPA business scenarios, as follows:

[0038] Primary scenario tags: Corresponding to the core business scenarios of the enterprise. In this example, five high-frequency scenarios are included: expense reimbursement approval, purchase order entry, bank statement reconciliation, report data export, and employee onboarding approval.

[0039] Secondary function tags: These correspond to business operation steps within the scenario. In this embodiment, they include six core steps: login verification, form filling, attachment uploading, approval operation, data query, and file export.

[0040] Level 3 element labels: These correspond to the specific business functions of UI elements. In this embodiment, they include standardized labels such as username input box, password input box, reimbursement amount input box, invoice upload control, submit approval button, reject button, date selection control, add detail row button, and export to Excel button.

[0041] Each third-level element tag is associated with a corresponding business context rule and interaction behavior feature. For example, the business context rule for the "Submit Approval Button" is: located at the bottom of the approval form, its adjacent sibling element is the "Reject Button", and its predecessor element is a form input control; the interaction behavior feature is: clicking it triggers form validation and submission operations, and after successful execution, the interface jumps to the approval list page.

[0042] Element recording steps: In this embodiment, the recording target is the "Submit Approval Button" on the expense reimbursement page of the web-based OA system. The specific steps are as follows:

[0043] S1. Construction of semantic topology graph for business scenarios:

[0044] A full analysis of the target business interface for expense reimbursement in the web-based OA system was performed: First, the complete DOM tree and structural attributes of the elements were obtained using ChromeDevToolsProtocol. At the same time, high-resolution screenshots of the target interface were taken to obtain visual information. Based on a pre-built functional semantic ontology library, functional semantics were extracted from each UI element in the interface using a finely tuned BERT model specific to the business scenario. The corresponding standardized functional semantic tags were obtained, such as "reimbursement amount input box", "invoice upload control", "submit approval button", and "rejection button".

[0045] Using each UI element in the interface as a node and the business semantic relationship between elements as an edge, a business scenario semantic topology graph independent of the client side is constructed. The types of edges include: subordinate relationship (such as reimbursement amount input box → expense reimbursement form), predecessor and successor relationship (such as invoice upload control → submit approval button), and sibling relationship (such as submit approval button → rejection button).

[0046] Simultaneously, visual shape invariants and interaction timing features of each UI element are extracted: the SIFT algorithm is used to extract visual shape invariants such as the outline and corners of the elements, and to extract interaction timing features such as the input timing and click-triggered feedback of the elements, generating a spatiotemporal feature fingerprint for each element, and writing the spatiotemporal feature fingerprint as an additional attribute of the corresponding node into the semantic topology graph of the business scenario.

[0047] S2, Cross-end generalization feature vector generation:

[0048] Based on the functional semantic label of the target element "Submit Approval Button" and its relative position features in the semantic topology graph of the business scenario, and by integrating business context association features and interaction behavior pattern features, a final cross-platform generalized feature vector is generated through weighted summation; the specific formula is as follows:

[0049]

[0050] in, The final cross-end generalized feature vector generated for the target element is the normalized multidimensional feature vector; The functional semantic label feature vector of the target element is generated by matching a pre-built functional semantic ontology library; This is the relative position feature vector of the target element in the semantic topology graph of the business scenario; Associate the feature vector with the business context of the target element; The feature vector of the interaction behavior pattern of the target element; , , , The weight coefficients for the above four types of features are respectively, satisfying the constraints. And mandatory constraints .

[0051] In this embodiment, the specific values ​​for the weights are: , , , The final generated cross-platform generalized feature vector for the "Submit Approval Button" is [0.92, 0.85, 0.78, 0.90], corresponding to the feature normalization scores of the four dimensions. This feature vector, along with the business scenario semantic topology graph, is persistently stored in the RPA process element library, completing the element recording.

[0052] III. Element Playback Steps: This embodiment provides the specific implementation process for three typical playback scenarios. All playback terminals adopt the same semantic matching logic to achieve one-time recording and full-terminal reuse.

[0053] In practical implementation, scenario 1 is a replay of a domestically produced PC client OA system:

[0054] The playback terminal is a domestically developed PC client OA system under the Tongxin UOS operating system. It performs the same expense reimbursement business as the recording terminal (web client), but the terminal type, UI framework, control properties, and interface layout are completely different. The specific steps are as follows:

[0055] S3, Global Topology Alignment and Business Semantic Subgraph Locking:

[0056] A full semantic analysis is performed on the target interface of the domestic client. The WindowsUIAutomation automation engine is used to obtain interface element information and extract functional semantic tags simultaneously to construct a real-time semantic topology graph. The GraphMatchingNetwork (GMN) algorithm is used to calculate the structural similarity between the real-time semantic topology graph and the recorded business scenario semantic topology graph. Based on the invariants of business semantic association (such as the subordinate relationship of the expense reimbursement form, the sibling relationship between the submit approval button and the rejection button), node matching is completed to lock the "expense reimbursement approval" business semantic subgraph consistent with the recorded scenario, and irrelevant elements such as the notification bar, menu bar, and quick function area on the interface are filtered out.

[0057] S4. Functional semantic matching and multimodal cross-validation:

[0058] Within the locked business semantic subgraph, calculate the cosine similarity between the feature vector of each element and the cross-generalized feature vector of the target element, and select the top 5 elements with the highest similarity as the candidate element set; then initiate the multimodal cross-validation process:

[0059] For each element in the candidate element set, calculate the matching score across three dimensions:

[0060] Structural similarity score : Compares the degree of matching between candidate elements and target elements in terms of control type, hierarchical position, and topological relationship, with a value range of 0 to 1;

[0061] Visual similarity score : Take a screenshot of the screen area of ​​the candidate element and match it with the element screenshot saved during recording. Calculate the cosine similarity, with a value range of 0 to 1.

[0062] Semantic similarity score : Calculate the semantic similarity between the functional semantic tags, contextual information, and recorded semantic descriptions of candidate elements, with a value ranging from 0 to 1.

[0063] The final matching score of the candidate elements is obtained by weighted summation, and the specific calculation formula is as follows:

[0064]

[0065] in, The final matching score for the candidate element is given, with a value ranging from 0 to 1. The higher the score, the higher the degree of matching with the target element. , , The weight coefficients for the three types of similarity scores mentioned above are respectively, satisfying the constraints. .

[0066] In this embodiment, the specific values ​​for the weights are: , , ; After calculation, the three scores for candidate element 1 are as follows: , , Substituting the values ​​into the formula, the final matching score is 0.94, which exceeds the preset threshold of 0.7 and is the highest score among all candidate elements. This element is then used as the final location result to complete the element playback and location.

[0067] In practical implementation, scenario 2 involves playback of the mobile OA mini-program:

[0068] The playback end is a WeChat mini-program OA system, which has the same business functions as the recording end web end, but the interface layout, control size, and interaction method are fully adapted to the mobile end. It executes the S3-S4 steps that are exactly the same as in scenario 1. Through global alignment of the business scenario semantic topology map, functional semantic matching, and multimodal cross-validation, the final positioning success rate is 97.1%, and there is no need to re-record the elements.

[0069] In practical implementation, scenario 3 involves the playback of a closed remote desktop system without interfaces:

[0070] The playback endpoint is a financial system within a remote desktop virtual machine without a UI automation interface. This is a closed scenario, making it impossible to obtain the structured attributes of elements.

[0071] Screenshots of remote desktops are processed using a visual-semantic joint perception model. The model adopts a joint architecture of YOLOv8 object detection network and semantic segmentation network to realize interface element detection, functional semantic label extraction, topological relationship construction, and generate a real-time semantic topology map. Subsequently, the global topology alignment, functional semantic matching, and multimodal cross-validation process, which are completely consistent with Scenario 1, are executed to finally complete element localization with a localization success rate of 94.3%.

[0072] IV. Self-evolutionary update steps:

[0073] After the element replay step is completed, the self-evolution update step is executed, and the specific process is as follows:

[0074] Record the execution results of this element location, including: location success or failure status, feature type used, matching time, changes to interface elements, matching score data, and build a strategy execution history database;

[0075] A risk assessment model for interface changes was constructed. Input features included: the version iteration history of the target application, the historical change patterns of element attributes, the general rules of interface changes in similar applications in the same industry, and prior knowledge of the stability of feature types. Each feature on which element positioning depends was assigned a risk score, ranging from 0 to 1, with higher scores indicating higher failure risks. It should be noted that the risk score for the "display text" feature was 0.8 (high risk), the risk score for the "functional semantic label" feature was 0.1 (low risk), and the risk score for the "topological relative position" feature was 0.25 (low risk).

[0076] For high-risk features with a risk score exceeding 0.7, alternative positioning strategies are generated in advance. In this embodiment, for the high risk of the "display text" feature, alternative positioning strategies based on topological relationships and semantic tags are supplemented in advance, and the semantic topology map of the business scenario and the cross-terminal generalized feature vector are updated incrementally in sync.

[0077] An exponentially weighted moving average algorithm is used to continuously adjust the weight coefficients of each feature based on historical execution results. The weight update formula is as follows:

[0078]

[0079] in, These are the feature weight coefficients updated in the current period; The feature weight coefficients of the previous period;

[0080] This is a smoothing coefficient, with a value range of 0 < <1, used to balance the impact of recent execution results and historical data, in this embodiment The value is 0.3; This represents the matching success rate of this feature within the current period, with a value ranging from 0 to 1. The value is 1 when the location is successfully located and 0 when the location fails.

[0081] In addition, corresponding weight adjustment rules are set for continuous execution results, and the specific formula is as follows:

[0082] Consecutive success weight boosting rules:

[0083]

[0084] Rule for reducing weight after consecutive failures:

[0085]

[0086] in, These are the adjusted feature weights; The feature weights before adjustment; This is the weight increase coefficient for a single consecutive success, which is set to 0.1 in this embodiment (i.e., a 10% weight increase for a single consecutive success). In this embodiment, the threshold for triggering weight boosting is the number of consecutive successful location attempts. ≥3; The weight reduction factor for a single consecutive failure is 0.2 in this embodiment (i.e., the weight is reduced by 20% for a single consecutive failure). In this embodiment, the threshold for triggering a weight reduction is the number of consecutive location failures. ≥3. In this embodiment, the weight of a feature that achieves three consecutive successful positioning attempts is increased by 10%; the weight of a feature that fails to locate three times consecutively is decreased by 20%, thus achieving continuous self-optimization of the positioning strategy.

[0087] Example 2:

[0088] A cross-platform, full-scenario RPA UI element localization system based on a native semantic system of business functions, used to execute the localization method in Example 1, includes a semantic ontology library construction module, an element recording module, an element playback module, and a self-evolving update module. The specific implementation methods of each module are as follows:

[0089] Semantic ontology library construction module: Deployed in the backend service of the RPA designer, using the graph database Neo4j as the storage carrier, it is used to execute pre-build steps and complete the construction, update and maintenance of the semantic ontology library for UI elements and functions specific to RPA business scenarios;

[0090] Element Recording Module: A visual element capture component integrated into the RPA designer, including an interface parsing submodule, a topology graph construction submodule, and a feature vector generation submodule. The interface parsing submodule is used to connect to various UI automation engines, obtain interface element information, and extract functional semantic tags. The topology graph construction submodule is used to construct a business scenario semantic topology graph that is independent of the client side. The feature vector generation submodule is used to generate cross-platform generalized feature vectors for target elements.

[0091] Element Replay Module: Integrated into the RPA execution engine, it includes a real-time topology construction submodule, a global topology alignment submodule, and a semantic matching and cross-validation submodule. The real-time topology construction submodule performs full semantic parsing on the replay interface to generate a real-time semantic topology graph. The global topology alignment submodule completes topology graph matching and business semantic subgraph locking. The semantic matching and cross-validation submodule completes functional semantic matching, multimodal cross-validation, and outputs the final location result.

[0092] The self-evolving update module is a background service deployed in the RPA execution engine. It includes an execution record submodule, a risk assessment submodule, and an incremental update submodule. The execution record submodule is used to store historical data of location execution. The risk assessment submodule is used to build a risk assessment model for interface changes and complete feature risk scoring. The incremental update submodule is used to generate alternative location strategies and incrementally update the topology map and feature vectors.

[0093] The above embodiments only illustrate preferred embodiments of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications, improvements, and substitutions without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A UI element localization method for cross-platform, full-scenario RPA based on functional semantics, characterized in that, This includes pre-build steps, element recording steps, and element playback steps: The pre-construction step involves pre-constructing a UI element functional semantic ontology library specific to RPA business scenarios. The semantic ontology library includes a standardized element functional semantic tag system independent of the client side, business context rules and interaction behavior features corresponding to each semantic tag; The element recording steps include: S1. Perform full analysis of the target business interface, extract the functional semantic tags of each UI element in the interface based on the functional semantic ontology library, and construct a business scenario semantic topology graph that is independent of the terminal side, with UI elements as nodes and business semantic relationships between elements as edges. S2. Based on the functional semantic tags of the target element and its relative position features in the semantic topology map of the business scenario, generate a cross-end generalized feature vector of the target element to complete element recording and persistent storage. The element playback step includes: S3. Perform full semantic parsing on the target interface of the playback end, construct a real-time semantic topology graph, and perform global topology alignment between the real-time semantic topology graph and the recorded business scenario semantic topology graph to lock the business semantic subgraph to which the target element belongs. S4. Within the business semantic subgraph, functional semantic matching is performed based on the cross-end generalized feature vector of the target element to filter out the candidate element set. After multimodal cross-validation, the final positioning result is output to complete the element playback positioning.

2. The UI element localization method for cross-platform, full-scenario RPA based on functional semantics according to claim 1, characterized in that, In the pre-construction step, the functional semantic tag system is hierarchically divided according to high-frequency RPA business scenarios, including first-level scenario tags, second-level function tags, and third-level element tags. The first-level scenario tags correspond to business scenarios such as form entry, approval process, data query, and report export. The second-level function tags correspond to business operation links within the scenario. The third-level element tags correspond to the specific business functions of UI elements.

3. The UI element localization method for cross-platform, full-scenario RPA based on functional semantics according to claim 1, characterized in that, In step S1, when constructing the semantic topology graph of the business scenario, the visual shape invariants and interaction timing features of each UI element are extracted to generate the spatiotemporal feature fingerprints of the elements. The spatiotemporal feature fingerprints are then written into the semantic topology graph of the business scenario as additional attributes of the nodes.

4. The UI element localization method for cross-platform, full-scenario RPA based on functional semantics according to claim 1, characterized in that, In step S2, when generating the cross-platform generalized feature vector, the business context association features and interaction behavior pattern features of the target element are also integrated, and the final cross-platform generalized feature vector is generated by weighted summation, wherein the weight of the functional semantic tag is not less than 50%.

5. The UI element localization method for cross-platform, full-scenario RPA based on functional semantics according to claim 1, characterized in that, In step S3, global topology alignment specifically involves: using a graph structure matching algorithm to calculate the similarity between the real-time semantic topology graph and the recorded business scenario semantic topology graph; completing node matching based on the invariants of business semantic association; locking the business semantic subgraph consistent with the recorded scenario; and filtering irrelevant interface elements.

6. The UI element localization method for cross-platform, full-scenario RPA based on functional semantics according to claim 1, characterized in that, In step S4, the multimodal cross-validation specifically involves: for each element in the candidate element set, calculating the structural similarity score, visual similarity score, and semantic similarity score respectively, obtaining the final matching score through weighted summation, and selecting the element with the highest matching score that exceeds a preset threshold as the final localization result; For closed scenarios without UI automation interfaces, a visual and semantic joint perception model is used to extract the functional semantic tags and topological relationships of the playback interface elements to complete matching and verification.

7. The UI element localization method for cross-platform, full-scenario RPA based on functional semantics according to claim 1, characterized in that, After the element playback step is completed, the self-evolution update step is also included: recording the execution result of this element positioning, constructing an interface change risk assessment model based on the version iteration information of the target application and the historical change pattern of interface elements, scoring the risk of the element positioning strategy, generating alternative positioning schemes in advance for high-risk strategies, and incrementally updating the semantic topology map of the business scenario and the cross-terminal generalized feature vector.

8. The UI element localization method for cross-platform, full-scenario RPA based on functional semantics according to claim 1, characterized in that, In the self-evolutionary update step, an exponentially weighted moving average algorithm is used to continuously adjust the weight coefficients of functional semantic labels, topological features, and visual features based on historical execution results. Features that are continuously successfully located are given increased weight, while features that are continuously unsuccessful in location are given decreased weight.

9. The UI element localization method for cross-platform, full-scenario RPA based on functional semantics according to claim 1, characterized in that, The playback terminals include web application terminals, PC desktop clients, mobile terminals, domestically produced terminals, interface-less closed systems, and remote desktop virtual machines.

10. A cross-platform, full-scenario RPA UI element location system based on a native semantic system of business functions, characterized in that, It includes a semantic ontology library construction module, an element recording module, and an element playback module, among which: The semantic ontology library construction module is used to pre-build a UI element function semantic ontology library specific to RPA business scenarios. The semantic ontology library includes a standardized element function semantic tag system that is independent of the client side, business context rules and interaction behavior features corresponding to each semantic tag; The element recording module is used to perform full parsing of the target business interface, extract the functional semantic tags of each UI element in the interface based on the functional semantic ontology library, construct a business scenario semantic topology map that is independent of the terminal side, and generate cross-terminal generalized feature vectors of the target elements to complete element recording and storage. The element playback module is used to perform full semantic parsing on the target interface of the playback end and construct a real-time semantic topology graph. After completing global topology alignment, it locks the target business semantic subgraph, completes functional semantic matching and multimodal cross-validation based on cross-end generalized feature vectors, and outputs the final element location result.

Images