A natural language-based RPA method and electronic device

Abstract

This invention discloses a natural language-based RPA method and electronic device. The method includes a configuration phase and an execution phase. In the configuration phase, a user's natural language description of the target operation steps is received, and attribute information of the target page elements and surrounding neighbor node information is collected to generate and store configuration data. In the execution phase, the Document Object Model (DOM) data of the current page to be operated on is obtained, and the DOM data is cleaned to remove noise. The cleaned DOM data, the natural language description from the configuration data, and the neighbor node information are input into a pre-trained multimodal large language model to generate intermediate-state instructions containing operation type and target location parameters. Based on the intermediate-state instructions, executable control instructions are generated, and an automated executor is controlled to perform the corresponding operation on the current page. This invention lowers the configuration threshold for RPA and improves adaptability to dynamic page changes and execution robustness.

Description

Technical Field

[0001] This invention relates to the field of robotic process automation (RPA) technology, specifically to a natural language-based RPA method and electronic device. Background Technology

[0002] Robotic Process Automation (RPA) is a technology that uses software robots to simulate human operations on computers, automatically executing repetitive and rule-based business processes. With the deepening of enterprise digital transformation, RPA has been widely applied in scenarios such as financial reconciliation, data entry, and report generation. Traditional RPA systems typically employ a "configuration-execution" separation architecture. This means that developers predefine the operation processes and page element positioning methods during the configuration phase and persist these configurations. During the execution phase, the RPA engine reads the configuration and strictly follows the preset instructions to operate the target application's interface elements.

[0003] In web application scenarios, the location of page elements primarily relies on Document Object Model (DOM) selectors, such as CSS selectors or XPath expressions. The DOM is the programming interface for HTML and XML documents, representing them as a tree structure of nodes and objects. Traditional RPA configuration requires technical personnel to deeply analyze the DOM structure of the target webpage, extracting unique attributes (such as ID, Class, Name, etc.) or hierarchical paths to construct a selector string that accurately points to the target element. This static selector-based approach is characterized by its "staticity," meaning the selector string determined during configuration remains unchanged during subsequent execution. The RPA engine directly uses this fixed string to match the DOM nodes of the current page. For example, to click the "Submit" button on a page, the configuration personnel might construct a CSS selector like `div.container>form#loginForm>button.submit-btn` and hardcode it into the script. Regardless of how the page loads or renders subsequently, the execution engine attempts to find the element using this fixed string.

[0004] However, this static approach, which relies on specific structural paths, reveals significant technical flaws in practical applications, primarily in the fragility of page structures and the high cost of maintenance. Modern web applications commonly employ dynamic rendering technology. Dynamic rendering refers to the real-time generation of page content based on user interactions, backend data status, or A / B testing strategies. This means that the DOM structure of the same functional module may differ at different times or from different user perspectives. For example, front-end frameworks (such as React and Vue) may change the nesting relationships of elements, dynamically generated class names, or attribute values ​​when re-rendering components. Once the page structure undergoes even a slight change, carefully debugged static selectors may fail, causing RPA robots to be unable to find target elements, leading to execution interruptions or erroneous operations. To address this issue, enterprises must invest significant manpower in script maintenance and regression testing. Whenever the target system is upgraded or redesigned, it is necessary to re-investigate and correct invalid selectors, which not only increases operational costs but also severely restricts the efficiency of large-scale RPA application.

[0005] To address the aforementioned issues, several improved solutions have emerged in existing technologies. For example, US Patent Publication No. US20240354510A1 discloses a robotic process automation method using generated semantic information. This solution uses generated semantic information to locate and identify UI elements when traditional element location methods fail during RPA script execution. Its core logic is "post-hoc remediation," meaning it first attempts to use traditional CSS / XPath locators; if that fails, it triggers a semantic information generation module to extract textual and structural features of the elements for similarity matching. While this solution improves the success rate to some extent, it has significant technical limitations: First, it is a passively triggered remediation mechanism, requiring a trial of traditional locators each time, increasing performance overhead and execution latency; second, its semantic matching is based on traditional feature-weighted algorithms, lacking an understanding of deep business semantics and exhibiting limited matching accuracy when facing complex page changes; finally, this solution still relies on predefined scripts and initial locators by professionals, failing to fundamentally lower the configuration threshold.

[0006] Another existing technology is the user interface design update automation method disclosed in US Patent Publication No. US11954463B2. This solution is mainly for batch adaptation scenarios after UI version updates. By comparing the old and new UI interfaces, it extracts component features and calculates similarity, thereby mapping the old configuration to the new interface. However, this solution is mainly applicable to offline batch processing mode, that is, a one-time migration is performed after the old and new versions are determined, rather than real-time adaptation of dynamically changing pages during RPA execution. Therefore, it cannot solve the problem of runtime dynamic page changes encountered during RPA execution, such as A / B test variants or personalized customized interfaces. In addition, this solution only focuses on the mapping and configuration migration of UI components, does not involve the generation mechanism of converting business intent into executable operation instructions, and cannot independently complete the closed loop of automated execution of the RPA process.

[0007] With the development of Large Language Model (LLM) technology, some solutions have attempted to introduce it into the RPA field. For example, Chinese Patent Publication No. CN119377360B discloses an AI Agent based on a large language model and a collaborative knowledge graph, and its implementation method. This solution combines LLM and knowledge graph, using domain knowledge retrieved from the knowledge graph to constrain the RPA process generated by LLM, thereby improving the standardization and accuracy of process generation. However, this solution focuses on the macroscopic generation of the complete RPA process, rather than the microscopic instruction generation of each step at runtime. The generated process is relatively fixed before execution, making it difficult to cope with real-time changes in page structure during execution. In addition, this solution heavily relies on a pre-built and maintained knowledge graph, resulting in a complex system architecture, and the updates to the knowledge graph often lag behind page changes, leading to insufficient adaptability when facing frequently iterating web applications. Furthermore, this solution does not involve precise positioning using neighbor element information or instruction caching optimization mechanisms, leaving room for improvement in execution efficiency and positioning accuracy.

[0008] In summary, existing RPA technologies still face the following technical challenges when dealing with dynamically changing web pages: First, they have a high configuration threshold, relying on specialized technical personnel for static selector debugging or complex script orchestration; second, they suffer from poor execution robustness, with static configuration or post-event remediation mechanisms struggling to adapt to real-time dynamic changes in page structure; third, their positioning accuracy is limited, lacking effective context-based mechanisms to resolve ambiguity issues related to elements with the same name; and fourth, their system efficiency is low, lacking caching optimizations for AI-generated instructions, resulting in high response latency and significant computational resource consumption. Therefore, lowering the professional threshold for RPA configuration, enabling intuitive configuration based on natural language, and dynamically generating precise instructions in runtime by real-time sensing of page structure and context information, while simultaneously ensuring execution efficiency, has become a pressing technical challenge in this field. Summary of the Invention

[0009] This invention provides a natural language-based RPA method and electronic device, aiming to solve the problems of high configuration threshold, poor adaptability to page structure changes, and low execution robustness of existing RPA technologies.

[0010] In a first aspect, embodiments of the present invention provide a natural language-based RPA method, including a configuration phase and a runtime phase, wherein: During the configuration phase, the system receives the user's natural language description of the target operation steps, collects the attribute information of the target page elements and the information of surrounding neighboring nodes, generates configuration data, and stores it. During the operational phase, this includes: Obtain the Document Object Model (DOM) data of the current page to be operated on, and clean the DOM data to remove noisy data; The cleaned DOM data, the natural language description in the configuration data, and the neighbor node information are input into a pre-trained multimodal large language model to generate intermediate instructions containing operation type and target location parameters. The intermediate state instructions are used to generate executable control instructions, and the automated actuator is controlled to perform corresponding operations on the current page.

[0011] In some embodiments, collecting the attribute information of the target page element and the information of its surrounding neighboring nodes includes: determining the position of the target element on the page; obtaining the neighboring elements above, to the left, below, and to the right of the target element according to a preset spatial order; extracting the element name and element type of the neighboring elements; and combining the natural language description, the attribute information of the target element, and the information of the neighboring elements arranged in the spatial order to form a configuration instruction.

[0012] In some embodiments, cleaning the DOM data and removing noisy data includes: traversing the DOM tree nodes, removing script nodes, style nodes and invisible nodes; removing empty data nodes and preset noise label nodes; retaining element nodes that are visible and interactive to form a candidate element set.

[0013] In some embodiments, the step of inputting the cleaned DOM data, the natural language description in the configuration data, and the neighbor node information into a pre-trained multimodal large language model to generate an intermediate instruction containing operation type and target position parameters includes: constructing a prompt word, the prompt word including the cleaned DOM data structure, the natural language description of the current step, and the neighbor node information saved in the configuration stage; inputting the prompt word into the multimodal large language model; and receiving the intermediate instruction output by the multimodal large language model, the intermediate instruction including the operation function name and parameters, the parameters including the coordinate position of the target element on the page.

[0014] In some embodiments, before inputting the cleaned DOM data, the natural language description in the configuration data, and the neighbor node information into the pre-trained multimodal large language model, the method further includes: checking whether there is a cached instruction in the instruction cache that matches the current page state and the current step; if there is a matching cached instruction, then directly calling the cached instruction as an intermediate state instruction; if there is no matching cached instruction, then performing the step of inputting the cleaned DOM data, the natural language description in the configuration data, and the neighbor node information into the pre-trained multimodal large language model.

[0015] In some embodiments, checking whether there is a cached instruction in the instruction cache that matches the current page state and the current step includes: obtaining the configuration instruction identifier corresponding to the current step; obtaining the coordinate range of the candidate target element in the current page; determining whether there is a record in the cache with the same configuration instruction identifier and whose cached coordinates fall within the coordinate range; if so, determining that there is a matching cached instruction.

[0016] In some embodiments, generating executable control instructions based on the intermediate state instructions and controlling the automated executor to perform corresponding operations on the current page includes: parsing the operation function name and parameters in the intermediate state instructions; mapping the operation function name to a native operation interface supported by the automated executor; calling the native operation interface to perform click, input, or information retrieval operations on page elements based on the coordinate positions in the parameters; receiving the operation execution result and feeding back the execution result to drive subsequent steps.

[0017] Secondly, embodiments of the present invention provide a method for generating intelligent RPA configuration data, applied to a browser plugin or configuration terminal, comprising: receiving a user's natural language description of a target operation step in a graphical user interface; determining the position of the target page element in the Document Object Model (DOM) tree in response to the user's selection operation of a target page element; collecting information of neighboring elements around the target page element according to a preset spatial order, wherein the information of the neighboring elements includes element name and element type; combining the natural language description, the attribute information of the target page element, and the information of the neighboring elements to generate configuration data; and persistently storing the configuration data so that it can be used for instruction generation in conjunction with real-time DOM data during the runtime phase.

[0018] Thirdly, embodiments of the present invention provide an intelligent RPA execution method, applied to a browser plugin or client executor, comprising: receiving a task start instruction, the task start instruction being associated with pre-stored configuration data, the configuration data including a natural language description of the target operation steps and information of the surrounding neighbor nodes of the target page element; obtaining the Document Object Model (DOM) data of the current page to be operated on, and cleaning the DOM data to remove noise data to obtain a candidate element set; sending the candidate element set, the natural language description, and the surrounding neighbor node information to a server to trigger the server to call a pre-trained multimodal large language model to generate an intermediate state instruction containing operation type and target position parameters; receiving an executable control instruction generated based on the intermediate state instruction returned by the server; parsing the executable control instruction, and performing corresponding operations on the current page according to the target position parameters therein.

[0019] Fourthly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in the first, second, or third aspects.

[0020] This invention lowers the professional threshold for RPA process configuration by introducing natural language descriptions as the configuration entry point, enabling non-technical personnel to easily define automated tasks. In runtime, it dynamically generates precise operation instructions by collecting and cleaning DOM data in real time, combining it with neighbor node context information from the configuration process, and leveraging the semantic understanding and reasoning capabilities of a multimodal large language model. This approach no longer relies on fixed static selectors but makes real-time decisions based on the current semantic structure and spatial relationships of the page, significantly improving the robustness and success rate of RPA execution in scenarios with dynamically changing page structures. Simultaneously, the auxiliary location using neighbor node information effectively resolves the ambiguity problem of elements with the same name, further enhancing the accuracy of instruction generation. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is an overall flowchart of a natural language-based RPA method provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram illustrating the collection of target element and neighbor node information during the configuration phase.

[0024] Figure 3 This is a flowchart illustrating the logic of DOM data cleaning and candidate element set generation.

[0025] Figure 4 This is a schematic diagram illustrating the construction and reasoning of the Prompt for generating intermediate state instructions in a multimodal large language model.

[0026] Figure 5 This is a flowchart of the logical judgment process for the instruction cache matching mechanism.

[0027] Figure 6 It is an execution flowchart of intermediate state instruction parsing and native control interface mapping.

[0028] Figure 7 This is a flowchart of the method for generating intelligent RPA configuration data (configuration side only).

[0029] Figure 8 This is a flowchart of the intelligent RPA execution method (execution end, one side).

[0030] Figure 9 This is an architectural block diagram of a natural language-based RPA system provided in an embodiment of the present invention.

[0031] Figure 10 This is a schematic diagram of the hardware structure of an electronic device. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0033] In one embodiment of the present invention, a natural language-based RPA method is provided. This method aims to address the technical problems of traditional Robotic Process Automation (RPA) when dealing with dynamic web pages, such as high configuration barriers, high maintenance costs, and poor execution robustness. The overall process of this method is as follows: Figure 1 As shown, the system mainly consists of two core parts: the configuration phase and the runtime phase. In the configuration phase, the system focuses on understanding the user's business intent and constructing configuration data rich in contextual information. In the runtime phase, the system focuses on real-time perception of page status, dynamically generating executable instructions using artificial intelligence models, and driving automated executors to complete operations.

[0034] Step S110: During the configuration phase, receive the user's natural language description of the target operation steps, collect the attribute information of the target page elements and the information of surrounding neighboring nodes, generate configuration data and store it.

[0035] In real-world business scenarios, operations personnel or business users typically lack the technical background to write CSS selectors or XPath expressions. To lower the configuration barrier, this embodiment allows users to describe their desired actions using natural language. For example, during the configuration process of an e-commerce order management page, a user can enter "click the query button" or "enter ${order_id} in the order number input box" in the graphical user interface. This natural language description directly maps to the user's business intent, avoiding coupling with technical implementation details.

[0036] However, descriptions based solely on natural language are often ambiguous. For example, a page may contain multiple buttons named "Query" or multiple input boxes with similar structures. To resolve this ambiguity, this embodiment introduces a mechanism for collecting information on surrounding neighboring nodes during the configuration phase. When a user selects a target element in a browser plugin or configuration terminal, the system not only records the target element's attribute information (such as tag name, ID, Class, text content, etc.), but also automatically identifies and collects its surrounding neighboring elements.

[0037] like Figure 2 As shown, the configuration terminal determines the position of the target element 201 in the DOM tree through the Document Object Model (DOM) interface provided by the browser. Subsequently, the system scans the areas above, to the left, below, and to the right of the target element 201 according to a preset spatial order, identifying adjacent neighboring elements 202, 203, 204, and 205. This spatial order can be counter-clockwise (top, left, bottom, right) or any other agreed-upon order; the key is to maintain consistency between the configuration state and the runtime state. For each neighboring element, the system extracts its key identification information, such as the element name (TagName) and element type (Type), and in some cases, a short text fragment.

[0038] The configuration directive is formed by combining natural language descriptions, target element attribute information, and spatially ordered neighbor element information. This directive constitutes the core of the configuration data. For example, the configuration data can be represented as a structured object containing fields such as "description" (value: "Click the query button"), "target_selector" (initial locator of the target element, optional), and "neighbors" (a list of neighbors, such as [{tag: 'div', text: 'Order Search'}, {tag: 'input', placeholder: 'Please enter the order number'}]). This data structure not only preserves business semantics but also embeds the local topological features of the page.

[0039] The generated configuration data is persistently stored in a database or configuration file. During storage, a unique task ID can be assigned to each configuration task, and a step ID can be assigned to each step. This storage method decouples the configuration data from the specific execution environment, facilitating reuse across different times and sessions. It's worth noting that the configuration phase is a one-time or infrequent operation, requiring reconfiguration only when business processes change or the page structure undergoes significant restructuring. This significantly reduces maintenance frequency compared to traditional RPA, which requires manual correction of the Selector for every minor page tweak.

[0040] Step S120: During the running phase, obtain the Document Object Model (DOM) data of the current page to be operated on, and clean the DOM data to remove noisy data.

[0041] When an RPA task is triggered, the system enters the runtime phase. At this time, the page may have undergone multiple renderings, dynamic loading, or A / B test variant switching, and its DOM structure may have changed slightly or even significantly compared to when it was configured. Therefore, the static DOM snapshot at the time of configuration cannot be used directly; instead, the current real-time DOM data must be obtained.

[0042] Intelligent RPA executors (which can be browser plugins, standalone applications, or embedded scripts) fetch the complete or partial DOM tree of the page to be manipulated via browser debugging protocols (such as the Chrome DevTools Protocol, or CDP) or the DOM API. Since the DOM trees of modern web applications are typically very large and contain a lot of irrelevant information, directly inputting them into a large language model can lead to context window overflow, increased inference latency, and distraction. Therefore, this embodiment performs data cleaning operations immediately after obtaining the DOM data.

[0043] like Figure 3As shown, the cleaning process includes traversing the DOM tree nodes and removing script nodes (such as...). <script>）、样式节点（如<style>）以及不可见节点。不可见节点的判断依据可以是CSS样式中的display:none、visibility:hidden或opacity:0等属性，也可以是元素尺寸为0的情况。此外，系统还会剔除空数据节点以及预设的噪声标签节点，如<meta>、<link>、等对业务逻辑无实质贡献的元素。< / script> After cleaning, elements that are visible and interactive are retained, forming a candidate element set. To improve efficiency, the cleaning process can employ breadth-first or depth-first search strategies, combined with heuristic pruning. For example, if a parent node is determined to be invisible or noisy, all its child nodes can be directly removed without further inspection. The final candidate element set is a streamlined DOM subtree or a flattened list of elements, preserving the core structure and semantic information of the page while significantly reducing the amount of data. This cleaning mechanism not only improves the speed of subsequent model inference but also enhances the model's focus on key information, thereby indirectly improving the accuracy of instruction generation.

[0044] Step S130: Input the cleaned DOM data, the natural language description in the configuration data and the neighbor node information into the pre-trained multimodal large language model to generate intermediate state instructions containing operation type and target location parameters.

[0045] This is the core innovative step of this embodiment. The system takes the cleaned DOM data, the natural language description saved in the configuration stage, and the neighbor node information as input, constructs a prompt, and sends it to the pre-trained multimodal large language model (LLM).

[0046] like Figure 4 As shown, the prompt words are constructed following a specific template structure. The System Prompt section defines the model's role as a "smart RPA instruction generation assistant" and specifies that the output format is a JSON object. The User Prompt section contains three main modules: Current page structure information: The cleaned DOM data is presented as a simplified HTML string or JSON tree. To conserve tokens, deeply nested structures can be truncated or summarized, retaining only the levels relevant to the potential target.

[0047] Business intent description: A natural language description derived from configuration data, such as "Click the query button".

[0048] Context constraint information: Neighbor node information from configuration data, such as "There should be an 'Order Search' label and a 'Please enter order number' input box near the target button".

[0049] After receiving these inputs, the multimodal large language model leverages its powerful semantic understanding and reasoning capabilities to search the current DOM structure for elements that best match the natural language description and satisfy the neighbor context constraints. The model not only understands the semantics of the "query button" but can also exclude other irrelevant "query" links or buttons on the page based on neighbor information. For example, if there are two "query" buttons on the page, one in the "order search" area and the other in the "user management" area, the model will accurately locate the former based on the neighbor information for "order search".

[0050] The model output is an intermediate instruction that includes an operation type (func_name) and target location parameters (params). The operation type can be click, type, hover, extract, etc. The target location parameters typically include the element's coordinates (x, y) on the page, or the element's unique identifier (such as a temporary ID confirmed by the model). In some embodiments, the intermediate instruction may also include a confidence score for subsequent quality checks.

[0051] This LLM-based dynamic generation mechanism endows RPA systems with human-like visual and cognitive abilities. It no longer relies on rigid path matching, but instead "understands" and "decides" based on semantics and context. Even if the page layout shifts, as long as the semantic features of the target element and its relative neighbor relationships remain unchanged, the model can generate the correct instructions. This fundamentally solves the page fragility problem of traditional RPA.

[0052] Step S140: Generate executable control instructions based on the intermediate state instructions, and control the automated actuator to perform corresponding operations on the current page.

[0053] The intermediate instructions generated by LLM are typically abstract descriptions, independent of the specific execution engine. To drive the browser or application to perform actual operations, these need to be translated into executable control instructions.

[0054] like Figure 6 As shown, the executor parses the operation function name and parameters in the intermediate instruction. If the parameter contains coordinates, the executor can directly call the underlying mouse event interface (such as MouseEvent) to trigger a click or input action at the specified coordinates. If the parameter contains an element reference, the executor can obtain the element object through the DOM API and call its native methods (such as element.click() or element.value = ...).

[0055] During execution, the executor can also handle variable substitution. For example, "${order_id}" in the natural language description is a variable placeholder. The executor retrieves the actual order number value from the output of the upstream step or the global variable pool and populates it into the input instruction.

[0056] After execution, the executor collects the operation results (such as success status, new page URL, extracted data, etc.) and feeds them back to the server or process controller to drive the execution of subsequent steps. If execution fails (such as elements being obscured or invalid coordinates), the system can trigger a retry mechanism or exception handling process, such as re-invoking the LLM generation instructions or logging errors for manual troubleshooting.

[0057] The embodiment comprising steps S110 to S140 lowers the barrier to entry through natural language configuration, enhances the accuracy of instruction generation through neighbor node context, and achieves dynamic adaptation to page changes through real-time DOM cleaning and LLM inference. This technical architecture enables the RPA system to run stably in complex and dynamically changing web environments, significantly reduces the workload of script maintenance, and improves the success rate and robustness of automated processes.

[0058] In a different embodiment, considering the cost and latency of LLM calls, an instruction caching mechanism can be introduced before step S130. Specifically, the system checks whether there is a cached instruction in the instruction cache that matches the current page state and the current step. If it exists, the cached instruction is directly reused, skipping the LLM inference process; if it does not exist, LLM inference is performed and the cache is updated. This mechanism significantly improves execution efficiency while ensuring intelligence, and is particularly suitable for scenarios with relatively stable page structures and high repetition frequency.

[0059] In step S110, the collection and processing of neighbor node information is a key means to improve the accuracy of instruction generation. In actual engineering implementation, simply recording the attributes of the target element is often insufficient to uniquely identify an element in a complex DOM structure. For example, in an e-commerce list page containing multiple identical product cards, each card has an "Add to Cart" button, and their tag names, class names, and even text content are completely identical. If only these attributes are relied upon, the LLM will be unable to distinguish which specific button the user intends to point to.

[0060] To address this issue, this embodiment employs a neighbor node encoding strategy based on spatial topology relationships. For example... Figure 2As shown, when an operator selects the target element 201 in the configuration interface, the system not only obtains the DOM node object of that element, but also determines its geometric center coordinates by calculating its bounding rectangle in the viewport. and width and height Subsequently, the system extends the search area in four directions—up, down, left, and right—centered on the target element.

[0061] The search region can be defined as a fixed pixel threshold, such as expanding outwards by 50 pixels; or it can be a dynamic region based on a relative proportion, such as expanding the target element's width by 50%. Within the search region, the system traverses all visible DOM nodes and sorts them according to their Euclidean or Manhattan distance from the center of the target element, selecting the closest elements as candidate neighbors.

[0062] To eliminate ambiguity and reduce data volume, this embodiment extracts information from neighboring elements in a structured manner. The extracted information mainly includes the element name (TagName), element type (e.g., button, input, div), and the most easily identifiable text fragment (Text Content). For image elements, their alt or aria-label attributes can also be extracted. These neighboring elements are arranged in a preset spatial order, such as a counter-clockwise order of "top, left, bottom, right". The consistency of this order is crucial because it provides a stable contextual reference system for the LLM.

[0063] When constructing configuration data, neighbor node information is serialized into a compact string format or a JSON array. For example: neighbors: ["top:div.order-header", "left:label.price-tag", "bottom:span.stock-status"] This format preserves spatial relationships while providing semantic clues. During runtime, after receiving such contextual information, the LLM can construct a local "semantic map." Even if the absolute position of the target element changes, as long as its relative topological relationship with its neighbors remains unchanged, the LLM can accurately locate the target by matching this local map.

[0064] Furthermore, considering extreme changes in page layout, this embodiment also introduces a fault-tolerance mechanism for neighbor nodes. If, during runtime, a configured neighbor element is missing from the current page (e.g., the "Inventory Status" label is hidden), the LLM can utilize the remaining neighbor information for weighted matching, or, as a fallback, rely solely on the semantic features of the target element itself and other existing neighbors for reasoning. This flexible matching mechanism further enhances the robustness of the system.

[0065] In step S120, the DOM data cleaning process directly affects the efficiency and quality of subsequent LLM inference. The original DOM tree often contains a lot of noise, such as advertising scripts, tracking code, and hidden auxiliary elements. If all this data is input into the LLM, it will not only consume a lot of token costs, but also introduce interference information, causing the model's attention to be distracted.

[0066] like Figure 3 As shown, the cleaning module employs a multi-layered filtering strategy. The first layer is a tag whitelist / blacklist filtering. The system maintains a list of noisy tags, such as ['script', 'style', 'link', 'meta', 'noscript', 'iframe'], and directly removes these tags and their subtrees. Simultaneously, core tags with business semantics are retained, such as ['div', 'span', 'button', 'input', 'a', 'table', 'ul', 'li'].

[0067] The second layer is visibility filtering. Web pages contain numerous elements used for layout or accessibility but are not visible. The system checks the elements' CSS computed styles and removes elements with `display: none`, `visibility: hidden`, and `opacity: 0`. Furthermore, elements with a size of 0 (offsetWidth or offsetHeight of 0) are also removed unless they have a specific ARIA role (such as for screen readers). This step can significantly reduce the size of the DOM tree, typically reducing the number of nodes by 60%-80%.

[0068] The third layer is interactive filtering. The main purpose of RPA is to simulate user actions, so only interactive elements are the focus. The system checks whether an element has event listeners (such as click, change, submit) or whether it is a native form control (such as...). <input> , <select>,< / select> <textarea>,< / textarea> <button>The `<interactive>` symbol is used to mark interactive elements. For non-interactive container elements (such as...), ... If they contain interactive child elements, their structural skeleton is preserved so that LLM can understand the hierarchy; if they are purely empty shells or decorative elements, they are pruned.

[0069] After the above cleaning process, the resulting candidate element set is a concise and semantically rich DOM subgraph. To facilitate LLM processing, this embodiment serializes this subgraph into a simplified HTML string or JSON structure. During serialization, lengthy attribute values ​​(such as complete base64 image data and long class names) are removed, retaining only key identifying attributes (such as id, name, role, aria-label) and text content. This formatted data structure preserves the page's logical topology while significantly reducing the model's parsing burden.

[0070] Prior to step S130, to optimize performance, this embodiment introduces an instruction caching mechanism, such as... Figure 5 As shown, while LLM's inference process is intelligent, it is time-consuming (typically on the order of seconds) and incurs high API call costs. In real-world RPA execution scenarios, many steps are repeated, or the page structure remains stable for short periods. Therefore, utilizing caching can significantly reduce latency and cost.

[0071] The core of caching lies in how to define "matching". This embodiment adopts a dual verification mechanism based on "configuration identifier + coordinate range".

[0072] First, each configuration step has a unique configuration ID. The system maintains a cache table where the key is the configuration ID and the value is the most recently successfully generated intermediate command and its corresponding page fingerprint.

[0073] A page fingerprint can consist of the hash value of the current URL and the hash value of the main DOM structure. However, in dynamic pages, DOM hashes are highly volatile. Therefore, this embodiment prefers to use coordinate range verification.

[0074] Specifically, when an instruction needs to be generated, the system first looks up the cache based on the configuration ID. If a cached record is found, the system retrieves the coordinates of the target element in the cached instruction. Then, the system repositions the element within the current page (through a simple DOM lookup or lightweight visual matching) and obtains its current coordinates. .

[0075] The system calculates the distance between two coordinates. : in, and These represent the x and y coordinates of the candidate target element on the current page, respectively. and These represent the x and y coordinates of the target element in the cached record, respectively. This represents the square root operation.

[0076] if Less than the preset threshold If the current element still satisfies the neighbor constraints in the configuration (e.g., 10 pixels), then a cache hit is determined. In this case, the system directly returns the intermediate state instruction of the cache, skipping the LLM call.

[0077] if Greater than the threshold If the corresponding element cannot be found on the current page, the cache is deemed invalid. The system executes the complete LLM inference process, generates new instructions, and updates the cache table.

[0078] This caching mechanism reduces the response time of frequently repeated steps from seconds to milliseconds while maintaining accuracy. Especially in scenarios such as list pagination and batch data entry, the cache hit rate is extremely high, thus significantly improving overall execution efficiency.

[0079] It is worth noting that cache invalidation strategies also need to be carefully designed. In addition to coordinate offsets, when a large-scale page reconstruction is detected (such as URL changes or drastic fluctuations in the number of DOM nodes), the system should proactively clear the relevant cache and force re-inference to avoid execution errors caused by cache pollution.

[0080] In step S130, the cleaned DOM data, natural language description, and neighbor node information are input into the pre-trained multimodal large language model. The core of this step lies in the construction strategy of the prompts. The quality of the prompts directly determines the accuracy and stability of LLM inference. This embodiment employs a structured prompt template, aiming to maximize the model's understanding of page semantics and spatial relationships.

[0081] like Figure 4 As shown, the Prompt mainly consists of three parts: System Prompt, Context Data, and Task Objective.

[0082] The system instruction section defines the model's role and behavioral constraints. For example: "You are an intelligent RPA instruction generation engine. Your task is to generate precise operation instructions based on the provided page structure and user intent. The output must be in standard JSON format, containing the func_name and params fields. If the target element is not found, return an error status." Such explicit constraints reduce the probability of the model outputting unstructured text or illusionary content.

[0083] The context data section contains cleaned and serialized DOM information. To accommodate the context window limitations of LLM and improve attention focus, this embodiment uses special serialization encoding for the DOM data. Unlike complete HTML strings, this embodiment employs a simplified tree-like representation. Each node retains only the tag name, key attributes (id, class, role, aria-label), and text content. For deeply nested structures, indentation or bracket levels are used to represent parent-child relationships. For example: <input placeholder="Search..."> < / button> <button> Go< / button> This notation removes redundant style and script information, highlighting the semantics and hierarchical structure of elements.

[0084] The task objective integrates natural language description and neighbor node information. The natural language description provides high-level business intent, such as "click the search button." Neighbor node information provides local spatial constraints, such as "the button is located to the right of the input box, and the input box's placeholder is 'Search...'." By combining these two, Prompt constructs a multi-dimensional localization cue. The model not only needs to know "what to do" (click), but also "where" (a specific location based on semantics and context).

[0085] After receiving the prompt, the multimodal large language model performs internal attention mechanism computation. The model first parses the natural language description, extracting key entities (such as the "search button") and actions (such as "click"). Then, it searches the serialized DOM structure for candidate elements that match these entities. During this process, neighbor node information plays a crucial role in filtering and validation. The model checks whether there are configured neighbor elements around the candidate element and whether their relative positions meet expectations. If multiple candidate elements satisfy semantic matching, the model ranks them according to the confidence of the neighbor matching and selects the element with the highest score as the target.

[0086] The intermediate instructions output by the model are structured JSON objects. For example: json { "func_name": "click", "params": { "selector_type": "coordinate", "x": 120, "y": 350, "element_tag": "button", "element_text": "Go" } } Here, `func_name` specifies the operation type, and `params` contains the specific parameters required for execution. (Coordinates) This is an absolute pixel value relative to the top-left corner of the viewport, providing precise physical positioning for subsequent execution. Meanwhile, retaining element labels and text information allows for secondary validation during execution, ensuring the idempotency and security of the operation.

[0087] In step S140, executable control instructions are generated based on intermediate state instructions, and the automated actuator is controlled to perform corresponding operations on the current page. This process involves a mapping from abstract semantics to specific API calls. For example... Figure 6 As shown, after receiving the intermediate state instruction, the executor first parses the func_name field and maps it to the native operation interface supported by the underlying automation framework (such as Playwright, Puppeteer, Selenium, etc.).

[0088] For example, when `func_name` is "click", the executor calls the browser's mouse event simulation interface. If the intermediate directive includes coordinate parameters, the executor can directly trigger `mousedown` and `mouseup` events at the specified coordinates. However, pure coordinate clicks are risky, such as the element being obscured or scrolling causing coordinate shifts. Therefore, this embodiment preferably adopts a hybrid execution strategy: the executor first attempts to find an element that matches the characteristics described in the intermediate directive (such as tag, text) through the DOM API. If a unique matching element is found, the `.click()` method of that element is called. This method is more stable and can trigger complete event bubbling. If the element cannot be uniquely identified by DOM characteristics, or if the element is in a dynamically loaded state, the executor falls back to the coordinate click mode and combines JavaScript injection techniques to ensure that the click event can penetrate the Shadow DOM or other isolation layers.

[0089] For input operations (type), the executor needs to handle variable substitution. Natural language descriptions may contain variable placeholders, such as "input ${username}". The executor maintains a context variable pool to store the output of upstream steps or global configuration parameters. Before execution, the executor parses the placeholders, retrieves the actual values ​​from the variable pool, and fills them into the input command. Subsequently, the executor calls the element's `.fill()` or `.type()` method to simulate keyboard input. To simulate real user behavior, random delays and keystroke intervals can also be introduced to circumvent front-end risk control detection.

[0090] After execution, the executor collects the results. For query operations, the executor extracts the text content or attribute values ​​of the target element, encapsulates them into structured data, and sends it back to the server or process controller. For navigation operations, the executor monitors changes in the page URL or loading status, and returns a success signal after confirming a successful redirect. If an exception occurs during execution (such as timeout, element not found, or script error), the executor captures the exception information and handles it according to a preset retry strategy. For example, it can wait for a certain period of time to re-fetch the DOM and try again, or trigger the LLM to regenerate instructions to deal with temporary page loading delays or dynamic rendering issues.

[0091] This chain of conversion from natural language to intermediate commands, and then to native control commands, decouples business intent from technical implementation. Operations personnel only need to focus on business logic, without needing to concern themselves with the underlying technical details. Simultaneously, the dynamically generated mechanism allows the system to adapt to minor changes in the page, significantly improving the lifecycle and maintenance efficiency of the RPA process.

[0092] In one alternative embodiment, considering the inference cost of multimodal large language models, distillation techniques can be employed to transfer knowledge from large general-purpose models to lightweight, specialized models. These specialized models are fine-tuned for RPA scenarios and can run on local devices or edge servers, further reducing latency and data transmission costs. Furthermore, reinforcement learning mechanisms can be introduced to automatically adjust the prompt policy or model parameters based on feedback from execution results, enabling the system to self-optimize and evolve.

[0093] In another embodiment of the present invention, a method for generating intelligent RPA configuration data is provided. This method is mainly applied to browser plugins or configuration terminal sides, focusing on how to efficiently and accurately capture user intent and construct configuration data structures rich in contextual information. For example... Figure 7 As shown, the execution subject of this method can be an extension running in the user's browser or a standalone desktop configuration client.

[0094] Step S210: Receive the user's natural language description of the target operation steps in the graphical user interface.

[0095] The configuration process typically begins with the user activating configuration mode on the target web page. At this point, the browser plugin overlays a semi-transparent interactive layer at the top of the page, providing a floating configuration panel. The user enters a natural language description in the panel's text input boxes, such as "click the submit order button." To enhance the user experience, the system can provide intelligent auto-completion, suggesting verbs based on common operation types on the current page (such as click, input, selection) or nouns based on prominent elements on the page.

[0096] In addition to text input, the system also supports speech-to-text or multimodal input. For example, users can speak commands through a microphone, and the system uses local or cloud-based speech recognition services to convert them into text. This multimodal input method further lowers the configuration barrier, allowing even users who are not proficient in keyboards to easily participate in process design.

[0097] Step S220: In response to the user's selection operation on the target page element, determine the position of the target page element in the Document Object Model (DOM) tree.

[0098] After entering a natural language description, the user needs to move the mouse over the page and select the corresponding target element. As the mouse moves across the page, the plugin listens for the mouseover event in real time and obtains the DOM node of the currently hovering element through event bubbling. To help the user confirm their selection, the plugin draws a highlighted border around the current element and displays a label next to it, showing brief information about the element (such as label name, ID, class name, or the first few characters of text).

[0099] When the user clicks the left mouse button to confirm the selection, the system locks the current target element. At this point, the system obtains a reference to the element through the browser's DOM API (such as document.elementFromPoint or directly using the target property in the event object). Subsequently, the system traverses the DOM tree, recording the path of the element from the root node to the current node, as well as its index position among its sibling nodes. This information constitutes the basic positioning characteristics of the target element.

[0100] Step S230: Collect information about neighboring elements around the target page element according to a preset spatial order. The information of the neighboring elements includes the element name and element type.

[0101] This is one of the core steps in this embodiment. To construct robust context information, the system needs to collect neighboring nodes around the target element. For example... Figure 2 As shown, the system first calculates the bounding client rectangle of the target element and obtains the coordinates of its top-left corner. and the coordinates of the bottom right corner .

[0102] Next, the system defines four search areas: Upper area: and exist Within the range; Left side area: and exist Within the range; Below area: and exist Within the range; Right side area: and exist Within the range.

[0103] in, It is an extended pixel value, such as 20 pixels, used to capture immediately adjacent related elements.

[0104] Within each search area, the system iterates through all visible DOM elements, calculating their closest distances to the target element's boundary. It selects the 1-2 closest elements as neighbors in that direction. For each selected neighbor element, the system extracts its element name (TagName, such as div, span, label) and element type (Type, such as button, text). If the neighbor element contains significant text content (between 5 and 20 characters in length), it is also extracted.

[0105] To improve data compactness, the system can hash or abbreviate neighbor information. For example, ... <label class="form-label"> Order ID:< / label> This is simplified to {tag: 'label', text: 'OrderID:'}. Ultimately, the system arranges the neighbor information from these four directions in the order of "top, left, bottom, right," forming an ordered neighbor list.

[0106] Step S240: Combine the natural language description, the attribute information of the target page element, and the information of the neighboring elements to generate configuration data.

[0107] The system encapsulates the collected information into a standard configuration object. This object typically contains the following fields: step_id: An automatically generated unique step identifier; description: A natural language description entered by the user; target_info: A snapshot of the target element's attributes, including tag, id, class, text, etc.; neighbors: An ordered list of neighbors, containing tags, types, text, etc. of neighbors in each direction; timestamp: Configures the generated timestamp; page_url_pattern: The URL matching rule for the current page, used to limit the range of pages to which this configuration applies.

[0108] This structured configuration data not only preserves the user's business intent but also embeds rich page topology features. Compared to traditional CSS Selectors, this data format is more tolerant of minor changes in page structure. For example, even if the class name of the target element changes due to changes in front-end build tools, the system can still accurately identify it as long as its semantic tags and relationships with surrounding neighbors remain unchanged.

[0109] Step S250: Persist in storing the configuration data so that it can be combined with real-time DOM data for instruction generation during the runtime phase.

[0110] The generated configuration data is sent to a backend server or stored in local storage (such as IndexedDB). Before saving, the system can perform a pre-validation to check the integrity of the configuration data. For example, it verifies that the natural language description is not empty, the target element exists, and the neighbor information in at least one direction is valid. If the validation fails, the system prompts the user to reselect or supplement the information.

[0111] Once persisted to storage, this configuration data becomes part of the RPA process. Users can then add subsequent steps to create a complete business process diagram. This configuration method is intuitive and flexible, significantly shortening the development cycle of RPA processes.

[0112] In another embodiment of the present invention, an intelligent RPA execution method is provided. This method is mainly applied to browser plugins or client-side executors, focusing on how to dynamically perceive page states at runtime and collaborate with the server to generate and execute instructions. For example... Figure 8 As shown, the execution entity of this method is a lightweight executor deployed in the user's environment.

[0113] Step S310: Receive a task start instruction, which is associated with pre-stored configuration data, including a natural language description of the target operation steps and information on the surrounding neighbor nodes of the target page element.

[0114] When a user triggers an RPA task, the executor loads the corresponding configuration data from the server or local cache. This configuration data includes the previously defined natural language description and neighbor information. The executor parses this data and prepares to enter the execution state.

[0115] Step S320: Obtain the Document Object Model (DOM) data of the current page to be operated on, and clean the DOM data to remove noise data and obtain a set of candidate elements.

[0116] The executor captures the DOM tree of the current page in real time via CDP or the DOM API. Because the execution environment may differ from the configuration environment (e.g., different screen resolutions, browser versions, or dynamic content loading), the current DOM structure may differ from that configured. Therefore, the executor must perform cleaning logic similar to, but more rigorously than, that configured on the configuration side.

[0117] The cleaning process includes removing invisible elements, script-style nodes, and irrelevant decorative elements. Specifically, the executor prioritizes elements within the visible area based on the current viewport size, as RPA operations typically target visible elements. The cleaned set of candidate elements is serialized into a compact format, ready to be sent to the server or local inference engine.

[0118] Step S330: Send the candidate element set, the natural language description, and the surrounding neighbor node information to the server to trigger the server to call the pre-trained multimodal large language model to generate an intermediate state instruction containing operation type and target location parameters.

[0119] The executor constructs a request packet containing the cleaned DOM fragments, natural language descriptions from the configuration, and neighbor information. This request packet is sent to the intelligent RPA server via WebSocket or HTTP protocol. Upon receiving the request, the server, following the logic of step S130 above, invokes the multimodal LLM for inference to generate intermediate instructions.

[0120] In a variant embodiment, if the executor has sufficient computing resources (such as a high-performance PC or a small locally deployed LLM), the inference process can also be completed locally. In this case, the executor directly loads the pre-trained model weights and performs local inference, thereby avoiding network latency and data privacy issues. Regardless of whether inference occurs in the cloud or locally, the core logic remains the same: dynamically generating instructions based on real-time DOM and context information.

[0121] Step S340: Receive the executable control instructions generated based on the intermediate state instructions returned by the server.

[0122] The executor receives intermediate instructions returned by the server. If the instruction contains an error flag (such as "ElementNot Found"), the executor can trigger a retry mechanism, such as waiting a few seconds to re-fetch the DOM and request again, or reporting the exception for manual intervention. If the instruction is valid, the executor proceeds to the next step.

[0123] Step S350: Parse the executable control instruction and perform corresponding operations on the current page according to the target position parameter therein.

[0124] The executor parses the `func_name` and `params` in the instruction. For a click operation, the executor first attempts to find the corresponding element through a DOM lookup. If found, it calls the native click method; otherwise, it uses the coordinates in the instruction. Simulates mouse clicks. For input operations, the executor locates the target input box, clears the existing content, and inputs the specified text character by character, simulating the effect of real typing.

[0125] Once execution is complete, the executor feeds back the results to the process controller, driving the execution of the next step. This client-server collaborative architecture leverages the powerful computing capabilities of the cloud while maintaining the lightweight and flexible nature of the client, making it suitable for large-scale distributed RPA deployment scenarios.

[0126] In one embodiment of the present invention, a natural language-based RPA system is provided. This system aims to decouple configuration, inference, and execution through a modular architecture design, thereby supporting high-concurrency, low-latency automated task processing. Figure 9 As shown, the system mainly includes a configuration module 301, a server module 302, and an execution module 303. These three modules interact with each other via a network to complete a closed loop from business intent to physical operation.

[0127] The configuration module 301 typically exists as a browser plugin or a standalone desktop application, primarily targeting business operations personnel. Its core functions include intent capture, context acquisition, and configuration management. When a user interacts with a target page, the configuration module 301 listens for mouse and keyboard events, capturing in real-time information about the selected element and its surrounding neighboring nodes. Simultaneously, it provides a natural text input interface, allowing users to describe their operational intent using everyday language. The configuration module 301 encapsulates this multimodal information (text + DOM structure + spatial relationships) into a standard configuration data package and uploads it to the server module 302 for persistent storage via HTTPS. To enhance user experience, the configuration module 301 also includes a visual editor, allowing users to sort, modify, or delete generated configuration steps to create a complete business process diagram.

[0128] Server-side module 302 is the system's intelligent hub, deployed in a cloud server cluster or private data center. It is primarily responsible for the storage and management of configuration data, the scheduling and inference of large language models, and the management of instruction caches. Server-side module 302 contains several sub-modules: Configuration storage submodule 3021: Used to store and manage all user-uploaded configuration data. It uses a distributed database (such as MongoDB or MySQL) to store structured configuration objects and supports fast retrieval by task ID, user ID, or page URL.

[0129] Instruction Generation Submodule 3022: This is the core computing unit of the system. When it receives an inference request from the execution end, this submodule is responsible for constructing a Prompt and calling the pre-trained multimodal large language model API. To optimize cost and performance, instruction generation submodule 3022 integrates cache management logic. Before calling LLM, it first queries a high-speed cache cluster such as Redis to check for the existence of historical instructions with high hit rates and that have not expired. If the cache is hit, the result is returned directly; otherwise, the expensive LLM inference request is initiated, and the newly generated instructions are written to the cache.

[0130] Model Adaptation Submodule 3023: Given the existence of various large language models on the market (such as GPT-4, Claude, Llama, etc.), this submodule is responsible for shielding the differences in APIs between different models and providing a unified calling interface. It also supports hot-swapping and canary releases of models, so that when a model service becomes unstable, it can automatically downgrade to a backup model, ensuring high system availability.

[0131] The execution module 303 runs in the user's local environment and can be a browser plugin, an Electron application, or a standalone agent program. Its main responsibilities are to monitor page status in real time, communicate with the server, and execute specific automated operations. The execution module 303 includes the following key components: DOM Collector 3031: Responsible for real-time crawling of the DOM tree of the current page using underlying libraries such as Chrome DevTools Protocol (CDP) or Puppeteer / Playwright. To improve efficiency, DOM Collector 3031 incorporates the aforementioned cleaning logic, extracting only visible and interactive element nodes and serializing them into a compact JSON format.

[0132] Communication agent 3032: Responsible for managing bidirectional communication with server module 302. It uses the WebSocket protocol to maintain a long connection, enabling low-latency command issuance and status reporting. In case of network instability, communication agent 3032 has the ability to reconnect after disconnection and local queue buffering to ensure that commands are not lost.

[0133] Instruction Executor 3033: Receives intermediate instructions from the server and converts them into native events recognizable by the browser or operating system. Instruction Executor 3033 implements complex mapping logic, capable of handling various action types such as coordinate clicks, DOM element manipulation, and keyboard input. Furthermore, it is responsible for post-execution result verification and exception handling, such as detecting whether the page redirection was successful or whether an element is obscured.

[0134] In actual operation, the interaction sequence between the modules is as follows: First, the configuration module 301 sends the user-configured process data to the configuration storage submodule 3021 of the server module 302.

[0135] Secondly, when the task is triggered, the DOM collector 3031 of the execution module 303 obtains the cleaned DOM data of the current page and sends it to the server module 302 through the communication proxy 3032.

[0136] Next, the instruction generation submodule 3022 of the server module 302 combines the configuration data and real-time DOM data, performs a cache check, calls the multimodal LLM to generate intermediate state instructions, and returns them to the execution module 303.

[0137] Finally, the instruction executor 3033 of the execution module 303 parses the instruction, drives the browser to complete the operation, and feeds back the execution result to the server module 302 so as to record logs or drive subsequent steps.

[0138] This distributed system architecture offers significant advantages. First, it offloads computationally intensive LLM inference tasks to the cloud, allowing lightweight clients to enjoy intelligent, automated services. Second, the separation of configuration and execution makes process maintenance more flexible; operators can update configurations centrally in the cloud, with all terminals taking effect immediately. Finally, the modular design facilitates system expansion and maintenance; for example, LLM models can be upgraded independently without affecting the stable operation of clients.

[0139] In one different embodiment, to protect user privacy, the DOM data of sensitive pages can be locally anonymized in the execution module 303 before being sent to the server. For example, all personally identifiable information (PII) can be replaced with placeholders, or only the hash fingerprint of the DOM structure can be sent instead of the complete content. The server performs inference based on the anonymized data, and the generated instructions are then sent back to the local machine for execution. This privacy protection mechanism enables the system to be applied in industries with extremely high data security requirements, such as finance and healthcare.

[0140] In another embodiment of the invention, an electronic device is provided. For example... Figure 10As shown, the electronic device 400 includes a memory 401, a processor 402, and a computer program 403 stored in the memory 401 and capable of running on the processor 402.

[0141] The memory 401 can be high-speed random access memory (RAM) or non-volatile memory, such as disk storage, flash memory, or solid-state drive (SSD). The memory 401 is used to store the operating system, applications, and program code and data related to the RPA method described in this embodiment. For example, it can store configuration data, cached instructions, DOM snapshots, and weight files of the multimodal large language model (if deployed locally).

[0142] Processor 402 can be a central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), or other application-specific integrated circuit (ASIC). Processor 402 is responsible for executing the computer program 403 in memory 401, thereby implementing the various steps of the natural language-based RPA method described above. In a multi-core processor architecture, different cores can process tasks such as DOM cleaning, network communication, and instruction parsing in parallel to improve the overall throughput of the system.

[0143] Communication interface 404 is used for data interaction with other devices or networks. For example, it can connect to the Internet via an Ethernet interface or Wi-Fi module to access a server module in the cloud; or connect to peripherals via Bluetooth or USB interface.

[0144] The display screen 405 and the input device 406 (such as a keyboard, mouse, or touch screen) constitute the human-computer interaction interface, allowing users to view the configuration panel, input natural language commands, and monitor the execution status of RPA tasks.

[0145] The electronic device 400 can be a personal computer, laptop, server, tablet computer, or smartphone. When the computer program 403 is executed by the processor 402, the electronic device 400 can perform functions such as configuration data generation, dynamic instruction reasoning, and automated execution as described above. This hardware-level implementation makes the technical solution not limited to the software level, but forms a complete hardware and software integrated solution, enhancing the substantive nature and practicality of patent protection.

[0146] The technical solutions constituted by the above embodiments can produce significant technical effects in practical applications. These effects are not arbitrary assertions, but rather natural deductions based on the system architecture and data processing mechanisms.

[0147] First, this solution significantly reduces the configuration threshold and maintenance cost of RPA processes. Traditional RPA relies on static selectors; once the page DOM structure undergoes even minor changes (such as class name renaming or hierarchy adjustment), the script becomes ineffective, requiring professional re-debugging. In this embodiment, however, the "neighbor node information" collected during the configuration phase provides stable local topological constraints for the LLM. Even if the global path of the target element changes, as long as its relative spatial relationship with surrounding key identifying elements (such as labels and input boxes) remains unchanged, the LLM can still accurately locate it through semantic reasoning. This mechanism shifts the maintenance focus from "correcting code" to "updating semantic descriptions," enabling even non-technical operations personnel to handle maintenance. In actual testing, for regular front-end iterations, the maintenance-free cycle of this solution is more than three times longer than traditional solutions.

[0148] Secondly, this solution enhances the robustness and adaptability of automated execution. By acquiring and cleaning DOM data in real-time during runtime, the system always makes decisions based on the "current truth" of the page, rather than relying on outdated static snapshots. The DOM cleaning mechanism eliminates a large amount of noise interference, allowing LLM to focus on core business elements and reducing misjudgments caused by pop-up ads and hidden layouts. Simultaneously, the instruction caching mechanism solves the problem of high inference latency in LLM while maintaining intelligence. For highly repetitive steps with stable page structures, cache hits keep response times within milliseconds, meeting the stringent timeliness requirements of high-frequency transactions or batch processing scenarios.

[0149] Furthermore, this solution boasts excellent scalability and compatibility. Thanks to its standardized intermediate command format, the execution end can be adapted to various underlying automation frameworks (such as Selenium, Playwright, Appium, etc.), extending beyond web pages to mobile apps and desktop applications. The introduction of a multimodal large language model endows the system with a certain degree of "common sense reasoning" capability, enabling it to handle ambiguous scenarios that traditional rule engines cannot cover, such as identifying occluded but partially visible elements or understanding implicit business logic relationships.

[0150] In a different application scenario, this solution can be applied to the automatic reporting of financial reports. Finance personnel only need to describe in natural language that "data will be read from Excel and entered into the online banking system," and the configuration end collects neighbor node information from the online banking login interface. During runtime, even if the bank system undergoes a UI redesign, as long as the relative positions and semantics of the login box and buttons remain unchanged, the executor can automatically adapt and complete the reporting, without requiring IT intervention to modify the script.

[0151] In another application scenario, this solution can be used for data scraping and monitoring in the e-commerce industry. Operations personnel configure the system to "get product prices and inventory," and the system records the characteristics of the neighbors surrounding the price tags. When e-commerce platforms adjust their page layout to prevent web scraping, this solution can dynamically adapt to the new DOM structure, continuously and stably acquiring data, while traditional web scraping based on fixed XPath is prone to failure.

[0152] It should be noted that although the above embodiments mainly describe RPA scenarios based on web pages, the core idea of ​​this technical solution is also applicable to other graphical user interface (GUI) automation scenarios. For example, in desktop application automation, the DOM tree can be replaced with a UI Automation tree or Accessibility Tree, HTML elements can be replaced with window controls, and neighbor nodes can be defined as adjacent objects of the controls in the screen coordinate system. Multimodal large language models can also receive this structured data for inference and generate mouse and keyboard event commands. This cross-platform versatility further broadens the scope of protection and application value of this patent.

[0153] In summary, this invention constructs an intelligent and adaptive RPA method by introducing techniques such as natural language configuration, neighbor node context enhancement, real-time DOM cleaning, and dynamic reasoning using a multimodal large language model. This method effectively solves the technical problems of high configuration thresholds, poor page adaptability, and high maintenance costs in existing technologies, achieving a technological leap from "static script execution" to "dynamic semantic understanding," demonstrating significant progress and practical value.

[0154] It should be understood that the method steps in the foregoing embodiments can be implemented by program instructions controlling related hardware, or by dedicated circuits, programmable logic devices, or a combination thereof. Correspondingly, the systems, devices, modules, units, or components in the foregoing embodiments can be implemented in software, hardware, or a combination of both. The division of modules, units, or components is merely a logical division for the purpose of illustrating the technical solution; in actual implementation, they can be combined, split, or integrated as needed.

[0155] In one embodiment, the electronic device may include a processor, a memory, and a communication interface, wherein the memory is used to store program instructions, and the processor is used to call and execute the program instructions to implement all or part of the steps in the foregoing method embodiments. The electronic device may be a server, a terminal device, an edge computing node, a cloud computing device, or other device with data processing capabilities.

[0156] In one embodiment, this application may also be implemented in the form of a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement all or part of the steps in the foregoing method embodiments. The computer-readable storage medium may be a read-only memory, random access memory, flash memory, hard disk, solid-state drive, optical disk, or other non-transitory storage medium.

[0157] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0158] Furthermore, the terms "including," "comprising," and "having" used in the specification are all non-exclusive inclusions; the terms "first," "second," etc., are only used to distinguish technical features and do not indicate limitations on order, quantity, or importance. The execution order of each step in the method embodiments is also not absolutely limited. Without departing from the technical concept of this application, the steps can be adjusted in order, executed in parallel, combined, or split for execution.

Claims

1. A natural language-based RPA method, comprising a configuration phase and a runtime phase, wherein: During the configuration phase, the system receives the user's natural language description of the target operation steps, collects the attribute information of the target page elements and the information of surrounding neighboring nodes, generates configuration data, and stores it. During the operational phase, this includes: Obtain the Document Object Model (DOM) data of the current page to be operated on, and clean the DOM data to remove noisy data; The cleaned DOM data, the natural language description in the configuration data, and the neighbor node information are input into a pre-trained multimodal large language model to generate intermediate instructions containing operation type and target location parameters. The intermediate state instructions are used to generate executable control instructions, and the automated actuator is controlled to perform corresponding operations on the current page.

2. The natural language-based RPA method as described in claim 1, wherein, The collection of attribute information of the target page element and information of surrounding neighboring nodes includes: Determine the position of the target element on the page; According to the preset spatial order, obtain the neighboring elements above, to the left, below and to the right of the target element; Extract the element name and element type of the neighboring element; The natural language description, the attribute information of the target element, and the information of the neighboring elements arranged in the spatial order are combined to form a configuration instruction.

3. The natural language-based RPA method as described in claim 1, wherein, The step of cleaning the DOM data and removing noisy data includes: Traverse the DOM tree nodes, removing script nodes, style nodes, and invisible nodes; Remove empty data nodes and preset noise label nodes; Retain the visible and interactive element nodes to form a candidate element set.

4. The natural language-based RPA method as described in claim 1, wherein, The step of inputting the cleaned DOM data, the natural language description in the configuration data, and the neighbor node information into a pre-trained multimodal large language model to generate intermediate instructions containing operation type and target location parameters includes: Construct prompt words, which include the cleaned DOM data structure, the natural language description of the current step, and the neighbor node information saved during the configuration phase; The prompt words are input into a multimodal large language model; Receive intermediate state instructions output by a multimodal large language model. The intermediate state instructions include operation function names and parameters, including the coordinate position of the target element on the page.

5. The natural language-based RPA method as described in claim 1, wherein, Before inputting the cleaned DOM data, the natural language description in the configuration data, and the neighbor node information into the pre-trained multimodal large language model, the method further includes: Check if there are any cached instructions in the instruction cache that match the current page state and the current step; If a matching cache instruction exists, the cache instruction is directly invoked as an intermediate instruction. If no matching cache instruction exists, then the step of inputting the cleaned DOM data, the natural language description in the configuration data, and the neighbor node information into the pre-trained multimodal large language model is executed.

6. The natural language-based RPA method as described in claim 5, wherein, The check for the presence of cached instructions matching the current page state and current step includes: Obtain the configuration instruction identifier corresponding to the current step; Get the coordinate range of the candidate target element in the current page; Determine whether there are records in the cache with the same configuration instruction identifier and whose cache coordinates fall within the range of the specified coordinates; If so, then a matching cache instruction is found.

7. The natural language-based RPA method as described in claim 1, wherein, The step of generating executable control instructions based on the intermediate state instructions and controlling the automated actuator to perform corresponding operations on the current page includes: Parse the operation function name and parameters in the intermediate state instruction; Map the operation function names to the native operation interfaces supported by the automated actuator; The native operation interface is invoked to perform click, input, or information retrieval operations on page elements based on the coordinate positions in the parameters. Receive the operation execution result and feed it back to drive subsequent steps.

8. A method for generating intelligent RPA configuration data, applied to a browser plugin or configuration terminal, comprising: Receive natural language descriptions of the target operation steps from the user in the graphical user interface; In response to the user's selection of a target page element, determine the position of the target page element in the Document Object Model (DOM) tree; According to a preset spatial order, information about neighboring elements around the target page element is collected, including the element name and element type. The configuration data is generated by combining the natural language description, the attribute information of the target page element, and the information of the neighboring elements. The configuration data is persistently stored so that it can be combined with real-time DOM data for instruction generation during runtime.

9. A smart RPA execution method, applied to a browser plugin or client executor, comprising: Receive a task start instruction, which is associated with pre-stored configuration data, including a natural language description of the target operation steps and information on the surrounding neighbor nodes of the target page element; Obtain the Document Object Model (DOM) data of the current page to be operated on, and clean the DOM data to remove noisy data to obtain a set of candidate elements; The candidate element set, the natural language description, and the surrounding neighbor node information are sent to the server to trigger the server to call the pre-trained multimodal large language model to generate an intermediate state instruction containing operation type and target location parameters. Receive executable control instructions generated based on the intermediate state instructions returned by the server; The executable control instructions are parsed, and corresponding operations are performed on the current page based on the target position parameters therein.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the natural language-based RPA method as described in any one of claims 1 to 7, or to implement the intelligent RPA configuration data generation method as described in claim 8, or to implement the intelligent RPA execution method as described in claim 9.

Images