System for constructing web-domain knowledge graphs for robotic process automation

Abstract

A system for constructing a web domain knowledge graph for robotic process automation is disclosed. A system for constructing a web domain knowledge graph for robotic process automation according to one embodiment of the present invention includes: a connection relationship automatic search module that automatically searches the link structure and URL patterns between pages of a web domain using Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms; a page interaction extraction module that analyzes the Document Object Model (DOM) structure of a web page, detects user interaction elements, and interprets them semantically through a Large Language Model (LM); a page interaction relationship analysis module that synthesizes the search and extraction results of the connection relationship automatic search module and the page interaction extraction module to analyze the relationship between actions and results of each page and derives state transitions between pages; and a knowledge graph generation module that generates and manages a knowledge graph by configuring pages and interaction elements as nodes and transition relationships as edges using the analysis results of the page interaction relationships.

Description

Technology Field

[0001] The present invention relates to a system for constructing a web domain knowledge graph for robotic process automation. Background Technology

[0003] Recently, there has been a growing trend among companies to automate repetitive and routine tasks to increase productivity and concentrate their workforce on high-value-added work. In particular, as web-based work becomes central due to the spread of cloud environments, the demand for automating tasks across various web services and applications is exploding. Amidst this trend, Robotic Process Automation (RPA) combined with artificial intelligence is garnering attention as a core technology capable of simultaneously improving work efficiency, reducing costs, and minimizing errors.

[0004] RPA technology is evolving beyond simple macros into intelligent and flexible automated web agents by integrating with artificial intelligence technologies such as natural language processing, computer vision, and machine learning. These agents replace repetitive tasks previously performed by humans by navigating web pages, extracting and inputting data, or executing procedures based on user commands.

[0005] However, existing automated web agents have limitations because they operate by relying simply on UI elements without understanding the structure or data semantics of specific websites; consequently, errors occur or processes halt when page structures or URLs change. This can lead to serious problems, particularly in large-scale web environments where complex business logic is applied. The problem to be solved

[0007] The technical problem that the present invention aims to solve is to provide a web domain knowledge graph construction system that eliminates inefficiencies and errors arising from an automated web agent operating in a web-based business environment without prior structural and semantic knowledge of a target web domain, and supports the automated web agent in stably performing tasks by automatically analyzing the website's page structure, the meaning of data fields, transition conditions, and operation rules of dynamic elements, and expressing them in the form of a knowledge graph composed of nodes and edges, thereby enabling the automated web agent to understand the context of the entire domain and independently select the optimal procedure.

[0008] However, the technical problems that the present invention aims to solve are not limited to those described above, and other unmentioned problems can be clearly understood by those skilled in the art from the description of the invention below. means of solving the problem

[0010] A web domain knowledge graph construction system for robotic process automation according to an embodiment of the present invention for solving the above technical problem comprises: a connection relationship automatic search module that automatically searches the link structure and URL patterns between pages of a web domain using Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms; a page interaction extraction module that analyzes the Document Object Model (DOM) structure of a web page, detects user interaction elements, and interprets them semantically through a Large Language Model (LM); a page interaction relationship analysis module that synthesizes the search and extraction results of the connection relationship automatic search module and the page interaction extraction module to analyze the relationship between actions and results of each page and derives state transitions between pages; and a knowledge graph generation module that generates and manages a knowledge graph by configuring pages and interaction elements as nodes and transition relationships as edges using the analysis results of the page interaction relationships.

[0011] In one embodiment, the connection relationship automatic search module automatically searches for pages within a site by performing BFS and DFS searches in parallel from a starting URL, and analyzes the parameter structure of the URL to group pages that differ only in their values ​​into the same type.

[0012] In one embodiment, the connection relationship automatic detection module prevents duplicate visits by converting the DOM structure of each page into a hash value and detects asynchronous page transitions by hooking router events in a Single Page Application (SPA) environment.

[0013] In one embodiment, the page interaction extraction module uses LLM to analyze the label, surrounding context, aria attribute, and hierarchy of each UI element to infer the semantic function of the element.

[0014] In one embodiment, the page interaction extraction module injects test input values ​​to learn validation rules and error message patterns for each input field, and extracts and records page transition relationships that occur upon form submission.

[0015] In one embodiment, the page-to-page interaction relationship analysis module analyzes the impact of an action within a page on movement to another page or data processing results, and models the action-result relationship as an edge.

[0016] In one embodiment, the knowledge graph generation module version controls the generated graph and improves maintenance efficiency by updating only the changed nodes when the UI of the web domain changes or the page structure changes.

[0017] A web domain knowledge graph building system for robotic process automation according to one embodiment of the present invention further includes an RPA request analysis module that analyzes a user's natural language command or business request to extract an intent, an object, and a constraint, and breaks them down into execution procedure units.

[0018] A web domain knowledge graph construction system for robot process automation according to one embodiment of the present invention further includes a path search module that searches for possible procedural paths based on a knowledge graph and derives an optimal path by evaluating success probability and efficiency through simulation.

[0019] In one embodiment, the path search module includes a Monte Carlo Tree Search (MCTS) based simulator and repeatedly performs random search and probabilistic evaluation.

[0020] A web domain knowledge graph construction system for robot process automation according to one embodiment of the present invention further includes an optimal path selection module that determines an optimal procedure by comprehensively evaluating efficiency, success rate, and error avoidance possibility based on the simulation results of the path search module.

[0021] A web domain knowledge graph construction system for robot process automation according to one embodiment of the present invention further includes a context providing module that configures execution context information, such as input fields, button locations, error handling rules, and alternative procedures required for each step of a path by referring to the knowledge graph, and provides this information to an automation web agent in real time. Effects of the invention

[0023] According to the present invention, by combining a structural understanding of a web domain with a semantic context, a Web Automation Agent can make judgments and perform tasks like a human, thereby achieving the following effects.

[0024] Through BFS / DFS-based relationship exploration and LLM-based inter-page relationship analysis, the page structure, UI elements, and transition relationships of the target web domain can be clearly identified. By structuring this identified information into a knowledge graph, it supports automated web agents in understanding and processing the overall context of the web domain. Consequently, unnecessary attempts or incorrect path selections that occurred during the automation process are minimized, and the understanding of the web domain is dramatically improved.

[0025] Furthermore, during RPA execution, execution errors can be minimized and success rates increased by exploring the optimal path based on a knowledge graph and providing essential information as context for each path step. BFS / DFS-based structural exploration automatically analyzes link relationships between pages and URL patterns, maintaining adaptability even when the site structure changes, while LLM-based semantic analysis accurately understands the role of each interaction element, enabling semantic-based automation rather than simple location-based automation. Consequently, the stability and accuracy of RPA are significantly improved, and adaptability to environmental changes, such as UI modifications or page structure variations, is enhanced, resulting in a substantial reduction in maintenance costs compared to conventional rule-based RPA.

[0026] Furthermore, by utilizing Monte Carlo Tree Search (MCTS) to explore possible paths from multiple angles and comprehensively evaluating efficiency, success probability, and constraints to select the optimal path, the overall automation processing time can be shortened by reducing unnecessary work procedures and duplicate waypoint pages. Through MCTS-based pathfinding, multiple paths can be probabilistically simulated and the procedure with the highest success probability can be automatically selected, maximizing work efficiency and significantly reducing the execution error rate.

[0027] Furthermore, by version-managing the collected and refined knowledge graph, it can be reused for repetitive tasks within the same domain or for similar domains. Additionally, the same pipeline can be applied to new web domains to automatically build and utilize knowledge. This ensures the reusability and scalability of knowledge, enhancing the versatility of the automation system and facilitating maintenance. Even when UI changes or feature additions occur, only the modified parts need to be updated without the need to relearn the entire structure, enabling continuous system evolution and efficient operation.

[0028] Furthermore, the capabilities of the AI-based automation web agent are enhanced through domain-specific contexts provided during the RPA execution process. By utilizing context information provided at each stage in real time, the automation web agent can go beyond simple command execution to autonomously formulate and execute optimal execution plans for complex procedural requests. Execution and error logs are reflected in the knowledge graph, allowing the system to self-improve, and accuracy and stability continue to improve over time.

[0029] The present invention provides a technical effect that enhances web domain understanding, automation reliability, knowledge reusability, and the autonomous decision-making ability of AI, while ensuring a high level of stability and efficiency compared to existing RPA.

[0030] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0032] The following drawings attached to this specification illustrate preferred embodiments of the present invention and serve to further enhance understanding of the technical concept of the present invention together with the detailed description of the invention provided below; therefore, the present invention should not be interpreted as being limited only to the matters described in such drawings. FIG. 1 illustrates a web domain knowledge graph construction system for robot process automation according to one embodiment of the present invention. FIG. 2 illustrates the specific configuration of an automatic connection relationship search module (100) according to one embodiment of the present invention. FIG. 3 illustrates the specific configuration of a page interaction extraction module (200) according to one embodiment of the present invention. FIG. 4 illustrates a knowledge graph utilization system when performing RPA according to an embodiment of the present invention. FIG. 5 is a block diagram showing a computer system for implementing a method according to one embodiment of the present invention. Specific details for implementing the invention

[0033] A preferred embodiment of the present invention will be described in detail below with reference to the attached drawings. Prior to this, terms and words used in this specification and claims should not be interpreted as being limited to their ordinary or dictionary meanings. Instead, based on the principle that the inventor can appropriately define the concept of a term to best describe their invention, they should be interpreted in a meaning and concept consistent with the technical spirit of the present invention. Therefore, the embodiments described in this specification and the configurations illustrated in the drawings are merely some of the most preferred embodiments of the present invention and do not represent all of the technical spirit of the present invention. It should be understood that various equivalents and modifications capable of replacing them may exist at the time of filing this application.

[0034] Additionally, as used herein, “comprise, include” and / or “comprising, including” specify the presence of the mentioned features, numbers, steps, actions, parts, elements, and / or groups thereof, and do not exclude the presence or addition of one or more other features, numbers, actions, parts, elements, and / or groups.

[0035] Throughout the specification, unless specifically stated otherwise, each component may be singular or plural.

[0036] Furthermore, where one component is described as being "on," "connected to," or "coupled to" another component, it should be understood that while the components may be directly connected or coupled to each other, another component may be "interposed" between each component, or that each component may be "connected," "coupled," or "coupled" through another component.

[0037] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Additionally, the use of “may” when describing embodiments of the invention relates to “one or more embodiments of the invention.” Expressions such as “one or more” and “one or more” preceding a list of elements modify the entire list of elements and do not modify individual elements of the list.

[0038] Throughout the specification, "A and / or B" means A, B, or A and B unless specifically stated otherwise, and "C to D" means C or more and D or less, unless specifically stated otherwise.

[0039] When syntax such as "at least one of A, B, and C", "at least one of A, B, or C", "at least one selected from the group of A, B, and C", or "at least one selected from A, B, and C" is used to specify a list of elements A, B, and C, the syntax can refer to any suitable combination.

[0040] The term "use" may be considered synonymous with the term "utilize." As used herein, terms such as "substantially," "about," and similar terms are used as terms of approximation rather than degree, and are intended to account for the inherent variation of a measured or calculated value that would be recognized by a person skilled in the art.

[0041] The terms used in this specification are for describing embodiments of the invention and are not intended to limit the invention.

[0043] Below, the background of the proposed invention is explained, followed by an explanation of embodiments of the invention.

[0044] Recently, there has been a rapidly increasing demand in companies and organizations to automate repetitive and routine tasks to improve productivity and reallocate human resources to higher value-added work. In particular, with the spread of cloud environments and the establishment of web-based work environments as core operational platforms for enterprises, the demand to automate business processes performed on various web services and web applications has exploded. Amidst this trend, Robotic Process Automation (RPA) technology utilizing artificial intelligence (AI) is emerging as a key technology capable of simultaneously achieving business efficiency, cost reduction, and error minimization.

[0045] RPA technology is evolving beyond simply mimicking human actions such as clicking, inputting, and sending via macros. It is now combining with various AI technologies, including Natural Language Processing (NLP), Computer Vision, and Machine Learning, to form Web Automation Agents capable of controlling web processes more intelligently and flexibly. Web Automation Agents replace repetitive web-based tasks by navigating web pages based on user commands or scenarios, extracting or inputting necessary information, and automatically executing specific procedures. For example, Web Automation Agents can perform tasks such as automatically querying, registering, and generating reports in corporate ERP systems, collecting and uploading product information on e-commerce platforms, or regularly crawling data and uploading documents on public institution websites.

[0046] However, these automated web agents still have inherent limitations. Automated web agents based on conventional technology often operate without prior knowledge of the target web domain site. In other words, they execute procedures based solely on superficial UI information, without understanding the specific site's page structure, the meaning of data fields, or the conditions under which dynamic elements such as buttons and menus operate. Consequently, even slight changes in page structure or URL patterns can cause the automation process to fail or trigger exceptions, frequently leading to the accumulation of unexpected errors that halt the entire process. Such errors are particularly critical in large-scale web environments or services containing complex business logic.

[0047] Since the structure and rules vary from website to website, even pages performing the same function differ in HTML structure, DOM tree depth, button names, and field placement across different services. Accordingly, for AI-based automated web agents to operate stably and accurately, it is essential to have a means to pre-learn knowledge specific to the web domain and provide it as context. This context must not be merely keyword-level information, but rather domain-specific structural and semantic knowledge that includes the relationships between pages, the meaning of data fields, the transition relationships of user interactions, and the hierarchical structure of URL patterns.

[0048] In other words, to enable automated web agents to move beyond simply sequentially traversing pages within a web domain and understand what functions each page performs and under what conditions transitions occur, the structural information and semantic relationships inherent in the web domain must be uncovered and systematically represented. In this context, the graph is the ideal data representation method for abstracting the entire domain and expressing relationships. By representing the connections between pages as nodes and edges through a graph-shaped knowledge structure, and by including semantic information such as field details, transition conditions, state changes, and functional descriptions in each node, automated web agents can refer to this graph to understand and make judgments about the domain, much like a human.

[0049] Therefore, by discovering structural and semantic knowledge about a specific web domain and organizing it in the form of a graph, the automated web agent can dramatically improve its understanding of that domain. This approach helps the automated web agent perform more sophisticated and stable tasks during the RPA execution process. For example, if there is a multi-step process involving authentication on a login page, querying data meeting specific conditions, and inputting the results into another system, the knowledge graph-based automated web agent can independently recognize the intent and conditions of each step, select an appropriate path, and branch to an alternative path in the event of an error.

[0050] This technology enhances the capabilities of AI in the field of automated web agents and provides a foundation for expanding the scope of RPA application from simple structured tasks to complex unstructured web environments. Furthermore, because it can continuously learn and update structural and semantic knowledge, it can respond immediately to frequent UI changes or policy modifications in web services. As a result, this technology maximizes the efficiency of business automation for enterprises and holds significance as a core infrastructure that presents the direction of evolution for AI-based RPA.

[0052] According to one embodiment of the present invention, the structural relationships of a web domain are automatically analyzed, and semantic information of each page is understood through a Large Language Model (LM) and expressed as a graph structure, thereby supporting RPA to perform web processes more intelligently.

[0053] Furthermore, it utilizes Breadth-First Search (BFS) or Depth-First Search (DFS) algorithms to systematically explore the relationships between pages within a domain, normalizes the DOM structure and URL parameters of each page to minimize redundancy, and classifies pages using the same template into a single type. Based on this collected structural information, LLM is used to interpret the functions and meanings of interaction elements within the page, such as input fields, buttons, and dropdowns, and to analyze what actions each element triggers. Subsequently, the Monte Carlo Tree Search (MCTS) algorithm is used to simulate all possible execution paths, evaluates the success probability and execution efficiency of each path, and automatically selects the optimal procedure. Through this process, an evolutionary automation structure is implemented that enables RPA to independently acquire domain-specific knowledge and adapt to changes in the web environment.

[0055] FIG. 1 illustrates a web domain knowledge graph construction system for robotic process automation according to an embodiment of the present invention. According to an embodiment of the present invention, structural information of a web domain is collected and analyzed, and semantic interpretation and knowledge graph generation based on LLM are performed.

[0056] A system according to one embodiment of the present invention is composed of an automatic connection relationship search module (100), a page interaction extraction module (200), an inter-page interaction relationship analysis module (300), and a knowledge graph generation module (400), and each module operates to automatically collect and analyze structural information of a web domain and to construct a knowledge graph by performing LLM-based semantic interpretation.

[0057] The connection relationship automatic search module (100) receives the starting URL of the target web domain, initializes the page visit queue and the visit set, and systematically searches the link structure between pages within the website by performing BFS (Breadth-First Search) and DFS (Depth-First Search) algorithms in parallel. In this process, the DOM (Document Object Model) structure of each page is analyzed to extract the layout signature, and this is stored as a hash value to prevent duplicate visits. If the same hash reappears during the search, it is determined that the page has already been visited and the search is omitted, thereby increasing search efficiency.

[0058] In addition, by performing parameter normalization, which tokenizes URL query parameters and replaces the value portion with a wildcard form, pages with identical structures but different values ​​are grouped into a single type. For example, " / product?id=1" and " / product?id=2" are determined to have the same structure and are represented by a normalized pattern in the form of " / product?id=*". This normalization is performed based on multiple metrics that consider DOM structure similarity, CSS selector patterns, and the distribution of JavaScript event listeners, rather than simple string comparison, thereby effectively identifying pages that perform the same functions from the perspective of actual users.

[0059] The connection relationship automatic detection module (100) also collects information on the link's rel, role, and aria attributes, the form's action, method, and enctype, and information on event listeners bound to each element, and marks candidate actions that are likely to cause page transitions as tags. For web applications with a Single Page Application (SPA) structure, it detects page transitions that occur asynchronously by hooking the browser's router events and recognizes asynchronous page state changes by monitoring DOM change signals. The information collected in this way is stored in the form of page-unit metadata, and information such as URL patterns, access rights, whether authentication is required, and transition conditions is recorded together.

[0060] The page interaction extraction module (200) receives a DOM snapshot of each page that has been explored, analyzes user interaction (UI) elements within the page, and interprets them semantically. It automatically detects interactive elements such as input fields, buttons, dropdowns, checkboxes, radio buttons, and links, and collects attribute information such as name, id, type, placeholder, aria-label, and value for each element. In addition, it analyzes the label text around the element, the semantic block of the parent container, and visual grouping information together to infer the functional meaning of the element.

[0061] The page interaction extraction module (200) interprets the meaning of each interaction element using a Large Language Model (LLM). Based on semantic patterns learned from large-scale language data, the LLM analyzes the names and contexts of UI elements and determines their functional roles. For example, "Order Number," "Order ID," and "OrderNo" are recognized as having the same meaning and are integrated into a single semantic node. Additionally, if a "search term input" field and a "search" button exist within the same area, a semantic relationship is inferred that the two elements combine to form a single "search operation."

[0062] The page interaction extraction module (200) performs execution validation in addition to semantic analysis. This is a process of automatically learning input validation rules and error handling patterns by injecting test input values ​​into each field and observing the response of the page. For example, when abnormal input such as "AA! eyzz wpol el" or "Qe} Ole] PAlee" is inserted into a field and a message such as "Invalid value" or "Incorrect format" is generated, the system determines that there is validation logic in that field. During the validation process, changes in the DOM structure, the output location of error messages, and whether CSS classes have changed are recorded together, and this information is used to identify the functional attributes of the interaction.

[0063] The page-to-page interaction relationship analysis module (300) determines the page-to-page transition relationship by synthesizing the processing results of the connection relationship automatic discovery module (100) and the page interaction extraction module (200). First, the relationship between Action and Result is derived by cross-comparing the transition trigger candidates indicated by the connection relationship automatic discovery module (100) with the transition trigger probability determined by the page interaction extraction module (200). When a specific action leads to multiple results, it is modeled as a Conditional Transition, and a condition tag (e.g., success, failure) and an observation probability are assigned to each transition. For example, when the "Login" button is clicked, if authentication is successful, the user moves to the dashboard page, and if authentication fails, an error message is displayed. The module generates two transition edges and records the transition probability and state for each.

[0064] The page-to-page interaction relationship analysis module (300) collects the trigger element identifier for each transition, the configuration of the submitted input field, the response time, the status code, the network request path, the error message, etc., together and stores them as edge attributes. Through this, the reliability, success rate, and probability of error occurrence of the transition path can be quantitatively modeled.

[0065] The knowledge graph generation module (400) constructs a knowledge graph in the form of nodes and edges based on the analysis results. The nodes consist of page type nodes and interaction element nodes, and the page type nodes include metadata such as URL patterns, whether authentication is required, access policies, layout hashes, and main widget configurations. The interaction element nodes include attributes such as semantic tags, field constraints, error patterns, and whether transitions are triggered.

[0066] Edges are defined by three basic types: "contains", "triggers", and "transitionsTo". Each edge includes execution characteristics such as transition conditions, average response time, success probability, and messages generated upon failure as annotations. The knowledge graph generation module (400) performs a graph version management function. If the structure of a page changes and the DOM signature changes, the reliability of the corresponding node is temporarily lowered, and after re-analysis, it is updated to a new version. This allows only the changed parts to be efficiently updated without rebuilding the entire graph.

[0067] According to one embodiment of the present invention, parameter normalization and DOM hash-based duplicate removal techniques of the connection relationship automatic search module (100), LLM-based semantic analysis and execution verification of the page interaction extraction module (200), probability-based conditional transition modeling of the page interaction relationship analysis module (300), and version control and reliability update mechanisms of the knowledge graph generation module (400) work complementarily to enable the generation of a knowledge graph having a higher level of structural completeness and semantic consistency than existing static rule-based automation systems.

[0068] According to one embodiment of the present invention, by integrally performing structural exploration and semantic interpretation of a web domain, an automated web agent is enabled to understand the entire context of the domain and perform stable and intelligent robotic process automation.

[0070] FIG. 2 illustrates the specific configuration of an automatic connection relationship search module (100) according to one embodiment of the present invention.

[0071] An automatic connection relationship search module (100) according to one embodiment of the present invention automatically searches for structural connection relationships between all pages within a web domain, collects them in a normalized form, and models the structural topology of the web domain by outputting them in the form of an adjacency graph between pages.

[0072] The connection relationship automatic search module (100) is configured to include a BFS / DFS-based coverage search unit (110), a URL link extraction and normalization unit (120), a similar page clustering unit (130), and an inter-page adjacency graph output unit (140).

[0073] The BFS / DFS-based coverage search unit (110) receives a starting URL that serves as the starting point for the search, initializes a visit queue and a visit set, and then performs BFS (Breadth-First Search) and DFS (Depth-First Search) algorithms in parallel. The BFS method is advantageous for quickly grasping the global structure of the entire domain, while the DFS method is suitable for tracking the detailed procedures of a specific link path to the end. According to one embodiment of the present invention, the two methods are combined in a dynamic hybrid form to ensure a balanced breadth and depth of the search. For example, the ratio between BFS and DFS is automatically adjusted according to the density of transition links or the search depth threshold.

[0074] During the navigation process, the Document Object Model (DOM) structure of each page is parsed, and information such as links (anchors), forms, navigation (nav) elements, and event listener bindings within the page is collected. Each page stores a layout signature value generated from the DOM structure in the form of a hash; if the same signature appears again during the navigation process, the page is considered to have already been visited, preventing duplicate visits. This improves navigation efficiency and blocks abnormal navigation, such as infinite loops or circular references.

[0075] The URL link extraction and normalization unit (120) extracts the URLs of all links found during exploration and performs normalization by analyzing the parameter structure of the URLs. URL parameter normalization is performed to integrate identical pages with only different values ​​into a single type, and replaces the value part of the key-value pair of parameters in the query string with a wildcard or variable form. For example, " / product?page=1" and " / product?page=2" are recognized as having the same structure and are normalized to " / product?page=".

[0076] The aforementioned normalization process is performed not through simple string comparison, but by comprehensively considering the DOM structure of each page, layout similarity, CSS selector patterns, and types of event triggers. Additionally, dynamic segments within the URL path (e.g., / user / 123 / profile) are extracted and generalized into the form " / user / {id} / profile". These normalized URLs are used as base node identifiers to represent the hierarchical structure of page types within the domain.

[0077] Similar page clustering unit (130) clusters pages by type based on normalized URLs and DOM signatures. During the clustering process, structural similarity between pages, DOM tree depth, common UI components, JavaScript event listener patterns, etc., are evaluated in an integrated manner. Pages with a similarity exceeding a set threshold (e.g., 0.8 or higher) are classified into the same type and integrated into a single cluster. This allows for the effective identification of duplicate page templates within the site and reduces the complexity of the overall graph.

[0078] The similar page clustering unit (130) applies a multi-stage clustering method to first perform DOM-based clustering, and then perform URL pattern-based clustering. Through a double verification structure, not only structural identity but also semantic equivalence is guaranteed. For example, " / product?id=" and " / item?no=" are integrated into a single product detail page type if their DOM structures are identical.

[0079] The page-to-page adjacency graph output unit (140) outputs the adjacency relationships of web domains in the form of a graph based on the results of exploration and clustering. Each page type is represented as a Node, and the movement relationships between pages, such as links or form submissions, are represented as Edges. The Edges include metadata such as "Transition Condition," "Trigger Element," "Response Code," and "Average Latency."

[0080] The page-to-page adjacency graph output unit (140) stores graph data in a dynamic expandable structure so that when a new page is explored, it is merged into the existing graph in real time. Through this, even if the web domain is updated or new features are added, it can be updated incrementally without repeating the entire exploration.

[0081] Once the traversal is complete, traversal statistics such as access rights for each page node, authentication requirements, the number of transitive actions, the number of internal links, and structural depth are recorded. These statistics are subsequently used as a basis for calculating page importance or determining traversal priorities during the page semantic analysis and knowledge graph construction processes.

[0082] As described above, the connection relationship automatic search module (100) according to one embodiment of the present invention automatically models the structural connection relationships of a web domain by integrating a BFS / DFS parallel search structure, URL parameter normalization, similar page clustering, and adjacency graph output function. Through this, the complex navigation structure of a website is semantically simplified, and the navigation context of the entire domain can be clearly grasped. These results are subsequently utilized as core foundational data in the knowledge graph construction and RPA execution stages.

[0083] According to one embodiment of the present invention, by automatically exploring the structure of a web domain, normalizing URL patterns, and clustering based on structural similarity to output in the form of an adjacency graph, an automated web agent is supported to accurately recognize structural relationships throughout the web domain.

[0085] FIG. 3 illustrates the specific configuration of a page interaction extraction module (200) according to an embodiment of the present invention. The page interaction extraction module (200) according to an embodiment of the present invention is configured to analyze the structural DOM (Document Object Model) information of a web page, interpret the meaning of each UI element using an LLM (Large Language Model), verify the functional attributes of the element through an action execution simulation, and finally correct the consistency of the UI element analysis.

[0086] A page interaction extraction module (200) according to one embodiment of the present invention comprises a DOM parsing and UI element attribute extraction unit (210), an LLM-based UI element semantic analysis unit (220), an action execution simulator (230), and a UI element analysis consistency verification and correction unit (240).

[0087] The DOM parsing and UI element attribute extraction unit (210) collects the HTML structure of a web page and parses it into a DOM tree to detect all interactive UI elements present in the page. The targets for detection include input fields, buttons, selects, radio buttons, checkboxes, anchors, toggles, and AJAX triggers. For each UI element, attribute information such as name, id, class, type, placeholder, aria-label, and value is extracted, and label text around the element, semantic blocks of the parent container, and visual grouping information (e.g., fieldset, div grouping) are collected together.

[0088] The extracted data is not a simple list of attributes, but is organized at the level of element contextual metadata. For example, if an "input field" appears with "label='Order Number'", the element is defined as an "order number input field". Furthermore, the present invention utilizes DOM hierarchy information to identify sections containing UI elements (e.g., login forms, search areas, filter areas) and records the semantic roles of each section. In this process, hierarchical relationships between elements, visual positions (coordinates), and style attributes (CSS classes) are metadataized together and utilized as auxiliary information during the LLM semantic interpretation stage.

[0089] The LLM-based UI element semantic analysis unit (220) receives extracted metadata and interprets the semantic function of each UI element. According to one embodiment of the present invention, a Large Language Model (LLM) classifies the meaning of an element by analyzing the attribute name, label, and surrounding context of the UI element based on pre-learned language semantic knowledge. For example, when notations such as "Order Number," "Order ID," "Order No," and "Customer Number" appear, the model recognizes all of them as "identifier input fields."

[0090] LLM learns the semantic associations between surrounding interactions and infers that if a search input field and a lookup button exist in the same UI section, these two elements are grouped into a single functional unit called a Search Action. Through this, it can automatically construct a Semantic Interaction Set that goes beyond the level of simple attributes.

[0091] An LLM-based UI element semantic analysis unit (220) according to one embodiment of the present invention performs semantic similarity-based analysis rather than simple sentence matching, and determines that even different language expressions, such as "product name," "Product," and "Item name," have the same meaning if the semantic similarity in the vector space is above a certain threshold. In addition, a confidence score is assigned to the LLM analysis result, and results with low confidence are set to be re-examined in a subsequent consistency verification step.

[0092] The action execution simulator (230) empirically analyzes the functional characteristics of each element by injecting test input values ​​and observing the page's response to verify the actual operation of the UI elements. For example, it checks whether a DOM change or a network request occurs when the "login button" is clicked, and in the case of the "search button," it observes whether the input field is validated and the result page is moved.

[0093] The action execution simulator (230) reproduces various interactions in addition to simple click actions, such as field input, form submission, dropdown selection, asynchronous request (AJAX), and modal window activation. It also checks for the occurrence of error messages by inputting abnormal strings such as "AA! eyzz wpol el" and "Qe} Ole] PAlee" and identifies the existence of validation rules. For example, if feedback is received that the email format is incorrect, it records that the field contains a validation rule.

[0094] The action execution simulator (230) collects DOM change logs, network request logs, console logs, response times, and error events for each action result, and stores them as functional verification data for each UI element. In addition, through this execution-based verification, the transition relationships of interactions (e.g., login → dashboard, search → result page) can be automatically extracted.

[0095] The UI element analysis consistency verification and correction unit (240) compares the LLM-based semantic analysis results with the execution simulation results to verify the consistency of the analysis and, if necessary, correct the semantic tagging, and uses a hybrid semantic verification technique for this purpose.

[0096] The hybrid semantic validation technique compares three information axes: semantic tags predicted by the LLM, structural roles derived from DOM analysis, and actual results from an execution simulator. If the results of the three axes match, the final semantic tagging is confirmed; if they do not match, a reinterpretation procedure is performed to correct the semantic tags. For example, if the LLM classifies a specific button as a "search button," but it is confirmed that clicking it actually performs a "form initialization" action, the system corrects the semantic tag to "initialization button."

[0097] The UI element analysis consistency verification and correction unit (240) performs confidence-driven re-learning. By feeding back patterns of repeated inconsistencies during the analysis process to the LLM and re-learning UI elements of the same structure, the accuracy of future analysis is gradually improved.

[0098] A page interaction extraction module (200) according to one embodiment of the present invention provides an intelligent analysis structure that combines semantic understanding and behavioral validation, going beyond simple static HTML analysis. By obtaining structural information through DOM parsing, interpreting contextual meaning through LLM-based semantic analysis, verifying actual functions through an action execution simulator, and cross-evaluating results through consistency verification, the interaction structure of a web page can be understood at a human level.

[0099] According to one embodiment of the present invention, by precisely identifying the relationship between the semantic function and actual operation of UI elements, the automation web agent is supported to perform semantic-based judgments beyond simple location-based control. This enables stable and accurate automation even in unexpected UI changes or multilingual environments when performing RPA.

[0101] FIG. 4 illustrates a knowledge graph utilization system when performing RPA according to an embodiment of the present invention.

[0102] According to one embodiment of the present invention, it is configured to perform stable and intelligent robotic process automation by interpreting a user's natural language-based request, searching for an optimal path based on Monte Carlo Tree Search (MCTS) by referring to a knowledge graph, and providing context information required at each step to an automated web agent.

[0103] A knowledge graph utilization system for RPA execution according to one embodiment of the present invention comprises an RPA request analysis module (500), a path search module (600), an optimal path selection module (700), a context provision module (800), and a knowledge graph (900).

[0104] The RPA request analysis module (500) analyzes requests entered by the user in natural language or based on commands and breaks down tasks to be performed into detailed procedure units. The RPA request analysis module (500) consists of an input data receiving unit (510), a knowledge graph binding unit (520), and a task analysis unit (530). The input data receiving unit (510) collects user requests, input parameters, constraints, etc., and converts them into a structured form that can be analyzed. For example, if a command such as "Save only completed orders as CSV and send an approval email" is entered, the system breaks it down into procedure units such as "Login → Access order list page → Apply filter (delivery completed) → Export CSV → Send email".

[0105] The knowledge graph binding unit (520) connects each step of the analyzed procedure with a node within the knowledge graph. Since the knowledge graph stores the web domain's page structure, interaction elements, transition relationships, action-result relationships, etc., in the form of nodes and edges, each step of the request analysis result is mapped to a specific node and edge on the graph. For example, the 'access order list page' step is bound to the 'OrderListPage' node within the graph, and the 'export CSV' step is connected to 'ExportCSVAction' among the edges of the corresponding page.

[0106] The task analysis unit (530) defines execution goals based on mapped information and generates all possible paths for achieving the goals as candidates for exploration. At this time, the initial state, goal conditions, and constraints of the path exploration are explicitly defined, and the path exploration module uses them as input in subsequent steps.

[0107] The path search module (600) consists of a Monte Carlo Tree Search (MCTS) based simulator (610) and a simulator update unit (620). The path search module (600) probabilistically searches for all possible paths based on a knowledge graph. The MCTS-based simulator (610) represents paths in the form of state-action-result and randomly simulates multiple paths by expanding all possible actions at each node. Subsequently, it evaluates the success probability, estimated execution time, resource usage, and probability of error of each path to derive the statistically most efficient path.

[0108] The MCTS-based simulator (610) repeatedly performs four stages of search procedures: selection, expansion, simulation, and backpropagation. In the selection stage, nodes with high expected rewards are selected based on the search results to date, and in the expansion stage, the search space is expanded by adding new actions. In the simulation stage, the selected path is simulated in a virtual execution environment, and in the backpropagation stage, the simulation results are transmitted to the parent nodes to update the success rate and expected efficiency of each node.

[0109] The simulator update unit (620) collects actual RPA execution logs and corrects the statistical values ​​of the simulation model in real time. That is, if the error rate or success rate that occurred during actual execution differs from the expected value, it adjusts the weight of the corresponding node to improve search accuracy and has a self-improving structure in which accuracy is gradually improved through repeated execution.

[0110] The optimal path selection module (700) determines the path with the highest efficiency among multiple candidate paths transmitted from the path search module (600). The optimal path selection module (700) comprehensively evaluates the total number of steps, estimated execution time, success probability, resource cost, and error avoidance possibility of the path. For example, if two paths can achieve the same result but one has a simpler authentication process or a higher transition success rate, the score of that path is calculated higher and selected as the optimal path.

[0111] The optimal path selection module (700) dynamically adjusts the weights of path selection by referring to the log of past execution data. For example, if a specific path has succeeded 9 out of 10 times in the past, the reliability of that path is strengthened, and the weight of a path that has repeatedly failed is lowered. Such dynamic weight adjustment forms an adaptive optimization structure that improves the quality of path selection over the long term.

[0112] The context providing module (800) generates context information required for each step of the confirmed optimal path and transmits it to the automation web agent. The context providing module (800) consists of a core knowledge extraction unit (810) and a context packaging unit (820). The core knowledge extraction unit (810) refers to a knowledge graph to extract field constraints, button locations, authentication information, error message patterns, alternative procedures, etc., required for each step. For example, in the login step, execution rules such as 'ID field: required', 'Password: 8 characters or more', and 'retry once upon authentication failure' are generated.

[0113] The context packaging unit (820) converts these execution rules and state information into an executable data structure and transmits it to the web agent. For example, input field mappings for each stage, action commands, exception handling rules, restoration procedures, etc., are configured in the form of JSON or a command tree. The context packaging unit (820) also monitors exception situations occurring during execution in real time, and immediately applies an alternative path when an error occurs, or triggers a graph update by providing feedback on information about the failed node.

[0114] The knowledge graph (900) serves as a core data repository and holds page structures, interaction elements, transition conditions, input constraints, error patterns, etc., within a web domain in the form of nodes and edges. Each node represents a state, and the edges connect possible actions and results. Additionally, each edge contains various metadata such as transition conditions, success probabilities, average execution times, and error logs.

[0115] The knowledge graph (900) is continuously updated even during RPA execution. When execution result logs are collected, the system compares and analyzes success and failure data to adjust the reliability value of each edge, and automatically adds new nodes when new actions or page structures are detected. This structure enables the knowledge graph to function not as a simple static data store, but as an adaptive knowledge base that continuously grows through execution experience.

[0116] According to one embodiment of the present invention, an autonomous RPA execution environment is implemented through a complete loop of natural language request interpretation, MCTS-based pathfinding, execution log feedback, real-time context provision, and graph updating. Through this, it is possible to autonomously search for optimal automation procedures even in complex web environments, minimize the probability of failure, and adapt in real time to changing UIs or policies.

[0117] According to one embodiment of the present invention, by combining knowledge graph-based semantic understanding and probabilistic optimization search, a much higher level of intelligent automation performance and self-learning ability is provided compared to existing rule-based RPA systems.

[0119] FIG. 5 is a block diagram showing a computer system for implementing a method according to an embodiment of the present invention.

[0120] Referring to FIG. 5, a computer system (1300) may include at least one of a processor (1310), memory (1330), an input interface device (1350), an output interface device (1360), and a storage device (1340) that communicate via a bus (1370). The computer system (1300) may also include a communication device (1320) coupled to a network. The processor (1310) may be a central processing unit (CPU) or a semiconductor device that executes instructions stored in memory (1330) or storage device (1340). Memory (1330) and storage device (1340) may include various forms of volatile or non-volatile storage media. For example, memory may include read-only memory (ROM) and random access memory (RAM). In the embodiments of this description, memory may be located inside or outside the processor, and memory may be connected to the processor through various known means. Memory is a volatile or non-volatile storage medium of various forms, and for example, memory may include read-only memory (ROM) or random access memory (RAM).

[0121] Accordingly, embodiments of the present invention may be implemented as a method implemented on a computer or as a non-transient computer-readable medium storing computer-executable instructions. In one embodiment, when executed by a processor, the computer-readable instructions may perform a method according to at least one aspect of the present description.

[0122] The communication device (1320) can transmit or receive wired or wireless signals.

[0123] In addition, the method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable medium.

[0124] The above computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for embodiments of the present invention, or they may be known and available to a person skilled in the art of computer software. The computer-readable recording medium may include a hardware device configured to store and execute program instructions. For example, the computer-readable recording medium may be magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROM; RAM; flash memory, etc. The program instructions may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer through an interpreter, etc.

[0125] Although the present invention has been described above with reference to limited embodiments and drawings, the present invention is not limited thereto, and it is obvious that various modifications and variations are possible within the scope of the technical spirit of the present invention and the equivalent scope of the claims described below by those skilled in the art to which the present invention belongs.

[0126] delete Explanation of the symbols delete

Claims

Claim 1 A connection relationship automatic search module that automatically searches the link structure and URL patterns between pages of a web domain using Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms; a page interaction extraction module that analyzes the Document Object Model (DOM) structure of web pages, detects user interaction elements, and interprets them semantically through Large Language Model (LM); and a page interaction relationship analysis module that synthesizes the search and extraction results of the connection relationship automatic search module and the page interaction extraction module to analyze the relationship between actions and results of each page and derives state transitions between pages. A web domain knowledge graph construction system for robotic process automation, comprising a knowledge graph generation module that generates and manages a knowledge graph by configuring pages and interaction elements as nodes and transition relationships as edges using analysis results of interaction relationships between pages and pages, and the connection relationship automatic discovery module converts the DOM structure of each page into a hash value to prevent duplicate visits and detects asynchronous page transitions by hooking router events in a Single Page Application (SPA) environment. Claim 2 In claim 1, the above-mentioned connection relationship automatic search module automatically searches for pages within a site by performing BFS and DFS searches from a starting URL in parallel, and analyzes the parameter structure of the URL to group pages with only different values ​​into the same type, a web domain knowledge graph construction system for robotic process automation. Claim 3 delete Claim 4 In claim 1, the page interaction extraction module is a web domain knowledge graph construction system for robotic process automation that infers the semantic function of each UI element by analyzing the label, surrounding context, aria attribute, and hierarchical structure of each UI element using LLM. Claim 5 A web domain knowledge graph construction system for robotic process automation comprising: an automatic connection relationship search module that automatically searches the link structure and URL patterns between pages of a web domain using Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms; a page interaction extraction module that analyzes the Document Object Model (DOM) structure of a web page, detects user interaction elements, and interprets them semantically through Large Language Model (LM); a page interaction relationship analysis module that synthesizes the search and extraction results of the automatic connection relationship search module and the page interaction extraction module to analyze the relationship between actions and results of each page and derives state transitions between pages; and a knowledge graph generation module that generates and manages a knowledge graph by configuring pages and interaction elements as nodes and transition relationships as edges using the analysis results of the page interaction relationships, wherein the page interaction extraction module learns validation rules and error message patterns of each input field by injecting test input values, and extracts and records page transition relationships that occur upon form submission. Claim 6 In claim 1, the page-to-page interaction relationship analysis module analyzes the impact of actions within a page on movement to another page or data processing results, and models action-result relationships as edges, a web domain knowledge graph construction system for robotic process automation. Claim 7 A web domain knowledge graph construction system for robotic process automation, wherein the knowledge graph generation module version controls the generated graph and improves maintenance efficiency by updating only the changed nodes when the UI of the web domain changes or the page structure changes. Claim 8 A connection relationship automatic search module that automatically searches the link structure and URL patterns between pages of a web domain using Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms; a page interaction extraction module that analyzes the Document Object Model (DOM) structure of web pages, detects user interaction elements, and interprets them semantically through Large Language Model (LM); and a page interaction relationship analysis module that synthesizes the search and extraction results of the connection relationship automatic search module and the page interaction extraction module to analyze the relationship between actions and results of each page and derives state transitions between pages. A web domain knowledge graph construction system for robotic process automation, comprising a knowledge graph generation module that generates and manages a knowledge graph by configuring pages and interaction elements as nodes and transition relationships as edges using analysis results of interaction relationships between pages and pages, and further comprising an RPA request analysis module that analyzes a user's natural language command or business request to extract intent, object, and constraint, and breaks them down into execution procedure units. Claim 9 A web domain knowledge graph construction system for robotic process automation, further comprising a path search module that explores possible procedural paths based on a knowledge graph and derives an optimal path by evaluating success probability and efficiency through simulation in claim 1. Claim 10 In claim 9, the path search module comprises a Monte Carlo Tree Search (MCTS) based simulator and is a web domain knowledge graph building system for robotic process automation that repeatedly performs random search and probabilistic evaluation. Claim 11 A web domain knowledge graph construction system for robotic process automation, further comprising, in claim 9, an optimal path selection module that determines the optimal procedure by comprehensively evaluating efficiency, success rate, and error avoidance possibility based on the simulation results of the path search module. Claim 12 A web domain knowledge graph construction system for robotic process automation, comprising: a connection relationship automatic search module that automatically searches the link structure and URL patterns between pages of a web domain using Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms; a page interaction extraction module that analyzes the Document Object Model (DOM) structure of a web page, detects user interaction elements, and interprets them semantically through Large Language Model (LM); a page interaction relationship analysis module that synthesizes the search and extraction results of the connection relationship automatic search module and the page interaction extraction module to analyze the relationship between actions and results of each page and derives state transitions between pages; and a knowledge graph generation module that generates and manages a knowledge graph by configuring pages and interaction elements as nodes and transition relationships as edges using the analysis results of the page interaction relationships, and further comprising a context provision module that configures execution context information, such as input fields, button locations, error handling rules, and alternative procedures required for each step of a path by referencing the knowledge graph, and provides this information to an automated web agent in real time.

Images