Mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning

Abstract

This invention proposes an automated mobile software method based on semantic knowledge graphs and LLM-enhanced reasoning. It explores and constructs an initial knowledge graph through an automated interface, and leverages the strong semantic understanding capabilities of a large language model to efficiently complete and semantically enrich the graph, quickly forming a complete knowledge representation covering component attributes, operational logic, and cross-application dependencies. The structured knowledge of the knowledge graph effectively constrains the reasoning process of the large language model. The system automatically explores the application interface and constructs the knowledge graph using a hybrid traversal strategy, employs a template-instance abstraction mechanism to adapt to dynamic content, and performs multi-dimensional instruction matching and path planning based on the graph, ultimately transforming natural language tasks into executable system-level operation sequences. This invention achieves an end-to-end automated closed loop from instruction parsing to execution monitoring, covering diverse user operation scenarios. While ensuring response efficiency, it significantly reduces parsing and execution errors, effectively improving the adaptability, robustness, and user experience of mobile automated control.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, software automation, and mobile computing, and particularly to a mobile software automation method, apparatus, electronic device, computer-readable storage medium, and computer program product based on semantic knowledge graphs and LLM-enhanced reasoning.

[0002] More specifically, this invention relates to mobile application graphical user interface (GUI) automation technology, including automated testing, task script execution, and exploratory interaction; to knowledge graph construction and application technology, a knowledge modeling method used in software engineering to represent user interface structure, component semantics, and interaction logic; to the application of Large Language Models (LLM) in reasoning and decision-making in software automation, including how to use LLM to understand interface semantics, parse user intent, and generate reliable operation sequences; and to cross-application, multi-step mobile task automation technology, aiming to solve the end-to-end operation execution problem under conditions of dynamic interface changes and lack of stable API interfaces. This invention aims to provide mobile applications with an intelligent, autonomous automation solution that does not rely on source code, requires no predefined scripts or fixed APIs, and can adapt to interface changes. Background Technology

[0003] Software automation is a computer application field that uses technology to automate program design and documentation, with the core goal of improving software development and management efficiency.

[0004] With the rapid development of the mobile internet, mobile applications have become a core component of people's daily lives and work. This has led to a strong demand for efficient and reliable automation of mobile applications, with applications widely covering software testing, accessibility assistance, process automation, and personal task assistants. However, the inherent characteristics of mobile applications, such as frequent interface updates, diverse screen sizes and resolutions, rich dynamic content, and fragmented operating system versions, pose significant challenges to traditional automation technologies.

[0005] Existing mobile application automation technologies mainly follow several technical paths, each with its significant limitations. First, there are scripting and recording / playback tools, represented by Appium and UiAutomator. These tools require developers or testers to write interaction scripts based on UI element locators or generate scripts by recording the operation process. Their main problem lies in fragility; once the application's UI layout, component identifiers, or structure changes, the script becomes invalid, requiring manual relocation of elements or modification of the script, resulting in high maintenance costs. Furthermore, they struggle to handle unexpected dialog boxes or dynamically loaded content that pops up at runtime. Second, there are modeling and random exploration methods. These methods automatically or semi-automatically explore the application interface, building state machines, activity graphs, or event flow models to improve interface coverage. Examples include Monkey-based random testing and deep learning-based GUI traversal strategies. While they can discover more interface states, they generally lack advanced semantic understanding capabilities. Their optimization goals are usually to maximize structural metrics such as Activity or Widget coverage, rather than reliably completing user-specified semantic-level tasks. They cannot understand the meaning of a button's "submit" button, nor can they handle complex workflows that require decisions across multiple interfaces and based on context. The third approach is API- and intent-based assistants, such as Google Assistant and voice assistants from mobile phone manufacturers. These interact through predefined shortcuts or deep links within the application. While this method provides a natural language input channel, its capabilities are entirely limited to the API interfaces pre-registered and publicly available by the developer. It cannot operate any functions or interfaces within the application that are not exposed, and is therefore not true interface-level automation, lacking versatility.

[0006] In recent years, large language models, exemplified by ChatGPT, have demonstrated powerful natural language understanding and contextual reasoning capabilities, and have been introduced into the field of GUI automation. Some cutting-edge research attempts to enable LLMs to analyze screenshots or control trees to directly generate next action instructions, or to generate exploration scenarios during testing. These works demonstrate that LLMs possess preliminary potential for interface element recognition and intent reasoning, and can produce more human-like interactions than random exploration. However, current LLM-driven mobile automation solutions still have key shortcomings: They lack persistent, structured interface models. Most systems treat each interaction as an independent event, making decisions based solely on the instantaneous information of the current screenshot, failing to build and maintain a long-term, stable, and queryable application interface knowledge base. This results in the system's inability to accumulate experience, plan long-term tasks, and effectively understand the application's navigation structure. Semantic representation and grounding capabilities are weak. Existing methods typically rely on heuristic rules or simple text matching to ground user commands to UI elements, lacking deep semantic annotation of component functional roles and robust mechanisms for handling synonyms and dynamic content. When UI text labels and user command expressions are not entirely consistent, the grounding failure rate is high. Adaptability to dynamic content and interface changes is poor. Most research focuses on test generation or static interface analysis, lacking systematic strategies for handling dynamic lists, personalized recommendations, and layout adjustments brought about by application version updates prevalent in actual use. There is also the existence of illusions and unreliable reasoning. Without factual constraints, LLM may generate operation steps that do not exist in the current interface, leading to automation process interruptions or erroneous operations.

[0007] In summary, existing technologies have not yet provided a mobile autonomous automation solution that combines high robustness, high accuracy, strong adaptability, and good maintainability. Therefore, a novel method that integrates structured knowledge representation and semantic understanding reasoning is proposed to overcome the aforementioned bottlenecks. Summary of the Invention

[0008] This invention relates to a mobile software automation method based on structured semantic knowledge graphs and LLM-enhanced reasoning, belonging to the interdisciplinary field of artificial intelligence and mobile computing. This method aims to solve problems in current mobile application automation technologies, such as poor robustness, insufficient maintainability, and inaccurate natural language command parsing caused by frequent interface changes, uncontrollable dynamic content, and high script maintenance costs. Its core lies in achieving deep collaboration between a large language model and a semantic knowledge graph: on the one hand, an initial knowledge graph is constructed through automated interface exploration, and the strong semantic understanding capability of the large language model is used to efficiently complete and semantically enrich the graph, quickly forming a complete knowledge representation covering component attributes, operational logic, and cross-application dependencies; on the other hand, the structured knowledge of the knowledge graph effectively constrains the reasoning process of the large language model, thereby suppressing illusion generation and significantly improving the accuracy and reliability of command parsing. Illusions, for example, refer to situations where the knowledge graph stores explicit entities such as "mobile application page elements," relationships such as "button A → trigger function B," and attributes such as "input box C requires a mobile phone number." If the model generates content such as "Function B is associated with button D" or "Input box C supports email format" that does not exist in the knowledge graph during inference, then the model's output becomes an illusion. The system automatically explores the application interface and constructs a knowledge graph through a hybrid traversal strategy, adapts dynamic content using a template-instance abstraction mechanism, and performs multi-dimensional instruction matching and path planning based on the knowledge graph, ultimately transforming the natural language task into an executable system-level operation sequence. This method achieves an end-to-end automated closed loop from instruction parsing to execution monitoring, covering diverse user operation scenarios. While ensuring response efficiency, it significantly reduces parsing and execution errors, effectively improving the adaptability, robustness, and user experience of mobile automated control.

[0009] To address the shortcomings of existing technologies, such as Figure 6 As shown, this invention proposes an automated method for mobile software based on semantic knowledge graphs and LLM-enhanced reasoning, including:

[0010] The initial knowledge graph construction steps involve extracting interface information from mobile devices running mobile applications (APPs), and based on the content and function of each visited page in the interface information, exploring to obtain a unique page signature for each visited page in order to construct the initial knowledge graph. The node set V contains page nodes and their interactive component nodes, and the edge set E contains structural edges reflecting parent-child relationships and action edges recording operation jumps.

[0011] The semantic enhancement and knowledge enrichment steps, based on the lightweight large language model LLM and contextualized prompt templates, enrich the initial knowledge graph. Deep semantic completion is performed using LLM to infer the functional semantic roles of nodes based on component attributes, and the node text is normalized to generate a set of equivalent semantics to support fuzzy matching; this is combined with the initial knowledge graph. The structure is determined by classifying interface types using LLM to identify whether they belong to the normal, temporary, or recurring categories. Combining interface type, equivalent semantics, and functional semantic roles, LLM is used to infer logical relationships between components and cross-page functional dependencies not directly observed during exploration, thereby completing the initial knowledge graph. Based on the knowledge relationships, a semantically enhanced complete knowledge graph is obtained. ;

[0012] Dynamic content processing steps, within a complete knowledge graph The template instance separation representation mechanism is introduced to identify complete knowledge graphs. The dynamic region serves as a structural template node. It records its fixed layout pattern, component roles, and semantic tags; specific dynamic items are recorded as runtime instances. It includes real-time changing text and image features, and uses a dynamic binding mechanism to associate them with corresponding template nodes. Related;

[0013] The instruction parsing and graph-constrained reasoning steps involve receiving natural language instructions from the user. It then analyzes and performs multi-dimensional semantic grounding based on a complete knowledge graph. It calculates natural language instructions by comprehensively considering multiple dimensions such as text similarity, semantic embedding relevance, component role compatibility, and interface context. Matching score with candidate UI elements Select the candidate UI element with the highest score as the target component; perform graph-based action planning, modeling the task completion process as... The path search problem on the surface is to search for the shortest path consisting of legal action edges, starting from the current state node and ending at the ground target node, and generate a sequence of instruction operations.

[0014] The execution and monitoring steps involve converting the abstract action sequence of the instruction operation sequence into a system-level input event comprising multiple steps for automatic execution on the mobile device. During execution, changes in interface information are monitored, and after each step, the current control tree is re-acquired, the page signature is calculated, and compared with the expected state. If unexpected pop-ups, redirection failures, or timeouts are detected, the module automatically triggers an exception handling mechanism, taking measures such as closing the pop-up, retrying the operation, or replanning the path to complete the natural language instruction. .

[0015] The aforementioned mobile software automation method based on semantic knowledge graphs and LLM-enhanced reasoning includes the following execution and monitoring steps:

[0016] The incremental maintenance step of the knowledge graph continuously feeds back and updates newly discovered interface states, components, and interaction relationships to the complete knowledge graph during this automated execution process. This enables online learning and incremental expansion of the graph, allowing the system to continuously evolve.

[0017] The mobile software automation method based on semantic knowledge graph and LLM-enhanced reasoning includes the following initial knowledge graph construction steps: after exploring the content and function of each visited page in the interface information, to ensure the initial knowledge graph... To ensure accuracy and effectiveness, the interface information is pruned, including merging and integrating information at the same level through text merging, passing dynamic states through attribute inheritance mechanisms, and removing invalid nodes with empty text, garbled characters, and abnormal coordinates.

[0018] The mobile software automation method based on semantic knowledge graph and LLM-enhanced reasoning includes the following initial knowledge graph construction steps:

[0019] Initial knowledge graph Middle Node Set edge set , These are the structural edges, i.e., which nodes are contained in the page. It's an action edge; when the app runs, it displays the app's initial launch page. Included in the exploration queue ,Right now Initialize the set of signatures for visited pages. Action types include click, text input, swipe, and back operations;

[0020] queue Each page in Perform the following operation: Extract the control tree via system-level commands. Based on the layout structure, text content, and component type, the unique page signature is calculated as follows:

[0021]

[0022] in, Page A unique structural signature used for page deduplication and identity verification; Page The text token set is generated from the normalized text of the visible components; Page The layout hash value, through the control tree The hash calculation is used to capture the parent-child hierarchy and sorting characteristics of components; Page The component role set is extracted from component attributes and determined by LLM to represent the functional category of the component;

[0023] The similarity of page signatures is calculated using a weighted combination formula to determine whether a page has been visited.

[0024]

[0025] in, Page signature With Page signature The similarity between them ranges from [0,1]. These are the weight coefficients for text similarity, layout consistency, and character similarity, respectively, satisfying... ; This represents the Jaccard similarity function, used to calculate the degree of overlap between two sets; Pages respectively A collection of text tokens; Pages respectively The layout hash value; 1[⋅] is an indicator function that returns 1 when the condition in parentheses is true, otherwise returns 0; Pages respectively The set of component roles; if the similarity reaches a preset threshold, it is determined to be a duplicate page and subsequent processing is skipped; otherwise, the page signature is added to the visited set. ;

[0026] Enumerate the set of operable actions on the current page. Perform the actions one by one. Observe the generated new page The system uses heuristic rules to identify temporary pages such as pop-ups and permission prompts. It has a limited number of operable components, includes text indicating cancellation or permission, and allows users to return to the original page after executing a back action. If it is a temporary page, the redirection relationship is not recorded; otherwise, the redirection relationship is recorded. ), and will Joining the team The system returns to the page. ;

[0027] Based on the above exploration results, an initial knowledge graph was constructed. For each node corresponding to a unique page, the page signature is used as the unique identifier. For each visible and interactive component node in the control tree, the following conditions must be met:

[0028]

[0029] in Display text for the component. As a unique identifier for the component, For the component's coordinate range, These are the interactive properties of the component. This is a component type property.

[0030] The mobile software automation method based on semantic knowledge graphs and LLM-enhanced reasoning includes the following steps:

[0031] Introducing a lightweight large language model and contextualized prompt templates to the initial knowledge graph. Semantic completion is performed on the nodes and edges in the data; for each component node... Based on its original properties Functional semantic roles can be inferred using the following formula:

[0032]

[0033] in, Represents the primitive properties of a given component At that time, the component belongs to the conditional probability distribution of role c; c represents the predefined component functional role; This represents the set of primitive properties of component v, including Basic attributes; This represents a large language model used for semantic reasoning based on raw attributes. It identifies the role with the highest confidence level. And the confidence value is stored in the node attribute;

[0034] The text attributes of the component are normalized by... A set of semantically equivalent aliases is generated to support subsequent fuzzy matching; combining the page's structural features and navigation history, the page is divided into three categories—normal, temporary, and recurring—using a formula. It will also infer logical relationships between components and cross-page functional dependencies that were not directly observed during the exploration process, write these semantic attributes and completion relationships into a graph database, and finally form an augmented knowledge graph. .

[0035] The mobile software automation method based on semantic knowledge graphs and LLM-enhanced reasoning includes the following dynamic content processing steps:

[0036] Traversing the complete knowledge graph If multiple nodes in a given area share the same parent node, layout structure, and role type, differing only in text and image attributes, and the area supports scrolling, then this area is considered a dynamic region. A template node T is created for each dynamic region, and the template node is stored in... Remove duplicate instance nodes in the dynamic area, retaining only template nodes to control the graph size; during the automated execution of the execution and monitoring steps, extract dynamic instances from the current page in real time via system-level commands. Instances are matched using structural similarity. Bind to the corresponding template node T; when the user instruction involves a dynamic target, calculate the matching score between the instance and the instruction using a formula:

[0037]

[0038] in Represents a dynamic instance Query target with user command Match score; This represents a text similarity function used to calculate instance text. With the query target The degree of text overlap; Represents a dynamic instance Dynamic text properties; This represents a semantic feature similarity function used to calculate the semantic features of instances. With the query target semantic relevance; Represents a dynamic instance The set of semantic features; Representation of instances The positional weights in the dynamic area are assigned decreasing weights according to the display order, and the instance with the highest score is selected as the interaction target.

[0039] The mobile software automation method based on semantic knowledge graphs and LLM-enhanced reasoning includes the following steps: instruction parsing and graph-constrained reasoning.

[0040] Based on a complete knowledge graph Retrieve the interaction target, perform multi-dimensional semantic grounding, and calculate the matching score between candidate nodes and instructions using a formula: ,in, This represents candidate nodes retrieved from the knowledge graph; This represents the standardized user command query target; The weight coefficients are for text similarity, semantic embedding similarity, role compatibility, and context relevance, respectively; the node with the highest score is selected as the grounding target node. ;

[0041] Model the task completion process as The path search problem, based on the node corresponding to the current UI state. Starting from the grounding target node With the destination as the endpoint, the shortest path search algorithm is used to search for the optimal path composed of legal action edges, and the path is transformed into a sequence of instruction operations including click, input, and swipe operations.

[0042] The aforementioned mobile software automation method based on semantic knowledge graphs and LLM-enhanced reasoning includes the following execution and monitoring steps:

[0043] The instruction sequence is transformed into system-level input events. During execution, after each operation is completed, the control tree of the current page is extracted through system-level commands, and the page signature is calculated. , with the expected page signature Perform a comparison; if the conditions are met... sim() represents the page signature similarity function defined above, and θ represents the state matching threshold; then the current operation is determined to be successful and the next action is executed; otherwise, the exception handling mechanism is triggered.

[0044] The exception handling mechanism includes: identifying the exception type; if the exception type is a temporary pop-up, closing it and continuing the original sequence; if the exception type is a jump failure or dynamic instance change, repeating the current action, with the number of retries not exceeding a preset threshold; if a retry fails, it is based on... Replan the path. If the exception type is a timeout exception, terminate execution.

[0045] The task is considered successfully executed when the current page state and the page state corresponding to the target node meet the matching threshold and conform to the core intent of the user's instruction.

[0046] like Figure 7 As shown, this invention also proposes a mobile software automation device based on semantic knowledge graphs and LLM-enhanced reasoning, including:

[0047] The initial knowledge graph construction module extracts interface information from mobile devices running mobile applications (APPs). Based on the content and function of each visited page in the interface information, it explores and obtains a unique page signature for each visited page to construct the initial knowledge graph. The node set V contains page nodes and their interactive component nodes, and the edge set E contains structural edges reflecting parent-child relationships and action edges recording operation jumps.

[0048] The semantic enhancement and knowledge enrichment module, based on the lightweight large language model LLM and contextualized prompt templates, enhances the initial knowledge graph. Deep semantic completion is performed using LLM to infer the functional semantic roles of nodes based on component attributes, and the node text is normalized to generate a set of equivalent semantics to support fuzzy matching; this is combined with the initial knowledge graph. The structure is determined by classifying interface types using LLM to identify whether they belong to the normal, temporary, or recurring categories. Combining interface type, equivalent semantics, and functional semantic roles, LLM is used to infer logical relationships between components and cross-page functional dependencies not directly observed during exploration, thereby completing the initial knowledge graph. Based on the knowledge relationships, a semantically enhanced complete knowledge graph is obtained. ;

[0049] The dynamic content processing module, within the complete knowledge graph The template instance separation representation mechanism is introduced to identify complete knowledge graphs. The dynamic region serves as a structural template node. It records its fixed layout pattern, component roles, and semantic tags; specific dynamic items are recorded as runtime instances. It includes real-time changing text and image features, and uses a dynamic binding mechanism to associate them with corresponding template nodes. Related;

[0050] The instruction parsing and graph constraint reasoning module receives natural language instructions input by the user. It then analyzes and performs multi-dimensional semantic grounding based on a complete knowledge graph. It calculates natural language instructions by comprehensively considering multiple dimensions such as text similarity, semantic embedding relevance, component role compatibility, and interface context. Matching score with candidate UI elements Select the candidate UI element with the highest score as the target component; perform graph-based action planning, modeling the task completion process as... The path search problem on the surface is to search for the shortest path consisting of legal action edges, starting from the current state node and ending at the ground target node, and generate a sequence of instruction operations.

[0051] The execution and monitoring module converts the abstract action sequence of the instruction operation sequence into a system-level input event that includes multiple steps, enabling automatic execution on the mobile device. During execution, it monitors changes in interface information, re-acquires the current control tree and calculates the page signature after each step, comparing it with the expected state. If unexpected pop-ups, redirection failures, or timeouts are detected, the module automatically triggers an exception handling mechanism, taking measures such as closing the pop-up, retrying the operation, or replanning the path to complete the natural language instruction. .

[0052] The present invention also proposes an electronic device, including the aforementioned mobile software automation device based on semantic knowledge graph and LLM enhanced reasoning, which may be connected to an information display device for displaying the execution result of the natural language instruction by means of user-set display parameters, attributes or by means of an artificial intelligence model.

[0053] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning.

[0054] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning.

[0055] As can be seen from the above solutions, the advantages of the present invention are:

[0056] It achieves deep synergy between the semantic capabilities of large language models and the structured constraints of knowledge graphs. Large language models provide powerful understanding capabilities for the rapid construction and semantic enrichment of knowledge graphs, while knowledge graphs provide fact boundaries and navigation structures for the reasoning process of large language models, ensuring high precision and high reliability in the end-to-end automated process from natural language instructions to system-level operations.

[0057] The method of this invention has high interface coverage, high instruction parsing accuracy and reliable task completion rate in real Android applications, providing a well-structured, reusable and easy-to-maintain solution for mobile software automation. Attached Figure Description

[0058] Figure 1 This is an end-to-end execution flowchart of the mobile software automation method described in this invention.

[0059] Figure 2 This paper presents a core data processing chain diagram illustrating how raw UI information is gradually transformed into a sequence of executable actions within the framework of this invention, with a focus on data.

[0060] Figure 3 An automated UI exploration flowchart for building the initial knowledge graph.

[0061] Figure 4 This is a diagram illustrating the core interaction relationships between users, the large language model, and mobile devices.

[0062] Figure 5 The exception handling logic diagram during the execution of the action sequence is explained in detail.

[0063] Figure 6 This is a flowchart of the method of the present invention;

[0064] Figure 7 This is a block diagram of the device of the present invention;

[0065] Figure 8 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0066] Figure 9 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0067] Figure 10 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0068] Figure label:

[0069] A - First electronic device;

[0070] B-Mobile software automation device based on semantic knowledge graph and LLM enhanced reasoning;

[0071] C-Data acquisition equipment;

[0072] D-Information display device;

[0073] 1000 - Second electronic device;

[0074] Ⅰ-Computational Unit;

[0075] II-ROM;

[0076] III-RAM;

[0077] N-bus;

[0078] V-Interface;

[0079] VI - Input Unit;

[0080] VII - Output Unit;

[0081] VIII - Storage medium;

[0082] IX - Communication Unit. Detailed Implementation

[0083] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0084] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0085] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0086] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0087] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0088] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0089] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0090] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0091] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0092] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0093] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0094] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0097] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] This invention provides a mobile software autonomous automation method based on structured semantic knowledge graphs and LLM-enhanced reasoning. The core innovation lies in solving the technical pain points of existing mobile automation technologies, such as poor adaptability to interface changes, inaccurate parsing of natural language instructions, difficulty in handling dynamic content, and difficulty in reusing automated assets, through a collaborative framework of automated UI exploration and structured modeling, semantic knowledge graph enhancement, constrained reasoning and planning, and robust execution monitoring. This achieves high-precision, high-robustness, and maintainable mobile autonomous automation.

[0099] The automated control method of the present invention includes multiple modules, each of which works collaboratively in a closed-loop logic of data acquisition, knowledge construction, semantic enhancement, constraint reasoning, instruction parsing, and execution.

[0100] This approach utilizes ADB (Android Debug Bridge) technology to capture UI information (XML files) from the application. The real-time input of this runtime UI information into the automated UI exploration and initial knowledge graph construction module enables ADB to execute a series of actions, such as input and clicks, achieving system-level interactive command-driven automated exploration. The exploration process employs a hybrid traversal strategy, combining breadth-first and depth-first traversal to cover the application's navigation skeleton and deeper content. For each visited page, its control tree is extracted, and a unique page signature is calculated based on layout structure, text content, and component type to achieve deduplication of interface states and filtering of temporary pages. Based on the exploration results, an initial UI knowledge graph is constructed. The node set V contains page nodes and interactive component nodes, while the edge set E contains structural edges reflecting parent-child relationships and action edges recording navigation. Specifically, each layout element with a specific function on the page is abstracted as a node; for example, an app icon on the page can be abstracted as a corresponding page node.

[0101] The semantic enhancement and knowledge enrichment module receives the initial knowledge graph. This module incorporates a lightweight large language model and contextualized prompt templates to perform deep semantic completion on the knowledge graph. Utilizing LLM (Limited Language Modeling), it infers the functional semantic roles of components based on their text, type, and location attributes, identifying them as "search box," "submit button," or "back icon." Simultaneously, it normalizes the UI text and generates a set of semantically equivalent aliases to support subsequent fuzzy matching. Furthermore, the module combines page structure and navigation history, using LLM to categorize the entire interactive interface as "normal," "temporary," or "cyclical." In addition, it can infer logical connections between components and cross-page functional dependencies not directly observed during exploration, thus completing the knowledge relationships. Finally, the module outputs a complete knowledge graph with comprehensive semantic enhancement. .

[0102] The Dynamic Content Template-Instance Abstraction module is specifically designed to handle dynamically generated lists, cards, and other content within an application. This module introduces a template-instance separation representation mechanism into the knowledge graph; the graph only stores the structural template nodes for dynamic regions. This records its fixed layout pattern, component role, and semantic tags. Specific dynamic items are then recorded as runtime instances. It includes features such as text and images that change in real time, and is associated with corresponding template nodes T through a dynamic binding mechanism. That is, in the dynamic area, instances (dynamic items) are mutable and can be rebound in each update, while those that do not exist can be deleted. This design effectively ensures the compactness and stability of the knowledge graph, while ensuring accurate positioning and interactive operations for dynamic content.

[0103] In this context, a dynamic area refers to a region where the text content (dynamic items) can be changed, but the functionality within the region remains constant. For example, in a social media app's contact list, each contact leads to a message page where users can send messages; these are examples of the same function but different content. Similarly, in entertainment news apps, each news item has different content, but the overall functionality is the same. The area occupied by these dynamic areas on the page remains constant, hence the term "dynamic area."

[0104] The instruction parsing and graph constraint reasoning module receives natural language instructions input by the user. For example, when a user says "Click on the next song," the text sequence of that sentence is a natural language instruction. Its parsing process mainly consists of two steps: first, multi-dimensional semantic grounding, based on a semantic knowledge graph. The system comprehensively considers multiple dimensions, including text similarity, semantic embedding relevance, component role compatibility, and interface context, to calculate the relationship between instructions and candidate UI elements (a complete knowledge graph). The matching score of the middle node is calculated using the following formula: ,in, The total score represents the multi-dimensional matching score between candidate node c and query target q; c represents the candidate node retrieved from the knowledge graph; q represents the standardized user query target. These are the weight coefficients for text similarity, semantic embedding similarity, role compatibility, and context relevance, respectively. This represents a text and alias similarity function, which calculates the degree of overlap between the text / alias of candidate node c and q. The semantic embedding cosine similarity function is used to calculate the semantic vector relevance between candidate node c and q. This represents a role compatibility function that evaluates the degree of match between the functional role of candidate node c and the intent of q. This represents a context-dependent function that measures the relevance between the page containing candidate node c and the current navigation context. This allows for precise identification of the target component or template indicated by the instruction; secondly, graph-based action planning is performed, modeling the task completion process as... The path search problem involves finding the shortest path consisting of valid action edges, starting from the current state node and ending at the destination node, and generating a series of abstract operation sequences such as click, input, and swipe. For example, if there is a social app on the current screen, the current state node is the current page node, which is connected to the social app component node through relationship edges (inclusive). If the user command requests to enter the chat interface with Xiaoming, then Xiaoming in the contacts is the final destination node.

[0105] The robust execution and state monitoring module is responsible for converting the planned abstract action sequence into specific system-level input events, such as simulated touch and text injection, and executing them on mobile devices via ADB or accessibility service interfaces. During execution, this module continuously monitors changes in the interface state, re-acquiring the current control tree, calculating the page signature, and comparing it with the expected state after each operation. If unexpected pop-ups, failed redirects, or timeouts are detected, the module automatically triggers exception handling mechanisms, such as closing pop-ups, retrying operations, or replanning paths, thereby ensuring the reliable completion of tasks in dynamic environments.

[0106] In addition, the system introduces an incremental knowledge graph maintenance module as an optional component, which continuously feeds back and updates the semantic knowledge graph with newly discovered interface states, components, and interaction relationships during automated execution. This enables online learning and incremental expansion of the graph, allowing the system to continuously evolve.

[0107] To ensure the initial knowledge graph To ensure the accuracy and effectiveness of the graph, the UI information (information in the XML file) needs to be pruned and optimized after UI exploration. Specific measures include integrating fragmented information at the same level through text merging, passing dynamic states through attribute inheritance mechanisms, and removing invalid nodes with empty text, garbled characters, and abnormal coordinates, thereby improving the information density and quality of the graph.

[0108] Specifically, this involves: First, performing an invalid node cleanup operation. This involves traversing the component node set generated by UI exploration, verifying the `text`, `context`, and `bounds` properties of each node. If both the `text` and `context` properties are empty strings, or contain unparsed garbled characters, or if the `bounds` property (coordinate range) exceeds the current device screen resolution range, is negative, a non-numeric type, full-screen, or all zeros, then the node is marked as invalid. Subsequently, all component nodes marked as invalid are removed from the initial knowledge graph node set. After invalid node cleanup, fragmented information at the same level is integrated through text merging. This involves traversing the UI control tree (identifying text-type leaf nodes under the same parent node), determining the logical order of text concatenation based on the arrangement order of these text nodes in the control tree or their horizontal / vertical layout order, concatenating the `text` and `context` properties of the fragmented text nodes into a complete text string in this order, creating a new text component node to replace the original fragmented node, while maintaining its hierarchical association with its parent node and removing the original fragmented node. Then, dynamic states are passed through the property inheritance mechanism. The component nodes with parent-child hierarchical relationships in the UI control tree are located, and the fixed properties of the parent node (such as component type and basic layout constraints) are passed to the child node as default properties. If the dynamic properties of the child node, such as clickable and enabled, are missing or not specified, the default dynamic state is inherited from the component node of the same level and type. If the parent node has associated dynamic state rules, the rules are passed to the child node to complete its dynamic state properties.

[0109] The key technical point of this invention lies in the deep synergy between the semantic capabilities of a large language model and the structured constraints of a knowledge graph. The large language model provides powerful understanding capabilities for the rapid construction and semantic enrichment of the knowledge graph, while the knowledge graph provides fact boundaries and navigation structures for the reasoning process of the large language model, ensuring that the end-to-end automated process from natural language instructions to system-level operations has high precision and high reliability.

[0110] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0111] In summary, this invention provides a mobile software automation method based on structured semantic knowledge graphs and LLM-enhanced reasoning. This method transforms dynamically changing and unrepresented mobile interface information into a structured knowledge graph through automated UI exploration and semantic modeling. This enables the large language model to accurately grasp the semantic roles of UI components, page navigation relationships, and interaction logic. Simultaneously, optimization techniques such as page deduplication, semantic enrichment, and relationship completion improve the completeness and reliability of knowledge representation. To further address issues such as dynamic content adaptation, inaccurate natural language command implementation, and reasoning illusions, this invention introduces core technologies such as template-instance abstraction, multi-dimensional semantic matching, and graph-based action planning to impose closed-loop constraints on the entire automation process, ensuring that every step from command parsing to operation execution is consistent with the actual application interface logic. The following section will combine... Figure 1-5 The specific structure and processing flow of this automation method are described in detail.

[0112] Figure 1 This invention demonstrates the application of semantic UI knowledge graphs and... The overall implementation flowchart of the autonomous automation method for mobile software with enhanced reasoning clearly presents the closed loop of the entire process from interface information collection, knowledge modeling, semantic enhancement to instruction execution, and intuitively demonstrates the collaborative logic and data flow relationship of the six major functional modules. Figure 1 The diagram clearly presents the entire closed-loop process from initial knowledge graph construction, semantic completion and knowledge enrichment, dynamic content-instance abstraction, user command parsing, to final command execution. Arrows and module boxes clearly identify the data flow and collaboration logic between the six functional modules: automated UI exploration and initial knowledge graph construction module, semantic enhancement and knowledge enrichment module, dynamic content template-instance abstraction module, command parsing and constraint reasoning module, and robust execution and status monitoring module. This diagram intuitively demonstrates the integrated technical approach of this invention: "data acquisition - knowledge construction - semantic enhancement - constraint reasoning - command parsing - execution monitoring."

[0113] Figure 3 It demonstrates the complete exploration chain, from exploration initialization, page dequeueing, control tree extraction, page signature calculation, deduplication judgment, operable action enumeration and execution, to temporary page filtering, jump relationship recording, and new page enqueueing, intuitively presenting the construction process of the initial knowledge graph under the hybrid traversal strategy.

[0114] like Figure 3 As shown, the UI information during mobile application runtime first enters the automated UI exploration and initial knowledge graph construction module. The core objective is to transform the UI information, which lacks a unified structure, into a standardized initial knowledge graph. , where the node set edge set When the program runs, it first performs exploration initialization, setting the application's initial startup page. Included in the exploration queue ,Right now Initialize the set of signatures for visited pages. The action enumeration types are clearly defined, including four basic interactive operations: click, text input, swipe, and system return. The exploration process adopts a hybrid traversal strategy, combining breadth-first and depth-first traversal. First, breadth-first traversal is used to cover the application's navigation skeleton, and then depth-first traversal is used to explore the deep component structure within the page.

[0115] For each page in the queue Perform the following operation: Extract the control tree via system-level commands. A unique page signature is calculated based on the layout structure, text content, and component type, and it is defined as follows:

[0116]

[0117] in, Page A unique structural signature used for page deduplication and identity verification; Page The text token set is generated from the normalized text of the visible components; Page The layout hash value, through the control tree The hash calculation is used to capture the parent-child hierarchy and sorting characteristics of components; Page The component role set is extracted from component attributes and determined by LLM to represent the functional category of the component.

[0118] The similarity of page signatures is calculated using a weighted combination formula to determine whether a page has been visited.

[0119]

[0120] in, Page signature With Page signature The similarity between them ranges from [0,1]. These are the weight coefficients for text similarity, layout consistency, and character similarity, respectively, satisfying... ; This represents the Jaccard similarity function, used to calculate the degree of overlap between two sets; Pages respectively A collection of text tokens; Pages respectively The layout hash value; 1[⋅] is an indicator function that returns 1 when the condition in parentheses is true, otherwise returns 0; Pages respectively The set of component roles. If the similarity reaches a preset threshold, it is determined to be a duplicate page and subsequent processing is skipped; otherwise, the page signature is added to the visited set. .

[0121] Enumerate the set of operable actions on the current page. Perform the actions one by one. Observe the generated new page The system uses heuristic rules to identify temporary pages such as pop-ups and permission prompts. It has a limited number of operable components, includes text indicating cancellation or permission, and allows users to return to the original page after executing a back action. If it is a temporary page, the redirection relationship is not recorded; otherwise, the redirection relationship is recorded. ), and will Joining the team The system returns to the page. .

[0122] Based on the above exploration results, an initial knowledge graph was constructed. For each node corresponding to a unique page, the page signature is used as the unique identifier. For each visible and interactive component node in the control tree, the following conditions must be met:

[0123]

[0124] in Display text for the component. As a unique identifier for the component, For the component's coordinate range, These are the interactive properties of the component. This is a component type property.

[0125] To improve the quality of the initial knowledge graph, pruning and optimization are necessary: ​​invalid nodes containing empty text, garbled characters, or abnormal coordinates are removed; fragmented information at the same level is integrated through text merging; adjacent text components under the same parent node are integrated according to the page display order; dynamic states are passed through attribute inheritance to ensure the integrity of component attributes; ultimately, a clear and information-dense initial knowledge graph is formed. .

[0126] Initial knowledge graph Once constructed, it is input into the semantic enhancement and knowledge enrichment module. The core objective of this module is to leverage the semantic understanding capabilities of large language models to complete the semantic attributes and logical relationships of the knowledge graph, thereby generating an enhanced knowledge graph. .

[0127] A lightweight large language model and contextualized prompt templates are introduced to semantically complete the nodes and edges in the initial graph. For each component node... Based on its original properties Functional semantic roles can be inferred through formulas:

[0128]

[0129] in, Represents the primitive properties of a given component At that time, the component belongs to the conditional probability distribution of role c; c represents the predefined component functional role; This represents the set of primitive properties of component v, including Basic attributes; This represents a large language model used for semantic reasoning based on raw attributes. It identifies the role with the highest confidence level. The confidence value is stored in the node attribute.

[0130] The text attributes of the component are normalized by... A canonical_label and a set of semantically equivalent aliases are generated to support subsequent fuzzy matching. Combining the page's structural features and navigation history, pages are categorized into three types—normal, temporary, and recurring—using a formula. It will also infer logical relationships between components and cross-page functional dependencies that were not directly observed during the exploration process, and write these semantic attributes and completion relationships into a graph database, ultimately forming a semantically rich augmented knowledge graph that can be efficiently queried by role, text, and relationship type. .

[0131] Figure 2 The document describes in detail the complete data transformation process, starting from the initial input on a mobile webpage, sequentially involving initial knowledge graph construction, knowledge completion and semantic summarization driven by a large language model, construction of the knowledge graph prefix tree, graph constraint decoding, parsing of user natural language commands, and finally generating a structured sequence of action commands. For example... Figure 2 As shown, the Dynamic Content Template-Instance Abstraction Module is specifically designed to handle dynamically generated lists, cards, and other content in the application. Its core objective is to achieve precise interaction with dynamic content while ensuring the compactness and stability of the knowledge graph through a template-instance separation mechanism.

[0132] Traversal In a system, if multiple components share the same parent node, layout structure, and role type, differing only in attributes such as text and image, and their area supports scrolling, then this is considered a dynamic area. In this case, a template node T is created for each dynamic area, and the template node is stored in... Duplicate instance nodes are removed from dynamic areas, retaining only template nodes to control the graph size. During the automated execution phase, dynamic instances on the current page are extracted in real-time via system-level commands. Instances are matched using structural similarity. Bind to the corresponding template node T. When the user instruction involves a dynamic target, calculate the matching score between the instance and the instruction using a formula:

[0133]

[0134] in Represents a dynamic instance Query target with user command The higher the matching score, the higher the matching degree; This represents a text similarity function used to calculate instance text. With the query target The degree of text overlap; Represents a dynamic instance Dynamic text properties; This represents a semantic feature similarity function used to calculate the semantic features of instances. With the query target semantic relevance; Represents a dynamic instance The set of semantic features; Representation of instances The positional weights in the dynamic area are assigned decreasing weights according to the display order, and the instance with the highest score is selected as the interaction target.

[0135] The core objective of the instruction parsing and graph-constrained reasoning module is to transform the natural language instruction I input by the user into an executable sequence of abstract actions. This module preprocesses the natural language instruction I by removing redundant information through word segmentation and semantic normalization, extracting the core intent and key parameters, and transforming colloquial expressions into standardized queries.

[0136] Based on augmented knowledge graph Retrieve candidate nodes that match the instruction intent, perform multi-dimensional semantic grounding, and calculate the matching score between the candidate nodes and the instruction using a formula: ,in, This represents candidate nodes retrieved from the knowledge graph; This represents the standardized user command query target; These are the weight coefficients for text similarity, semantic embedding similarity, role compatibility, and context relevance, respectively. The node with the highest score is selected as the grounding target node. .

[0137] Model the task completion process as The path search problem, based on the node corresponding to the current UI state. Starting from the grounding target node With the destination as the endpoint, the shortest path search algorithm is used to search for the optimal path composed of legal action edges. The path is transformed into an abstract action sequence containing operations such as click, input, and swipe, and the target component, interaction type and parameters of each operation are clearly defined.

[0138] Figure 4 It demonstrates the complete interactive chain where, after a user inputs a natural language command, the command is parsed and planned by a large language model with integrated graph constraint reasoning capabilities, generating an executable operation sequence, and ultimately driving automated execution on a mobile device.

[0139] Figure 5 It elaborates on the closed-loop process from acquiring the action sequence, executing the current action, monitoring the interface status, to status determination, anomaly identification, and classification handling, clarifying the retry limit and the flow relationship of each link to ensure the robustness of the execution process. Figure 5 As shown, the instruction execution and status monitoring module is responsible for transforming abstract action sequences into system-level operations and executing them, while simultaneously monitoring the interface status in real time, handling exceptions, and ensuring reliable task completion. First, the abstract action sequence is transformed into specific system-level input events. During execution, after each step is completed, the control tree of the current page is extracted using system-level commands, and the page signature is calculated. , with the expected page signature Perform a comparison; if the conditions are met... If the current operation is successful, the next action will continue; otherwise, an exception handling mechanism will be triggered. In case of an exception, first, the exception type is identified. If it's a temporary pop-up, it will be closed and the original sequence will continue. If it's a failed redirect or a dynamic instance change, the current action will be repeated, with the number of retries not exceeding a preset threshold. If a retry fails, it will be based on... Replan the path. If a timeout occurs, terminate execution and report the reason for the exception. If the current page state and the page state corresponding to the target node meet the matching threshold and conform to the core intent of the user's command, the task is considered successful. Otherwise, it is considered a failure, and the reason for failure and manual intervention suggestions are output.

[0140] The knowledge graph incremental maintenance module, as an optional component, aims to enable online learning and incremental expansion of the knowledge graph, allowing the system to adapt to iterative updates of the application UI. During automated execution, it continuously monitors the interface status and detects any omissions or gaps in the knowledge graph. For new page signatures or new component nodes, an incremental exploration process is automatically triggered: extracting duplicate control trees from the new page, calculating page signatures, and performing deduplication checks; for new components, a semantic enhancement process is performed, adding new nodes and relationships to the new page. .

[0141] The above description is merely an exemplary embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0142] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0143] This invention also proposes a mobile software automation device based on semantic knowledge graphs and LLM-enhanced reasoning, comprising:

[0144] The initial knowledge graph construction module extracts interface information from mobile devices running mobile applications (APPs). Based on the content and function of each visited page in the interface information, it explores and obtains a unique page signature for each visited page to construct the initial knowledge graph. The node set V contains page nodes and their interactive component nodes, and the edge set E contains structural edges reflecting parent-child relationships and action edges recording operation jumps.

[0145] The semantic enhancement and knowledge enrichment module, based on the lightweight large language model LLM and contextualized prompt templates, enhances the initial knowledge graph. Deep semantic completion is performed using LLM to infer the functional semantic roles of nodes based on component attributes, and the node text is normalized to generate a set of equivalent semantics to support fuzzy matching; this is combined with the initial knowledge graph. The structure is determined by classifying interface types using LLM to identify whether they belong to the normal, temporary, or recurring categories. Combining interface type, equivalent semantics, and functional semantic roles, LLM is used to infer logical relationships between components and cross-page functional dependencies not directly observed during exploration, thereby completing the initial knowledge graph. Based on the knowledge relationships, a semantically enhanced complete knowledge graph is obtained. ;

[0146] The dynamic content processing module, within the complete knowledge graph The template instance separation representation mechanism is introduced to identify complete knowledge graphs. The dynamic region serves as a structural template node. It records its fixed layout pattern, component roles, and semantic tags; specific dynamic items are recorded as runtime instances. It includes real-time changing text and image features, and uses a dynamic binding mechanism to associate them with corresponding template nodes. Related;

[0147] The instruction parsing and graph constraint reasoning module receives natural language instructions input by the user. It then analyzes and performs multi-dimensional semantic grounding based on a complete knowledge graph. It calculates natural language instructions by comprehensively considering multiple dimensions such as text similarity, semantic embedding relevance, component role compatibility, and interface context. Matching score with candidate UI elements Select the candidate UI element with the highest score as the target component; perform graph-based action planning, modeling the task completion process as... The path search problem on the surface is to search for the shortest path consisting of legal action edges, starting from the current state node and ending at the ground target node, and generate a sequence of instruction operations.

[0148] The execution and monitoring module converts the abstract action sequence of the instruction operation sequence into a system-level input event that includes multiple steps, enabling automatic execution on the mobile device. During execution, it monitors changes in interface information, re-acquires the current control tree and calculates the page signature after each step, comparing it with the expected state. If unexpected pop-ups, redirection failures, or timeouts are detected, the module automatically triggers an exception handling mechanism, taking measures such as closing the pop-up, retrying the operation, or replanning the path to complete the natural language instruction. .

[0149] like Figure 8 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, which includes the aforementioned mobile software automation device based on semantic knowledge graph and LLM enhanced reasoning.

[0150] like Figure 9 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect natural language commands, such as "change to the next song" or "send XX information to contact Xiaoming". The information display device D is used to display the execution results of the natural language commands obtained by the present invention.

[0151] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with the specified key information based on user-defined display parameters and / or attributes, such as the data range and font, color, and scrolling options. Users can access this information more quickly without needing to navigate to secondary pages or scroll through pages, saving them time and effort. Alternatively, the information display mechanism can be an artificial intelligence (AI) display model that learns the user's key information interests based on past usage habits, such as viewing time, click count, and edit count, and automatically presents rich and necessary key information.

[0152] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the semantic knowledge graph-based and LLM-enhanced reasoning methods provided by the above methods.

[0153] In another embodiment of the present invention, a storage medium VIII is provided for storing a computer program that executes the aforementioned semantic knowledge graph and LLM-based enhanced reasoning method. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0154] Figure 10 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0155] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0156] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0157] The computing unit I can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S5. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0158] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A mobile software automation method based on semantic knowledge graph and LLM-enhanced reasoning, characterized in that, include: The initial knowledge graph construction steps involve extracting interface information from mobile devices running mobile applications (APPs), and based on the content and function of each visited page in the interface information, exploring to obtain a unique page signature for each visited page in order to construct the initial knowledge graph. The node set V contains page nodes and their interactive component nodes, and the edge set E contains structural edges reflecting parent-child relationships and action edges recording operation jumps. The semantic enhancement and knowledge enrichment steps, based on the lightweight large language model LLM and contextualized prompt templates, enrich the initial knowledge graph. Deep semantic completion is performed using LLM to infer the functional semantic roles of nodes based on component attributes, and the node text is normalized to generate a set of equivalent semantics to support fuzzy matching; this is combined with the initial knowledge graph. The structure is determined by classifying interface types using LLM to identify whether they belong to the normal, temporary, or recurring categories. Combining interface type, equivalent semantics, and functional semantic roles, LLM is used to infer logical relationships between components and cross-page functional dependencies not directly observed during exploration, thereby completing the initial knowledge graph. Based on the knowledge relationships, a semantically enhanced complete knowledge graph is obtained. ; Dynamic content processing steps, within a complete knowledge graph The template instance separation representation mechanism is introduced to identify complete knowledge graphs. The dynamic region serves as a structural template node. It records its fixed layout pattern, component roles, and semantic tags; specific dynamic items are recorded as runtime instances. It includes real-time changing text and image features, and uses a dynamic binding mechanism to associate them with corresponding template nodes. Related; The instruction parsing and graph-constrained reasoning steps involve receiving natural language instructions from the user. It then analyzes and performs multi-dimensional semantic grounding based on a complete knowledge graph. It calculates natural language instructions by comprehensively considering multiple dimensions such as text similarity, semantic embedding relevance, component role compatibility, and interface context. Matching score with candidate UI elements Select the candidate UI element with the highest score as the target component; perform graph-based action planning, modeling the task completion process as... The path search problem on the surface is to search for the shortest path consisting of legal action edges, starting from the current state node and ending at the ground target node, and generate a sequence of instruction operations. The execution and monitoring steps involve converting the abstract action sequence of the instruction operation sequence into a system-level input event comprising multiple steps for automatic execution on the mobile device. During execution, changes in interface information are monitored, and after each step, the current control tree is re-acquired, the page signature is calculated, and compared with the expected state. If unexpected pop-ups, redirection failures, or timeouts are detected, the module automatically triggers an exception handling mechanism, taking measures such as closing the pop-up, retrying the operation, or replanning the path to complete the natural language instruction. .

2. The mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning as described in claim 1, characterized in that, The execution and monitoring steps include: The incremental maintenance step of the knowledge graph continuously feeds back and updates newly discovered interface states, components, and interaction relationships to the complete knowledge graph during this automated execution process. This enables online learning and incremental expansion of the graph, allowing the system to continuously evolve.

3. The mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning as described in claim 1, characterized in that, The initial knowledge graph construction steps include: after exploring the content and functions of each visited page in the interface information, to ensure the initial knowledge graph... To ensure accuracy and effectiveness, the interface information is pruned, including merging and integrating information at the same level through text merging, passing dynamic states through attribute inheritance mechanisms, and removing invalid nodes with empty text, garbled characters, and abnormal coordinates.

4. The mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning as described in claim 1, characterized in that, The initial knowledge graph construction steps include: Initial knowledge graph Middle Node Set edge set , For structural edges, It's an action edge; when the app runs, it displays the app's initial launch page. Included in the exploration queue ,Right now Initialize the set of signatures for visited pages. Action types include click, text input, swipe, and back operations; queue Each page in Perform the following operation: Extract the control tree via system-level commands. Based on the layout structure, text content, and component type, the unique page signature is calculated as follows: in, Page A unique structural signature used for page deduplication and identity verification; Page The text token set is generated from the normalized text of the visible components; Page The layout hash value, through the control tree The hash calculation is used to capture the parent-child hierarchy and sorting characteristics of components; Page The component role set is extracted from component attributes and determined by LLM to represent the functional category of the component; The similarity of page signatures is calculated using a weighted combination formula to determine whether a page has been visited. in, Page signature With Page signature The similarity between them ranges from [0,1]. These are the weight coefficients for text similarity, layout consistency, and character similarity, respectively, satisfying... ; This represents the Jaccard similarity function, used to calculate the degree of overlap between two sets; Pages respectively A collection of text tokens; Pages respectively The layout hash value; 1[⋅] is an indicator function that returns 1 when the condition in parentheses is true, otherwise returns 0; Pages respectively The set of component roles; if the similarity reaches a preset threshold, it is determined to be a duplicate page and subsequent processing is skipped; otherwise, the page signature is added to the visited set. ; Enumerate the set of operable actions on the current page. Perform the actions one by one. Observe the generated new page The system uses heuristic rules to identify temporary pages such as pop-ups and permission prompts. It has a limited number of operable components, includes text indicating cancellation or permission, and allows users to return to the original page after executing a back action. If it is a temporary page, the redirection relationship is not recorded; otherwise, the redirection relationship is recorded. ), and will Joining the team The system returns to the page. ; Based on the above exploration results, an initial knowledge graph was constructed. For each node corresponding to a unique page, the page signature is used as the unique identifier. For each visible and interactive component node in the control tree, the following conditions must be met: in Display text for the component. As a unique identifier for the component, For the component's coordinate range, These are the interactive properties of the component. This is a component type property.

5. The mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning as described in claim 4, characterized in that, Semantic enhancement and knowledge enrichment steps include: Introducing a lightweight large language model and contextualized prompt templates to the initial knowledge graph. Semantic completion is performed on the nodes and edges in the data; for each component node... Based on its original properties Functional semantic roles can be inferred using the following formula: in, Represents the primitive properties of a given component At that time, the component belongs to the conditional probability distribution of role c; c represents the predefined component functional role; This represents the set of primitive properties of component v, including Basic attributes; This represents a large language model used for semantic reasoning based on raw attributes. It identifies the role with the highest confidence level. And the confidence value is stored in the node attribute; The text attributes of the component are normalized by... A set of semantically equivalent aliases is generated to support subsequent fuzzy matching; combining the page's structural features and navigation history, the page is divided into three categories—normal, temporary, and recurring—using a formula. It will also infer logical relationships between components and cross-page functional dependencies that were not directly observed during the exploration process, write these semantic attributes and completion relationships into a graph database, and finally form an augmented knowledge graph. .

6. The mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning as described in claim 5, characterized in that, Dynamic content processing steps include: Traversing the complete knowledge graph If multiple nodes in a given area share the same parent node, layout structure, and role type, differing only in text and image attributes, and the area supports scrolling, then this area is considered a dynamic region. A template node T is created for each dynamic region, and the template node is stored in... Remove duplicate instance nodes in the dynamic area, retaining only template nodes to control the graph size; during the automated execution of the execution and monitoring steps, extract dynamic instances from the current page in real time via system-level commands. Instances are matched using structural similarity. Bind to the corresponding template node T; when the user instruction involves a dynamic target, calculate the matching score between the instance and the instruction using a formula: in Represents a dynamic instance Query target with user command Match score; This represents a text similarity function used to calculate instance text. With the query target The degree of text overlap; Represents a dynamic instance Dynamic text properties; This represents a semantic feature similarity function used to calculate the semantic features of instances. With the query target semantic relevance; Represents a dynamic instance The set of semantic features; Representation of instances The positional weights in the dynamic area are assigned decreasing weights according to the display order, and the instance with the highest score is selected as the interaction target.

7. The mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning as described in claim 6, characterized in that, The steps of instruction parsing and graph constraint reasoning include: Based on a complete knowledge graph Retrieve the interaction target, perform multi-dimensional semantic grounding, and calculate the matching score between candidate nodes and instructions using a formula: ,in, This represents candidate nodes retrieved from the knowledge graph; This represents the standardized user command query target; The weight coefficients are for text similarity, semantic embedding similarity, role compatibility, and context relevance, respectively; the node with the highest score is selected as the grounding target node. ; Model the task completion process as The path search problem, based on the node corresponding to the current UI state. Starting from the grounding target node With the destination as the endpoint, the shortest path search algorithm is used to search for the optimal path composed of legal action edges, and the path is transformed into a sequence of instruction operations including click, input, and swipe operations.

8. The mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning as described in claim 7, characterized in that, The execution and monitoring steps include: The instruction sequence is transformed into system-level input events. During execution, after each operation is completed, the control tree of the current page is extracted through system-level commands, and the page signature is calculated. , with the expected page signature Perform a comparison; if the conditions are met... sim() represents the page signature similarity function defined above, and θ represents the state matching threshold; then the current operation is determined to be successful and the next action is executed; otherwise, the exception handling mechanism is triggered. The exception handling mechanism includes: identifying the exception type; if the exception type is a temporary pop-up, closing it and continuing the original sequence; if the exception type is a jump failure or dynamic instance change, repeating the current action, with the number of retries not exceeding a preset number; if the retrieval fails, it is based on... Replan the path. If the exception type is a timeout exception, terminate execution. The task is considered successfully executed when the current page state and the page state corresponding to the target node meet the matching threshold and conform to the core intent of the user's instruction.

9. A mobile software automation device based on semantic knowledge graph and LLM enhanced reasoning, characterized in that, include: The initial knowledge graph construction module extracts interface information from mobile devices running mobile applications (APPs). Based on the content and function of each visited page in the interface information, it explores and obtains a unique page signature for each visited page to construct the initial knowledge graph. The node set V contains page nodes and their interactive component nodes, and the edge set E contains structural edges reflecting parent-child relationships and action edges recording operation jumps. The semantic enhancement and knowledge enrichment module, based on the lightweight large language model LLM and contextualized prompt templates, enhances the initial knowledge graph. Deep semantic completion is performed using LLM to infer the functional semantic roles of nodes based on component attributes, and the node text is normalized to generate a set of equivalent semantics to support fuzzy matching; this is combined with the initial knowledge graph. The structure is determined by classifying interface types using LLM to identify whether they belong to the normal, temporary, or recurring categories. Combining interface type, equivalent semantics, and functional semantic roles, LLM is used to infer logical relationships between components and cross-page functional dependencies not directly observed during exploration, thereby completing the initial knowledge graph. Based on the knowledge relationships, a semantically enhanced complete knowledge graph is obtained. ; The dynamic content processing module, within the complete knowledge graph The template instance separation representation mechanism is introduced to identify complete knowledge graphs. The dynamic region serves as a structural template node. It records its fixed layout pattern, component roles, and semantic tags; specific dynamic items are recorded as runtime instances. It includes real-time changing text and image features, and uses a dynamic binding mechanism to associate them with corresponding template nodes. Related; The instruction parsing and graph constraint reasoning module receives natural language instructions input by the user. It then analyzes and performs multi-dimensional semantic grounding based on a complete knowledge graph. It calculates natural language instructions by comprehensively considering multiple dimensions such as text similarity, semantic embedding relevance, component role compatibility, and interface context. Matching score with candidate UI elements Select the candidate UI element with the highest score as the target component; perform graph-based action planning, modeling the task completion process as... The path search problem on the surface is to search for the shortest path consisting of legal action edges, starting from the current state node and ending at the ground target node, and generate a sequence of instruction operations. The execution and monitoring module converts the abstract action sequence of the instruction operation sequence into a system-level input event that includes multiple steps, enabling automatic execution on the mobile device. During execution, it monitors changes in interface information, re-acquires the current control tree and calculates the page signature after each step, comparing it with the expected state. If unexpected pop-ups, redirection failures, or timeouts are detected, the module automatically triggers an exception handling mechanism, taking measures such as closing the pop-up, retrying the operation, or replanning the path to complete the natural language instruction. .

10. An electronic device, characterized in that, The device includes a mobile software automation device based on semantic knowledge graph and LLM enhanced reasoning as described in claim 9. The electronic device may be connected to an information display device, which is used to display the execution result of the natural language instruction by means of user-set display parameters, attributes or by means of an artificial intelligence model.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the mobile software automation method based on semantic knowledge graph and LLM enhanced reasoning as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the mobile software automation method based on semantic knowledge graphs and LLM-enhanced reasoning as described in any of claims 1-8.

Images