Interactive GUI (Graphical User Interface) intelligent agent system for cross-application control of mobile equipment
By leveraging the global task management and collaborative work of the interactive GUI intelligent agent system's application-level task execution modules, the shortcomings of smartphones in executing complex tasks are addressed. This enables cross-application task decomposition and self-optimization, thereby improving the automation level and user experience of mobile devices.
Patent Information
- Application Number
- CN202511353025.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-22
- Publication Date
- 2025-12-19
AI Technical Summary
Existing smartphones are inadequate in automating complex tasks, struggling to handle the ever-changing page layouts on mobile devices and the complex and varied needs of users. Furthermore, the interaction between the intelligent agent and the user is not close enough, making it unable to effectively complete complex tasks across applications.
An interactive GUI intelligent agent system was designed, which achieves cross-application task decomposition, execution, and self-optimization through the collaborative work of a global task management module, an application-level task execution module, and a self-learning module. Utilizing a large language model and multiple types of intelligent agents, the system can understand natural language tasks, perform cross-application operations, and possess self-learning capabilities.
It enables seamless execution of complex tasks across applications on mobile devices, improves automation levels, ensures the accuracy and robustness of task execution, dynamically adapts to application updates, and enhances the user experience.
Smart Images

Figure CN121168504A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of artificial intelligence and mobile computing technology, and specifically relates to an intelligent agent system based on a graphical user interface (GUI), especially for a system for cross-application (App) automation control of a mobile device. BACKGROUND
[0002] Smart mobile devices such as smartphones have become an important tool for people's daily work and life. However, the existing smartphones have obvious deficiencies in the automation execution of complex tasks. Users expect to have a super intelligent agent to assist them in completing diversified and complex operation tasks, rather than the limited functions of the current voice assistants on the market.
[0003] In recent years, the rapid development of large language models and multi-modal large models has greatly improved the capabilities of mobile agents, but the current research on mobile agents has not fully solved the challenges faced by mobile tasks in the real world. On the one hand, mobile applications have a rich and varied page layout and continuous function updates, which pose high requirements for the intelligent agent to accurately identify, understand the graphical user interface, plan and execute specific operations. On the other hand, mobile users have complex and variable needs, which are often difficult to present at once, which means that the intelligent agent needs to maintain close communication with the user and shuttle between multiple application programs to complete the user's needs. SUMMARY
[0004] The purpose of the present application is to provide an interactive GUI intelligent agent system that can serve as a smart assistant for mobile devices. This system can achieve cross-application control and interact with users on demand, while gradually accumulating knowledge of mobile applications during task execution to achieve continuous evolution.
[0005] The present application involves three levels of technical problems: at the high level, the user's task needs to be decomposed into specific tasks that can be executed in a single mobile application; at the middle level, the task needs to be further decomposed into multiple sub-goals in combination with the knowledge of a specific mobile application; at the bottom level, the sub-goals need to be further converted into specific atomic operations and parameters.
[0006] The interactive GUI intelligent agent system for cross-application control of mobile devices provided by the present application utilizes large language models (LLM) and multiple types of agents (Agent) to work together to achieve understanding of user natural language tasks, cross-application task decomposition and execution, and continuous accumulation and self-optimization of knowledge. It can execute complex tasks across applications (App) on mobile devices and has self-learning and user interaction capabilities; including:
[0007] (1) a global task management module configured to receive a user input natural language task, and to decompose the natural language task into at least one sub-task instruction;
[0008] (ii) Application-level task execution module, in communication connection with the global task management module, configured to receive the sub-task instructions and decide and execute corresponding operations on the application interface of the mobile device according to the sub-task instructions;
[0009] (iii) Self-learning module, in communication connection with the application-level task execution module, configured to obtain the task execution history record of the application-level task execution module and learn the application knowledge according to the history record to generate a knowledge graph or an experience rule. See Figure 1 ;
[0010] Specifically,
[0011] (i) Global task management module, which decomposes user instructions into single-application tasks and manages the life cycle of the "application-level task execution module". The process is shown in Figure 2 . The global task management module includes three sub-modules: a global task planning agent for analyzing the natural language task and application information, generating or adjusting the task plan, and outputting the sub-task instructions; an application information manager for obtaining the installed application information in the user device; and a task manager for maintaining the life cycle of the sub-task planned by the global task planning agent and assisting in context transfer when the sub-task switches.
[0012] The specific content is as follows:
[0013] (1) Global task planning agent responsible for decomposing the user's complex instructions into multiple single-application tasks, divided into two stages of direct planning and adjustment:
[0014] (a) Direct planning: According to the user instruction I and the metadata of the m installed applications in the user's mobile phone formulate a global plan G and determine the execution of the first sub-task. Its formal representation is formula (1):
[0015]
[0016] G includes the overall plan G op , the sub-task G st to be executed next, and the context information G c needed for the execution of this task. Among them, the sub-task G st includes the unprocessed task instruction I T′ , the context extraction request G cr , and the App package name AM p corresponding to the task.
[0017] (b) Adjustment: This involves: a. checking the execution results of previous tasks; b. updating the task progress based on the execution results, or adjusting the previous task plan. Based on user command I and the trajectory record of the "application-level task execution module" which has undergone multiple rounds of action loops and user interaction loops. Determine if the task results meet expectations; then, based on the inspection results and the previous global plan G, update the overall plan G. op And determine the next subtask G st At the same time, summarize the context information G that needs to be passed to the next subtask. c Its formal representation can be expressed as formula (2):
[0018]
[0019] (2) Application Information Manager It is a tool for discovering, maintaining, and summarizing user equipment. Metadata of installed applications (call) Get the list of applications), for Provide a reliable base table of App capabilities, which will be dynamically updated as users' mobile applications change. This can be formally represented as formula (3):
[0020]
[0021] (3) Task Manager It is a tool used to maintain... The planned lifecycle of subtasks assists in context passing during subtask switching.
[0022] After receiving the current G, it will call Launch the application. It is an action executor tool. At the same time, LLM based on context information G c Raw Task Instruction I T′ Rewrite it. After the task is issued, This will activate and begin processing the task. It can be formally represented as formula (4):
[0023]
[0024] (ii) The application-level task execution module is the core of executing single application tasks planned by the "global task management module". It includes three sub-modules: action loop agent (group) for planning, decision-making and executing GUI operations according to the sub-task instructions; user interaction loop agent (group) for generating user interaction interfaces, collecting and parsing user input when needed; and short-term memory module for storing context information during the current task execution process for the action loop agent group and interaction loop agent group to call.
[0025] (A) The Action Loop Agent (Group) comprises five sub-modules: a Planner Agent, used to formulate an execution plan containing multiple sub-goals based on sub-tasks, and to reflect on or replan based on the execution results; an Action Decision Maker Agent, used to decide on the sequence of atomic operations based on the sub-goals; an Action Executor Tool, used to send the atomic operations to the mobile device for execution; a Screen Sensor Tool, used to acquire screenshots and UI hierarchy information of the mobile device; and a Key Context Extractor Agent, used to extract key context information from the screen for reference by the Planner and Executor. This Action Loop Agent (Group) is responsible for executing tasks in the "Application-Level Task Execution Module." These agents and tools continuously generate atomic operations, execute them, and observe the execution status in a loop called the "Action Loop" until the task is completely completed or it needs to enter the "User Interaction Loop." See the flowchart below. Figure 3 .
[0026] The details are as follows:
[0027] (1) Planner intelligent agent Responsible for according to The established tasks involve developing specific execution plans comprised of a series of sub-objectives. similar, It is also divided into two phases: direct planning and adjustment.
[0028] (a) Direct planning: According to task instruction I T The first screen-sensing information S 0 Tips and tricks Develop a plan P and determine the first sub-objective to be executed. This can be formally represented as formula (5).
[0029]
[0030] P includes the overall plan P Op The subtask P that will be executed next Sg .
[0031] (b) Adjustment: a. check the result of the action execution; b. update the task progress or adjust the previous task plan according to the execution result. First, According to the task instruction I T , the screen perception information S of the previous round and the execution record of the current round Check if the current screen change meets the action expectation and track the progress of the current subtask P Sg Output the reflection result R of the action execution of the current round. Next, according to the reflection result R and the previous plan P, key context information C, Update the overall plan P Op And determine the next subtask P Sg . Finally, Judge whether the user needs to interact with the user, output the user interaction request D R . Its formal representation is formula (6):
[0032]
[0033] D R includes the type of user interaction D Ic and the thought D It ; R includes the action result R Ar and the plan progress R Pr . Wherein, R Ar is an enumeration of the following values: A: the subgoal has been completed; B: the subgoal is partially completed; C: the action produces unexpected results; D: the action does not produce any changes. When the action result is not as expected (i.e. C, D), the potential error cause R Ec is also included for error correction.
[0034] When R Ar is A, The next subgoal will be selected from P Op for execution; otherwise, the current subgoal will not be adjusted. When R Ar of the current subgoal is C or D for three consecutive times, P Op will be adjusted and the subgoal will be reselected, and the adjustment will not change the completed part.
[0035] Support "independent" mode, in which mode, reflection and planning will be completed in two agents. Its responsibilities a, b will be allocated to the reflection module and the planning module This mode will reduce the overall execution speed, and can limitedly improve its stability and accuracy.
[0036] (2) Action decision maker agent is responsible for making decisions according to The formulated sub-goal plans the specific atomic operation. First, the action result R of the last action is obtained Ar , and the action decision prompt or the error recovery prompt is obtained T , the plan P, the current screen awareness information S, the last n times of execution records and the key context information C, and the atomic action sequence to be executed is planned and the action decision A is made. Its formal expression is formula (7):
[0037]
[0038] A includes the atomic action sequence A As and the action expected result A Er . A As is composed of one or more atomic operations. If the atomic operation needs parameters when it is executed, the parameters are also included in the sequence.
[0039] (3) Action executor tool is a tool for executing the atomic action sequence generated by The following atomic operations can be executed on the user equipment through the bridge library:
[0040] a. “Tap”: the operation parameter is x, y, and the phone screen coordinates (x, y) are tapped;
[0041] b. “Swipe”: the operation parameter is x1, y1, x2, y2, dur, and the phone screen coordinates (x1, y1) are swiped to the phone screen coordinates (x2, y2) within dur seconds;
[0042] c. “LongPress”: the operation parameter is x, y, dur, and the phone screen coordinates (x, y) are long pressed, and then released after dur seconds;
[0043] d. “Input”: the operation parameter is text, and text is input in the activated input box;
[0044] e. “ClearInput”: no operation parameter, and the activated input box is cleared;
[0045] f. “KeyEvent”: the operation parameter is event, and the Android key event event is sent to the phone, such as ‘BCAK’, ‘HOME’, etc. event;
[0046] g. “Wait”: the operation parameter is dur, and dur seconds are waited;
[0047] h. “Finish”: No operation parameter, used to mark that the task has been completed and no other action is needed;
[0048] i. “NeedInteraction”: No operation parameter, used to mark that user interaction is needed at present, rather than other actions;
[0049] j. “ListApps”: No operation parameter, lists all third-party applications installed on the device;
[0050] k. “StartApp”: The operation parameter is package, which starts a specific application with the package name package on the phone.
[0051] (4) Screen perceiver tool is a tool for obtaining and perceiving the current screen information. The screenshot S Sp and the Accessibility Tree (AT) XML S At can be obtained through the bridge library. Supports visual and non-visual
[0052] Two modes:
[0053] a. In visual mode, “Clickable” and “Scrollable” components will be extracted from , projected on according to their bounds attributes, and assigned a serial number. The mapping relationship between the serial number and the center coordinates (Set of Mark, SoM) is stored as In this mode, a coordinate-to-mark conversion will be performed based on before being handed over to for execution.
[0054] b. In non-visual mode, use the “AT processing algorithm” to remove useless nodes and attributes, complete the description information of the picture node through the MLLM model, merge and summarize the “Clickable” component subnodes, and finally output as text in md format.
[0055] Perceived information
[0056] The flowchart is shown in Figure 4 . Specifically, obtain the screenshot and AT through the bridge library, and after obtaining, the AT will be parsed and converted into a dictionary (Dict) for subsequent processing. The AT will be traversed to obtain the actionable nodes (i.e. nodes including clickable or scrollable properties) therein, and they will be recorded. For the visual mode, these nodes will be sequentially assigned with serial numbers, while the numbers and the corresponding center point coordinates (for clickable nodes) or bounding box coordinates (for scrollable nodes) will be recorded to Next, a drawing tool will draw the boxes of these components on the screenshot. The screenshot will be passed to the large model for action decision making later, and once the decision is made, the marker-coordinate converter can convert the marker-based decision information to coordinate-based decision information to control the phone execution. For the non-visual mode, The marker-based decision information will be converted to coordinate-based decision information to control the phone execution. For the non-visual mode, In the traversal of the AT, in addition to obtaining the actionable nodes, (1) the trivial node properties therein will be cleaned up, and only the non-negative values will be retained; and (2) the image nodes (nodes of types ‘View’ and ‘ImageView’) therein will be obtained. Next, the image node description completer will extract the corresponding pictures based on the coordinates of the image nodes, and use the MLLM model to complement their description information in parallel. Meanwhile, the node summarizer will summarize all the child nodes under the actionable nodes in parallel, assign their summarized descriptions to the actionable nodes, and remove all the summarized, non-actionable child nodes. Finally, The AT will be traversed again to compress and merge all the elements that have only one sub-element, and after that, the AT tree will be output as textual screen perception information in Markdown (a lightweight markup language) structure. The information includes the types of nodes, resource IDs, coordinates, key properties, and description information added in the previous steps. After the above processing, the textual screen perception information maintains the same structural information expression capability as the AT or the screenshot, greatly reduces the redundant information compared to the original XML, and the missing information is complemented.
[0057] In the non-visual mode, due to the characteristics of the AT itself, it is possible that components that do not actually exist in the screen (components that are covered and not displayed or available) appear in the screen perception information. To alleviate such problems, The existence of occlusion between components will be checked based on the coordinate information, and secondary verification based on OCR recognition will be performed for components that may have such a situation, so as to exclude nodes that have been occluded. In the visual mode, relying on the AT may cause components to be missed. Some components (such as the slide selector when selecting time) often lack scrollable properties, which will cause them to be excluded when obtaining actionable nodes, and thus not be marked in the screenshot, resulting in the action decision maker being unable to make action decisions for the component. To alleviate such problems, Additional 'View' nodes will be considered and their basic functions will be judged by the multi-modal large model, and they will be supplemented to the list of actionable nodes once missing. In the AT tree, the components at the top level are located further back than those at the lower level, and they will be drawn last. When the drawing tool draws the overlapping box, it will cover the box that has been drawn in front, which will destroy the label of the component located at the lower level so that it can no longer be clicked. To solve the component coverage problem also faced in the mode.
[0058] (5) Key context extractor agent Responsible for extracting key context information from the screen for the planner and executor to reference. In information collection type tasks (e.g., finding nearby Italian restaurants and summarizing reviews), The context information maintained will play an important role. will be based on task instructions I T , context extraction request G cr , plan P, current screen awareness information S and previous context information C t-1 , extract new key context information C t :
[0059]
[0060] At the end of each round of "action cycle" and the result R Ar is A or B, it will be activated. Its actions are parallel to other agents. Because the starting screen awareness information of the current round will contain the screen information of the last round, there is no need to wait for information collection to be completed. This design improves the response speed of the agent.
[0061] (B) User interaction cycle agent (group) is responsible for the core cycle of interaction with the user in the "application-level task execution module", which includes three sub-modules: user interaction agent, which is used to generate user prompts and parse user responses; planner agent, which is used to determine whether user interaction is needed and adjust the task plan according to user feedback; user chat room tool, which is used to display interaction content and collect user input. These agents and tools will continuously interact with the user and observe the execution in a cycle called "interaction cycle" until the user interaction task is completed. Its flow is shown in Figure 5 .
[0062] In the "action cycle", when the user interaction type D It is not 0, the current "action cycle" will be suspended and enter the "interaction cycle" to interact with the user. Among them, D ItIt is an enumeration of the following values: 1: User confirmation is required for sensitive or dangerous operations; 2: User confirmation is required for irreversible operations; 3: User selection is required for different options; 4: User clarification is required for further instructions; 0: No interaction is required. To avoid... We allow for missed user interactions due to illusions. Select the "NeedInteraction" action to correct the error. Once selected, This allows you to reflect on your mistakes in the next cycle and enter an "interactive loop".
[0063] The user interaction loop agent (group) consists of three parts: the user interaction agent, the planner agent, and the user chat room tool, as detailed below:
[0064] (1) User interaction intelligent agent It is responsible for continuously generating user prompts based on the sub-goals set by the planner, receiving and judging whether the user's response has met the requirements, and summarizing the user's dialogue if the requirements are met.
[0065] When D It When the value is not 0, or when a user reply D is received. Ur hour, It will be activated. Based on task instruction I T Plan P, User Interaction Request D R Based on the current screen perception information S and the previous context information C, the current user interaction plan D is generated. If k rounds of interaction have already occurred, It will also read historical interaction records. User prompts in D Up And reply D Ur It can be formally represented as formula (8).
[0066]
[0067] D includes the interaction state D Is And user prompt D Up Among them, D Is It is an enumeration of the following values: 0: further interaction with the user is needed; 1: the user has made a clear choice or clarified an ambiguous statement. When the interaction is complete (i.e., D...),... Is (1) will also include a summary of user responses (D) Us For updating I T use.
[0068] (2) User chat room tools It is a tool used to provide users with information through a UI interface and console. Display user prompt D Upand wait for user input for reply D Ur When interaction state D Is is 0, is activated. At evaluation, the evaluation system can also be accessed for automatic testing. Its formal representation is formula (9):
[0069]
[0070] (3) Planner Agent In addition to the two stages introduced in the "action cycle", it is also responsible for checking the results after the user interaction ends and adjusting or continuing the previous task planning as appropriate:
[0071] Adjustment (user interaction): According to user interaction request D R and current screen awareness information S, check whether the current user reply summary D Us has covered the information needed to be obtained from the user. Next, according to the reflection result and task instruction I T , plan P, key context information C, judge whether it is necessary to continue to interact with the user at present, output D R , and update the overall plan P Op and determine the next subtask P Sg accordingly. Its formal representation is formula (10).
[0072]
[0073] When the user interaction type D It is 0, the "interaction cycle" will be terminated at this time, and the "action cycle" is suspended and enters to make atomic operation decisions.
[0074] (C) Short-term memory module, when each agent and tool in the "application-level task execution module" executes a task, it needs to rely on a large number of antecedents and generate a series of consequents for use in subsequent processes. These working contexts need to be properly managed.
[0075] "Action cycle" short-term memory When the tth round of "action cycle" is completed, the Figure 6 demonstrates the memory scheduling logic between agents and tools .
[0076] "Interaction cycle" short-term memory When the kth round of "interaction cycle" is completed, theFigure 7 The agent is demonstrated The memory scheduling logic between the tool If the current round is 0, the response of the current interaction request is derived from After the "interaction cycle" will be terminated, k is reset to 0.
[0077] Suppose the "application-level task execution module" is executing the jth subtask, after the decision to give "Finish" in the tth round of "action cycle" The complete record is:
[0078]
[0079] The key trajectory record is:
[0080]
[0081] (Three) "Self-learning module" is the core of the application knowledge summary and evolution learning after task execution. The self-learning module includes two submodules: knowledge graph learning agent, which is used to summarize the page, component information and operation logic in the application according to the task execution history, and record it as a structured knowledge graph; Experience rule learning agent, which is used to summarize the experience knowledge of task planning, operation decision and error recovery according to the task execution history, and record it as natural language text. The specific content is as follows:
[0082] (1) Experience rule learning agent In high-level reflection of the entire subtask execution process, check the error steps and redundant steps generated in the execution, as well as the difference between the original plan and the final plan, and then summarize the experience T of avoiding problems and accurate planning in the future. After the "application-level task execution module" is executed, The key information in the subtask execution trajectory will be analyzed to generate three types of skills of planning T p , execution T p and error correction T exe for the current application AM err . Its formal representation is formula (11):
[0083]
[0084] In addition to the prompt generated by , the application skill T also includes the general skill T CommonIn use, the agent can generate retrieval-augmented generation (RAG) based on the keyword q Retrieval-augmented generation (RAG) based on the keyword q p from long-term memory. It can be formally represented as equation (12):
[0085]
[0086] (2) Knowledge graph learning agent tracks the changes of the page and the actions and triggers (page components) that cause the changes, and then establishes a knowledge graph reflecting the behavior of the components in the App page. After the execution of the App-Level Executor ends, each action execution record is analyzed to generate a description record for the page component . It can be formally represented as equation (13):
[0087]
[0088] After capturing the screen, the corresponding page component is matched from M, and if a relevant node is matched, the knowledge is attached to the textual perception information S output. Pt
[0089] Figure 8 The working process of the knowledge graph learning agent is shown. The learning process can refer to the green area in the figure. After the execution of the App-Level Executor ends, the screen perception information is extracted from each action execution record , including the screenshot and the AT. At this time the tree similarity algorithm is used to try to match the page from the page knowledge base of this App, and if a similar page cannot be found, it is marked as a new page; otherwise, the differences between the page and the original page are extracted, and the component nodes that are newly added or changed are recorded (as indicated by ② in the green area). For the new page or the newly added component node, its fragment and screenshot are input into the LMM model at the same time, so as to establish the description information of the relevant node (as indicated by ③ in the green area). Next, based on the action execution record The action information in the page is associated with the change of the page and the component triggering the change (as shown in ④ of the green area). Finally, the above information will be stored as a JSON (JavaScript Object Notation) format file, which records in detail each page visited in the App, the different component structures in the page and their function summaries, and the consequences of the actions of the triggers in the page (i.e. the description of the changes of the page or components that will occur after triggering). This will provide the action decision maker with the basis for decision-making, avoid relying on common sense and blindly guessing, and thus improve its accuracy and stability.
[0090] The extraction process of the App can refer to the blue area in the figure. When the App is running, the system will record the action information of the user in the App, and the action information of the user in the App will be associated with the change of the page and the component triggering the change (as shown in ④ of the green area). Finally, the above information will be stored as a JSON (JavaScript Object Notation) format file, which records in detail each page visited in the App, the different component structures in the page and their function summaries, and the consequences of the actions of the triggers in the page (i.e. the description of the changes of the page or components that will occur after triggering). This will provide the action decision maker with the basis for decision-making, avoid relying on common sense and blindly guessing, and thus improve its accuracy and stability. After obtaining the AT of the current screen, the same tree similarity algorithm will be used to try to match the page from the page knowledge base of the App (as shown in ② of the blue area). If such a page can be found, the description information of the corresponding component and the action consequences of the trigger will be directly extracted (as shown in ③ of the blue area), and will be attached to the screen perception information (for the visual mode, the information will be associated with the mark and output as additional textual perception information S Pt ; for the non-visual mode, it will be directly attached to the description of the corresponding component in the output S Pt .
[0091] The advantages and characteristics of the present application mainly include:
[0092] 1. The present application recognizes that user requirements are often complex and difficult to express completely at one time. Therefore, the present application designs an "interaction cycle intelligent agent (group)". At the key nodes of executing tasks, such as encountering ambiguous instructions, needing user confirmation of sensitive operations, or selecting from multiple options, the system can actively dialogue with the user. This dynamic correction mechanism based on interaction ensures the accuracy of task execution, effectively avoids failures caused by misunderstanding or operation errors, and greatly improves the user experience.
[0093] 2. The present application can understand complex natural language instructions of the user and intelligently decompose them into a series of sub-tasks that need to be executed in different applications. Through the top-level planning and coordination of the "global task management module" and the precise operation of the "application-level task execution module", the system can seamlessly switch between multiple applications and pass context information, thereby completing complex tasks across applications and long chains that traditional intelligent assistants cannot handle, and significantly improving the automation level of mobile devices.
[0094] 3. The application designs a unique "self-learning module" that can automatically summarize the structured "knowledge graph" and "experience rules" of the application after the task is completed. This continuous learning and evolution capability enables the system to dynamically adapt to the frequent interface and function updates of mobile applications, ensuring its long-term operation robustness and accuracy, and the intelligent level is continuously improved with use.
[0095] 4. The application adopts a clear three-module (global task management, application-level task execution, self-learning) architecture, and decomposes the complex execution process into the collaborative work of multiple agents and tools such as planners, decision makers, executors, and sensors. This high cohesion and low coupling design not only ensures the stability and robustness of the system operation, but also has good scalability, making it easy to upgrade and maintain future functions. BRIEF DESCRIPTION OF DRAWINGS
[0096] Figure 1 The application is an interactive GUI intelligent agent system framework for mobile device cross-application control.
[0097] Figure 2 The application is a global task management module structure and workflow diagram.
[0098] Figure 3 The application is a "action cycle" part structure and workflow diagram of the application-level task execution module.
[0099] Figure 4 The application is a screen perception tool module structure and workflow diagram.
[0100] Figure 5 The application is a "interaction cycle" part structure and workflow diagram of the application-level task execution module.
[0101] Figure 6 The application is a memory data access pipeline between each agent and tool in the "action cycle" part of the application-level task execution module.
[0102] Figure 7 The application is a memory data access pipeline between each agent and tool in the "interaction cycle" part of the application-level task execution module.
[0103] Figure 8 The application is a knowledge graph learning agent structure and workflow diagram in the self-learning module. DETAILED DESCRIPTION
[0104] The application will be further described below with specific examples combined with the drawings. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the application.
[0105] Please refer to the attached Figure 1 to the attached Figure 8 , the present application can be implemented according to the following embodiments, based on the Python programming language, large language models such as DeepseekV3, and bridge libraries to build an interactive GUI agent system for mobile devices.
[0106] As Figure 1 shown, the system architecture consists of three core components: "Global Task Management Module", "Application Level Task Execution Module" and "Self Learning Module". When the user issues a task instruction, the "Global Task Management Module" will decompose the task into specific subtasks that can be executed by each application, and hand it over to the "Application Level Task Execution Module" for execution. The "Application Level Task Execution Module" determines two key cycles - "Action Cycle" and "Interaction Cycle" through the "Planner Agent", continuously executes the task and interacts with the user. After the task is completed, the "Self Learning Module" will summarize the knowledge in the task and store it as long-term memory for future use.
[0107] As Figure 2 shown, the system architecture and core process, the task planning module at the top level of the system - "Global Task Management Module" includes "Global Task Planning Agent", which is responsible for task decomposition and reflection at the global level. It also includes "Application Information Manager" and "Task Manager" tools, respectively used to manage the metadata of installed applications in the device and distribute tasks to the "Application Level Task Execution Module" for execution and manage the life cycle of the task. When the user issues a task instruction, the "Global Task Planning Agent" receives the metadata of the installed applications in the "Application Information Manager" and the user's instruction, decomposes the task into specific subtasks that can be executed by each application, and hands it over to the "Task Manager", which starts the corresponding application, and finally hands over the control to the application level task execution module. When the subtask is completed, the application level task execution module returns the execution result to the "Global Task Planning Agent", which reflects on the execution result, confirms the execution status of the subtask and adjusts the global plan as appropriate to prepare for the execution of the next subtask. Each of the above modules can be implemented by using a program written in Python language, and using DeepseekV3 model as the backbone model for reasoning.
[0108] As Figure 3 shown, the system architecture and core process, the bottom-level task execution module of the system - "Application Level Task Execution Module" includes "Planner Agent" and "Action Decision Maker Agent" in the "Action Cycle", which is responsible for task planning and reflection at the middle level, and decision-making and execution of specific actions at the bottom level. It also includes "Action Executor" and "Screen Perceiver" (see Figure 4) tool is responsible for executing actions, and acquiring current screen information. In addition, the "key context extractor agent" continuously collects key context information on the screen in the "action loop" to ensure that important information during task execution is effectively delivered. When the "task manager" hands over control to the application-level task execution module, the "planner agent" receives the task, further decomposes it into sub-goals that need to be completed one by one, and hands it over to the "action decider agent" to make atomic action decisions based on screen perception information. Next, the "action executor" will execute this atomic action and notify the "screen perceiver" for new screen perception after the execution is completed to track the progress of the task. Finally, the "planner agent" reflects on the execution results, confirms the execution status of the sub-goals and adjusts the plan as appropriate to prepare for the execution of the next sub-goal. Before the task is first executed, the "screen perceiver" tool will be activated once to provide the initial screen perception information. The above process is the performance of the system in "mixed mode". In "independent mode", the reflection responsibility of the "planner agent" is split, a special "reflector agent" reflects on the execution results and confirms the execution status of the sub-goals, and the "planner agent" only needs to adjust the plan according to the reflection results to prepare for the execution of the next sub-goal. Each of the above modules is implemented by using a program written in Python language, and the DeepseekV3 model is used as the backbone model for reasoning. The bridge library for communication with the mobile phone can use UI Automator (an automatic testing tool for Android), Android Debug Bridge (ADB) or other bridge libraries.
[0109] As Figure 5The system architecture and core process are shown. The "interaction cycle" of the underlying task execution module of the system, "application-level task execution module", includes "user interaction agent" and "planner agent", which are responsible for interacting with users according to task needs, collecting feedback and adjusting the execution strategy of the task. It also includes the "user chat room" tool, which provides a friendly interaction interface to ensure smooth communication with the user. When the "planner agent" in the "action cycle" decides that user interaction is needed at the moment, the "user interaction agent" receives the interaction requirement, builds the user prompt and displays it to the user through the "user chat room". After the user responds, the "user chat room" will hand over the prompt and the user's reply to the "user interaction agent" for the next round of interaction planning. The "user interaction agent" will judge whether it still needs to interact with the user, if the user's response still does not meet the interaction needs, continue the above cycle, otherwise the "user interaction agent" will summarize the user's response and use it to update the task description, and finally hand it over to the "planner agent" to reflect on the above interaction, and make a new round of planning according to the user's supplemented needs. If the "planner agent" decides at this time that the user still needs to interact, repeat the above process, otherwise the "action cycle" is activated, and the "action decision agent" will start to decide atomic actions. Each of the above modules can be implemented by using a program written in Python language, and the DeepseekV3 model is used as the backbone model for reasoning. The "user chat room" tool can be realized through a Web interface, dialog box, voice or other user interaction methods.
[0110] As shown in Figure 6 , Figure 7 The system memory data storage and reading pipeline is shown. The short-term memory module of the system undertakes the context management function of all agents. As shown in Figure 6For the "action loop", first, the "planner agent" will get the start screen, plan, action and end screen after action execution of the Tth round from the memory manager, then make the reflection result of the Tth round action and give the plan of the T+1th round. After giving the reflection result, the memory of the Tth round is complete, at this time the memory manager will build a new round of "action loop" memory and take the end screen of the Tth round as the start screen of the T+1th round. Next, the "action decision agent" will get the start screen and plan of the T+1th round and the execution trajectory of [T, T-n]th round (only including actions and action results), then give the decision atomic action. After the action is executed by the "action executor", the "screen perceiver" will be activated and give the screen perception information of the T+1th round. Finally, the "planner agent" will repeat the previous process, so that the cycle enters the T+2th round. In addition to the above agents, when the "planner agent" judges that the Tth round task is normally executed, the "key context extractor agent" will be parallel with other agents, collect the key information of the Tth round end screen (also the T+1th round start screen) in the T+1th round, which will be stored as the key context of the Tth round. When executing the T+1th round task, the T-1th round key context is used, because the Tth round context information is still in collection, and since the initial screen of the T+1th round task already includes the Tth round context information, there is no need to repeat the supplement. In the Tth round, if the "planner agent" decides that user interaction is needed at present, the "interaction loop" will be activated and interact with the user for several rounds. For example Figure 7, first, the "user interaction agent" will get the starting screen and plan of the Tth round from the memory manager, and the prompt words and user responses of the [K, 1]th round of user interaction, to determine whether the current still needs to interact with the user. If the user's response still does not meet the interaction needs, the K+1th round of user prompt words will be constructed and displayed to the user through the "user chat room", and after the user responds, the prompt words and user responses will be stored as the K+1th round of user interaction records, and the next round of interaction planning will be continued by the "user interaction agent". Once the user's response has met the requirements, the "user interaction agent" will summarize the user's response and update the task description using it. At this time, the "planner agent" will get the starting screen, plan and updated task description of the Tth round, reflect on the Tth round of interaction, and make a plan for the T+1th round according to the user's supplemented requirements. After giving the reflection result, the memory manager will empty the incomplete part of the current "action cycle" memory at this time, and then a new "action cycle" memory will be created, and the starting screen of the Tth round will be directly used as the starting screen of the T+1th round. If the "planner agent" decides that the user still needs to interact at this time, the above process will be repeated, otherwise the "action cycle" will be activated, and the "action decision agent" will start to decide the atomic action. The above memory scheduling pipeline can be realized by using a program written in Python language.
[0111] The workflow of the system according to the present application will be described in detail below in combination with a specific user instruction execution scene:
[0112] Step (1): task decomposition
[0113] User Input a natural language instruction (for example: "Help me find a nearby Italian restaurant on Meituan, and then share the one with the highest rating to Zhang San on WeChat"). The global task planning agent in the global task management module receives the instruction and uses the installed application information provided by the application information manager to decompose the complex task into two sub-tasks that can be executed on a single application:
[0114] Sub-task 1: In the "Meituan" application, search for nearby Italian restaurants and find the one with the highest rating, and copy its information.
[0115] Sub-task 2: Open the "WeChat" application, find the contact "Zhang San", and send the copied restaurant information to him.
[0116] Step (2): start the first sub-task
[0117] Task manager Select the first sub-task, "Search for restaurants in Meituan", launch the "Meituan" app, and hand over control to the application-level task execution module.
[0118] Step (3): Execute the action loop
[0119] The action loop agent (group) of the application-level task execution module starts executing sub-task 1.
[0120] A. Planner agent Develop an execution plan, such as "Step 1: Click the search box", "Step 2: Enter 'Italian restaurant'", "Step 3: Click the search button", etc.
[0121] B. Action decision maker agent According to the first sub-goal "click the search box", decide on the "Tap" operation and target coordinates.
[0122] C. Action executor tool Execute the "Tap" operation on the phone.
[0123] D. Screen recognizer tool Get the new screenshot and AT after clicking.
[0124] E. The planner agent reflects on the new screen state and confirms that the search box has been successfully activated, then selects the next sub-goal "Enter 'Italian restaurant'" to continue the loop.
[0125] During this period, the key context extractor agent continuously observes the screen and extracts core information such as restaurant lists and ratings when search results appear.
[0126] Step (4): Execute the interaction loop (if necessary)
[0127] Suppose there are multiple restaurants with the same rating in the search results, the planner agent Decides that the user needs to make a choice, at which point it enters the interaction loop.
[0128] A. User interaction agent Generate a prompt: "There are multiple restaurants with the highest ratings, which one would you like to choose?".
[0129] B. User chat room tool Show the user the question and options, and collect the user's response.
[0130] C. User interaction agent Parse the user's response and summarize the user's final choice.
[0131] D. Planner agent The subsequent plan is updated according to the user's selection, and then the interaction cycle is exited, and the action cycle is returned to continue execution (for example, clicking on the restaurant selected by the user).
[0132] Step (5): Subtask completion and global reflection
[0133] When subtask 1 (finding and copying restaurant information in Meituan) is completed, the application-level task execution module returns the execution result (such as the copied restaurant information text) to the global task planning intelligent agent According to the execution result, it is confirmed that subtask 1 is successful, and the next subtask is prepared to be executed.
[0134] Step (6): Execute the next subtask
[0135] Task manager Subtask 2 is selected, the "WeChat" application is started, and the context information (restaurant information) obtained from subtask 1 is passed to the application-level task execution module. The module repeats the processes of steps (3) and (4) to complete the task of sending information in WeChat until all subtasks are completely executed.
[0136] Step (7): Self-learning and knowledge updating
[0137] After all subtasks are completed, the self-learning module is activated.
[0138] Application knowledge graph learning intelligent agent The operation records in "Meituan" and "WeChat" will be analyzed, and the knowledge graph about the two application page components and their functions will be updated or created.
[0139] Application experience rule learning intelligent agent The planning and execution experience of this task will be summarized, such as "when searching in Meituan, you can directly input in the search box on the homepage" or "when sharing information in WeChat to contacts, you should first click the search icon to find the contacts", and these experiences will be stored in long-term memory for future similar tasks.
[0140] Through comparative experiments on the open source evaluation benchmark RealMobile-Eval, it is shown that the user demand completion rate of the present application is increased by 33.7% compared with the existing method, and the redundant step rate is decreased by 58.5%.
[0141] The above is only a preferred embodiment of the present application, and therefore cannot limit the scope of the present application, that is, equivalent changes and modifications made according to the claims and content of the present application should still be within the scope covered by the present application.
Claims
1. An interactive GUI intelligent agent system for mobile devices, characterized in that, This system leverages a large language model and multiple intelligent agents to collaboratively understand user natural language tasks, decompose and execute cross-application tasks, and continuously accumulate and optimize knowledge. It is used to execute complex cross-application tasks on mobile devices and possesses self-learning and user interaction capabilities. The system includes: a global task management module, an application-level task execution module, and a self-learning module. (i) A global task management module, configured to receive natural language tasks from user input and decompose the natural language tasks into at least one sub-task instruction; This global task management module breaks down user commands into single application tasks and manages the lifecycle of the "application-level task execution module"; it includes three sub-modules: A global task planning agent is used to analyze the natural language task and application information, generate or adjust the task plan, and output the sub-task instructions; Application Information Manager is used to obtain information about applications installed on the user's device; The task manager is used to maintain the lifecycle of subtasks planned by the global task planning agent and assists in context passing when switching subtasks. (II) An application-level task execution module, communicatively connected to the global task management module, configured to receive the sub-task instructions and, based on the sub-task instructions, make decisions and execute corresponding operations on the application interface of the mobile device; it includes three sub-modules: an action loop intelligent agent group, a user interaction loop intelligent agent group, and a short-term memory module; wherein: (A) An action loop intelligent agent group, used to plan, make decisions, and execute GUI operations according to the sub-task instructions; it includes five sub-modules: The planner agent is used to formulate an execution plan containing multiple sub-goals based on sub-tasks, and to reflect on or replan based on the results of the operation execution; An action decision-maker agent, used to decide the sequence of atomic operations based on sub-goals; An action executor tool is used to send the atomic operations to a mobile device for execution; Screen sensor tools are used to obtain screenshots and UI hierarchy information of mobile devices; A key context extractor agent is used to extract key context information from the screen for reference by the planner and executor; This action loop agent group is responsible for executing tasks in the "application-level task execution module"; these agents and tools continuously generate atomic operations, execute them, and observe the execution status in the "action loop" until the task is completely completed; (B) User Interaction Loop Agent Group, used to generate user interaction interfaces and collect and parse user input when needed; this user interaction loop agent group is responsible for interacting with users in the "Application-level Task Execution Module"; it specifically includes three sub-modules: A user interaction intelligent agent used to generate user prompts and parse user responses; The planner agent is used to determine whether user interaction is needed and to adjust the task plan based on user feedback. User chat room tools are used to display interactive content and collect user input. These agents and tools continuously interact with the user and observe the execution in the "interaction loop" until the user interaction task is completely completed; (C) Short-term memory module, used to store context information during the current task execution process, for use by the action loop agent group and the interaction loop agent group; (III) A self-learning module, communicatively connected to the application-level task execution module, configured to acquire the task execution history of the application-level task execution module, and learn application knowledge based on the history to generate a knowledge graph or experience rules; it includes two sub-modules: A knowledge graph learning agent is used to summarize page, component information and operation logic in the application based on the task execution history, and record them as a structured knowledge graph. An experience-rule learning agent is used to summarize experience and knowledge regarding task planning, operational decisions, and error recovery based on the task execution history, and record it as natural language text.
2. The interactive GUI intelligent agent system according to claim 1, characterized in that, In the global task management module: (1) Global task planning agent Responsible for breaking down complex user commands into multiple single-application tasks, divided into two phases: direct planning and tuning. (a) Direct planning: Based on user instruction I and the metadata of m applications already installed on the user's phone Formulate the overall plan G and determine the execution of the first subtask, which is formally represented by formula (1): G includes the overall plan G op The subtask G to be executed next st and the context information G required to perform this task c Subtask G st Including unprocessed task instructions I T′ Context extraction request G cr and the corresponding task's App package name AM p ; (b) Adjustment: The results of previous tasks will be checked; the task progress will be updated based on the results, or the previous task plan will be adjusted. Based on user command I and the trajectory record of the "application-level task execution module" after multiple rounds of action loops and user interaction loops. Determine if the task results meet expectations; then, based on the inspection results and the previous global plan G, update the overall plan G. op And determine the next subtask G st At the same time, summarize the context information G that needs to be passed to the next subtask. c Its formal representation is formula (2): (2) Application Information Manager It is a tool for discovering, maintaining, and summarizing user equipment. Metadata of installed applications, for Provide a reliable base table of App capabilities, which will be dynamically updated as users' mobile applications change; its formal representation is formula (3): (3) Task Manager It is a tool used to maintain... The planned lifecycle of subtasks, assisting in context passing during subtask switching; After receiving the current G, it will call Launch the application; at the same time, LLM based on context information G c Instruction I for unprocessed tasks T′ Rewrite; after the task is issued, This will activate and begin processing the task; its formal representation is Equation (4):
3. The interactive GUI intelligent agent system according to claim 2, characterized in that, In the action loop intelligent agent group of the application-level task execution module: (1) Planner agent Responsible for according to The established tasks involve developing specific execution plans consisting of a series of sub-objectives, and... similar, It is also divided into two phases: direct planning and adjustment. (a) Direct planning: According to task instruction I T The first screen-sensing information S 0 Tips and tricks Develop a plan P and determine the first sub-objective to be executed; its formal expression is formula (5): P includes the overall plan P Op The subtask P that will be executed next Sg ; (b) Adjustment: The results of the actions will be checked, and the task progress will be updated or the previous task plan will be adjusted based on the results; firstly, According to task instruction I T Previous round screen perception information S and current round execution record Check if the current screen changes match the expected action and track the current subtask P. Sg The progress is recorded, and a reflection result R on the execution of this round of actions is output; next, based on the reflection result R and the previous plan P and key context information C, Update overall plan P Op And determine the next subtask P Sg ;at last, Determine whether user interaction is currently required, and output the user interaction request D. R Its formal representation is formula (6): D R Including user interaction type D Ic and thinking D It R includes the action result R. Ar and the progress of the plan R Pr Among them, R Ar It is an enumeration of the following values: A: Sub-goal has been completed; B: Sub-goal has been partially completed; C: The action produced an unexpected result; D: The action did not produce any change; When the action result does not meet expectations, i.e., C or D, it also includes the potential error cause R. Ec For error correction purposes; In R Ar When it is A, From P Op The next sub-goal is selected for execution; otherwise, the current sub-goal is not adjusted; in the R of the current sub-goal Ar When the result is C or D three times in a row, Adjustment to P will be considered. Op And then reselect sub-targets.
4. The interactive GUI intelligent agent system according to claim 3, characterized in that, In the action loop agent group of the application-level task execution module: (2) Action Decision Maker Intelligent Agent Responsible for according to The defined sub-goals plan the specific atomic operations; First, based on the result R of the previous action... Ar Get action decision prompts Or error recovery prompt Then based on task instruction I T Plan P, current screen awareness information S, and the most recent n execution records. Based on the key context information C, plan the sequence of atomic actions to be executed and make action decisions A; Its formal representation is formula (7): A includes atomic action sequence A As And the expected result of the action A Er A As It consists of one or more atomic operations.
5. The interactive GUI intelligent agent system according to claim 4, characterized in that, In the action loop agent group of the application-level task execution module: (3) Action Executor Tools Used to execute The generated atomic action sequence Through the bridging library, on user equipment Perform the following atomic operations: a. "Tap": The operation parameters are x, y. Tap the phone screen at coordinates (x, y). b. "Swipe": The operation parameters are x1, y1, x2, y2, dur. It slides from the phone screen coordinates (x1, y1) to the phone screen coordinates (x2, y2) within dur seconds. c. "LongPress": The operation parameters are x, y, dur. Press and hold the phone screen at coordinates (x, y) for dur seconds and then release. d.Input": The operation parameter is text, and text is entered in the active input box; e.ClearInput: No operation parameters are required; clears the active input field. f."KeyEvent": The operation parameter is event, which sends an Android key event to the phone; g.Wait”: The operation parameter is dur, and it waits for dur seconds; h."Finish": No operation parameters are required. This is used to mark the task as completed and no further action is needed. i. "NeedInteraction": No operation parameters are required. This is used to indicate that user interaction is needed at present, rather than other actions. j."ListApps": No operation parameters are required; lists all third-party applications installed on the device. k."StartApp": The operation parameter is package, which launches a specific application on the phone with the package name package.
6. The interactive GUI intelligent agent system according to claim 5, characterized in that, In the action loop agent group of the application-level task execution module: (4) Screen Sensor Tool Used to acquire and perceive current screen information; Get screenshot S using the bridging library Sp and Accessible Tree XMLS At ; Supports both visual and non-visual modes: a. In visual mode From Extract the "clickable" and "scrollable" components, and project them onto the screen based on their bounds property. The data is then assigned a sequence number, and the mapping between this sequence number and the center coordinates is stored as follows: In this mode, In hand over Before execution, based on Perform the transformation from markers to coordinates; b. In non-visual mode The algorithm removes useless nodes and attributes based on the "AT processing algorithm," completes the descriptive information of image nodes using the MLLM model, merges and summarizes the child nodes of the "Clickable" component, and finally outputs the text-based perceptual information in a Markdown-like format.
7. The interactive GUI intelligent agent system according to claim 6, characterized in that, In the action loop intelligent agent group of the application-level task execution module: (5) Key Context Extractor Agent It is responsible for extracting key contextual information from the screen for reference by the planner and executor; in information gathering tasks, The maintained context information will play an important role. Based on task instruction I T Context extraction request G cr The plan (P), the current screen awareness information (S), and the previous context information (C) t-1 Extract new key contextual information C t : At the end of each "action cycle" and the result R Ar It will only activate when it is A or B.
8. The interactive GUI intelligent agent system according to claim 7, characterized in that, In the user interaction loop intelligent agent group of the application-level task execution module: The "action loop" in Determine the type of user interaction D It When the value is not 0, the current "action loop" will be suspended and the "interaction loop" will be entered to interact with the user; where D It It is an enumeration of the following values: 1: User confirmation is required for sensitive or dangerous operations; 2: User confirmation is required for irreversible operations; 3: User selection is required for different options; 4: User clarification is required for the instruction; 0: No interaction is required; to avoid... Hallucinations and missing user interaction, allowing Choose the "NeedInteraction" action to correct it; once the selection occurs, In the next loop, reflect on the error and enter the "reciprocal loop"; specifically as follows: (1) User interaction intelligent agent Responsible for continuously generating user prompts based on the sub-goals set by the planner, receiving and judging whether the user's response has met the requirements, and summarizing the user's dialogue if the requirements are met. When D It When the value is not 0, or when a user reply D is received. Ur hour, It will be activated; Based on task instruction I T Plan P, User Interaction Request D R Based on the current screen perception information S and the previous context information C, the current user interaction plan D is generated; if k rounds of interaction have already been performed, Read historical interaction records User prompts in D Up And reply D Ur Its formal representation is Equation (8): D includes the interaction state D Is And user prompt D Up ; where D Is It is an enumeration of the following values: 0: further interaction with the user is needed; 1: the user has made a clear choice or clarified a vague statement; when the interaction is complete, D is... Is The value is 1, and it also includes user response summaries (D). Us For updating I T use; (2) User chat room tools Used to provide users with information through UI interfaces and consoles, etc. Display user prompt D Up And wait for the user to input a reply D Ur When the interaction state D Is When it is 0, Activated; during evaluation, Access the evaluation system for automated testing; its formal representation is Equation (9): (3) Planner agent In addition to the two phases described in the "Action Loop", They are also responsible for checking the results after user interaction ends and adjusting or continuing the previous task plan as needed: Adjusting user interaction: Based on user interaction request D R And the current screen perception information S, check the current user's reply summary D Us Have all the information that needs to be obtained from the user been covered? Next, based on the reflection results and task instructions I... T Plan (P), Key Contextual Information (C), Determine whether further interaction with the user is necessary, and output D. R And update the overall plan P as needed. Op With determining the next subtask P Sg Its formal representation is formula (10): When user interaction type D It When the value is 0, the "interactive loop" is terminated, the "action loop" is unsuspended and enters a new loop. Make atomic operation decisions.
9. The interactive GUI intelligent agent system according to claim 8, characterized in that, In the short-term memory module of the application-level task execution module, each agent and tool in the "application-level task execution module" relies on a large amount of context when executing tasks, and at the same time generates a series of contexts for use in subsequent processes; the short-term memory module properly manages these working contexts, specifically: "Action Loop" Short-Term Memory After completing the t-th round of the "action cycle", "Interactive Loop" Short-Term Memory After completing the k-th round of the "interactive loop", If this round If the value is 0, then this interaction request is valid. response Source After the "interactive loop" is terminated, k is reset to 0; Suppose that the "application-level task execution module" is executing the j-th subtask, and in the t-th round of the "action loop"... After giving the "Finish" decision, the complete record is as follows: The key trajectory is recorded as follows:
10. The interactive GUI intelligent agent system according to claim 9, characterized in that, In the self-learning module: (1) Experience rule learning agent The process involves high-level reflection on the entire sub-task execution process, examining erroneous and redundant steps, and discrepancies between the original and final plans. This leads to summarizing lessons learned for avoiding future problems and precise planning. This process is repeated after the "application-level task execution module" has finished executing. The execution trajectory of the sub-task Analyze key information in the data to generate an application-specific AM (Advanced Management Platform) for the current application. p Planning T p Execute T exe With error correction T err Three types of techniques; their formal representation is formula (11): Except by In addition to generating prompts, the application skills T also include general skills T provided by human experts. Common In use, the agent generates data based on the keyword q using search enhancement techniques. Retrieval from long-term memory using AM p Related information; Its formal representation is formula (12): (2) Knowledge Graph Learning Agent Track page changes and the actions and triggers that cause those changes, i.e., page components, and then build a knowledge graph reflecting the behavior of components within the app's pages; after the App-Level Executor finishes execution, Record the execution of each action. Perform analysis and generate results for page components. Description record Its formal representation is formula (13): exist After capturing the screen, it will attempt to match the corresponding page component from M. If a relevant node is matched, the knowledge will be appended to it. Output text-aware information S Pt middle.
