Task execution methods, devices, electronic equipment and storage media

Abstract

This disclosure relates to a task execution method, apparatus, electronic device, and storage medium, including: acquiring a target task, wherein the target task is a task to be executed by: determining a first interactive target based on a screenshot of the interface of a first application; determining a second interactive target after generating a screenshot of the interface of a second application through interaction with the first interactive target; and interacting with the second interactive target until the task execution target is achieved; constructing a first global state based on first context information corresponding to a first time step of the target task; the first context information refers to the context state managed by a context management strategy output by the task execution agent; inputting the first interface screenshot and the first global state into the task execution agent to obtain first task execution strategy information including at least one of an interface execution strategy and a context management strategy; and executing the target task in a second time step based on the first task execution strategy information until an execution result is obtained. This disclosure can execute long-term interface interaction tasks with high quality.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a task execution method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, task execution agents can perform short-term interface interaction tasks. For example, simple interface, swiping, and input operations can be automatically performed by the task execution agent. However, once the interface interaction task becomes longer, such as involving cross-application or multiple interaction steps, the task execution agent is prone to various problems, causing the task to fail. Therefore, there is currently no solution for using task execution agents to perform long-term interface interaction tasks with high quality. Summary of the Invention

[0003] This disclosure provides a task execution method, apparatus, electronic device, and storage medium to solve the problems in related technologies. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a task execution method is provided, the method comprising: The system acquires first context information corresponding to a target task and a first time step. The target task is a user interface interaction task, which is determined based on the task execution target. A first interaction target is identified in a screenshot of a first application's interface, and a second interaction target is identified after an interface screenshot of a second application is generated through interaction with the first interaction target. Interaction with the second interaction target continues until the task execution target is achieved. Both the first and second applications are applications with visual interfaces containing interaction targets. The first context information indicates the context state managed by the context management strategy output by the task execution agent during the execution of the target task up to the first time step. Based on the target task and the first context information, construct the first global state corresponding to the first time step; Obtain the screenshot of the first interface corresponding to the first time step; The first interface screenshot and the first global state are input into the task execution agent to perform task execution reasoning, and the first task execution strategy information is obtained. The first task execution strategy information includes at least one of the interface execution strategy and the context management strategy. Based on the first task execution strategy information, the target task is executed at the second time step until the execution result of the target task is obtained. The second time step is the next time step after the first time step.

[0004] In one exemplary implementation, the first task execution strategy information includes: The reasoning text in this step is used to explain the judgment result of the task execution agent on the execution status of the target task, as well as the reasoning process of the execution action; The history folding command specifies how to fold and compress historical records; A tool invocation instruction is used to specify a tool and the action to be performed by the tool, wherein the action is at least one of an interface action and a context management action; Interface observation information is used to record the interface data generated in the current situation; Action intent information, used to indicate the purpose of the action and its subsequent effects; The history folding instruction, the interface observation information, and the action intent information belong to the context management strategy, while the tool call instruction belongs to the interface execution strategy or the context management strategy.

[0005] In one exemplary implementation, the first context information corresponding to the first time step includes: Folded history, used to indicate the compressed history formed from the start of the execution of the target task; The collapsed interface state is used to record the interface records stored since the start of the execution of the target task; Recent step record, used to indicate the task execution details corresponding to at least one recent historical time step.

[0006] In one exemplary implementation, executing the target task at a second time step based on the first task execution strategy information includes: If the tool invocation instruction in the first task execution strategy information indicates an interface action, the first tool corresponding to the interface action is used to execute the interface action to obtain the interface action execution result; based on the interface action execution result, a second interface screenshot corresponding to the second time step is obtained; Update the folding history record according to the historical folding instructions; Update the most recent step record based on the interface observation information, the action intent information, the tool call command, and the current step reasoning text; Based on the updated folding history, the updated recent step record, and the folded interface state that has not been updated, the second context information corresponding to the second time step is generated; Based on the target task and the second context information, a second global state corresponding to the second time step is generated; The screenshot of the second interface and the second global state are input into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information. Based on the second task execution strategy information, the target task is executed at the third time step until the execution result of the target task is obtained. The third time step is the next time step after the second time step.

[0007] In one exemplary implementation, executing the target task at a second time step based on the first task execution strategy information includes: If the tool invocation instruction in the first task execution strategy information indicates a context management action, the second tool corresponding to the context management action is used to execute the context management action to obtain the managed collapsed interface state; Update the folding history record according to the historical folding instructions; Update the most recent step record based on the interface observation information, the action intent information, the tool call command, and the current step reasoning text; Based on the updated folding history, the updated recent step record, and the managed folding interface state, generate the second context information corresponding to the second time step; Based on the target task and the second context information, a second global state corresponding to the second time step is generated; The first interface screenshot and the second global state are input into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information. Based on the second task execution strategy information, the target task is executed at the third time step until the execution result of the target task is obtained. The third time step is the next time step after the second time step.

[0008] In one exemplary implementation, the history folding instruction includes a first parameter and a second parameter, the first parameter indicating the range of compressed historical steps, and the second parameter indicating the generated summary; updating the folding history according to the history folding instruction includes: When the first parameter indicates single-step compression, a corresponding summary is generated based on the most recent historical step; when the first parameter indicates multi-step compression, a corresponding summary is generated based on the specified multi-step steps. The generated summary is stored in the second parameter, and the historical records corresponding to the compressed steps are replaced based on the second parameter.

[0009] In one exemplary embodiment, the folded interface state includes multiple memory items, each of which includes a memory item identifier, a corresponding memory item description, and the corresponding original interface content. The memory item is the smallest operational unit of the context management action.

[0010] In one exemplary implementation, the execution of the context management action using the second tool corresponding to the context management action includes performing at least one of the following operations on the plurality of memory items: deleting, modifying, adding, querying, merging, or saving as a persistent state.

[0011] In one exemplary embodiment, the task execution agent is trained using the following method: Obtain basic tasks, which are standard interface interaction tasks with fixed scenarios, fixed operation objects, and fixed processes; The basic task is subjected to differential amplification processing to obtain a diversified task pool, which includes multiple sample tasks. The differential amplification processing includes at least one of entity replacement, memory operation enhancement, and task simplification. The task execution agent is trained based on the sample tasks in the diversified task pool.

[0012] In one exemplary implementation, training the task execution agent based on each of the sample tasks in the diversified task pool includes: For each of the sample tasks, the sample task is executed based on the teacher model, and the task execution strategy information output at each time step is recorded; the task execution strategy information is parsed to obtain the step execution state information corresponding to each of the time steps; and the interaction trajectory information is obtained based on the step execution state information. Filter the interaction trajectory information that meets the preset quality requirements to obtain the target interaction trajectory information; Each step in the target interaction trajectory information is labeled to obtain a labeled step sequence; Based on the labeled sequence of steps, a student model is trained, and the trained student model is used as the task execution agent.

[0013] According to a second aspect of the present disclosure, a task execution apparatus is provided, the apparatus comprising: The basic information acquisition module is configured to acquire the target task and the first context information corresponding to the first time step. The target task is an interface interaction task, which is a task that, based on the task execution target, determines the first interaction target in the interface screenshot of the first application, and determines the second interaction target after generating the interface screenshot of the second application through interaction with the first interaction target, and continues to interact with the second interaction target until the task execution target is achieved. Both the first application and the second application are applications with a visual interface containing the interaction target. The first context information is used to indicate the context state managed by the context management strategy output by the task execution agent during the execution of the target task to the first time step. The global state construction module is configured to construct the first global state corresponding to the first time step based on the target task and the first context information. The screenshot acquisition module is configured to acquire the first interface screenshot corresponding to the first time step; The task execution module is configured to input the first interface screenshot and the first global state into the task execution agent to perform task execution reasoning and obtain first task execution strategy information, which includes at least one of interface execution strategy and context management strategy; based on the first task execution strategy information, the target task is executed at a second time step until the execution result of the target task is obtained, where the second time step is the next time step after the first time step.

[0014] In one exemplary implementation, the first task execution strategy information includes: The reasoning text in this step is used to explain the judgment result of the task execution agent on the execution status of the target task, as well as the reasoning process of the execution action; The history folding command specifies how to fold and compress historical records; A tool invocation instruction is used to specify a tool and the action to be performed by the tool, wherein the action is at least one of an interface action and a context management action; Interface observation information is used to record the interface data generated in the current situation; Action intent information, used to indicate the purpose of the action and its subsequent effects; The history folding instruction, the interface observation information, and the action intent information belong to the context management strategy, while the tool call instruction belongs to the interface execution strategy or the context management strategy.

[0015] In one exemplary implementation, the first context information corresponding to the first time step includes: Collapsed history, used to indicate the history formed after compression since the start of the execution of the target task; The collapsed interface state is used to record the interface records stored since the start of the execution of the target task; Recent step record, used to indicate the task execution details corresponding to at least one recent historical time step.

[0016] In one exemplary implementation, the task execution module is configured to execute: If the tool invocation instruction in the first task execution strategy information indicates an interface action, the first tool corresponding to the interface action is used to execute the interface action to obtain the interface action execution result; based on the interface action execution result, a second interface screenshot corresponding to the second time step is obtained; Update the folding history record according to the historical folding instructions; Update the most recent step record based on the interface observation information, the action intent information, the tool call command, and the current step reasoning text; Based on the updated folding history, the updated recent step record, and the folded interface state that has not been updated, the second context information corresponding to the second time step is generated; Based on the target task and the second context information, a second global state corresponding to the second time step is generated; The screenshot of the second interface and the second global state are input into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information. Based on the second task execution strategy information, the target task is executed at the third time step until the execution result of the target task is obtained. The third time step is the next time step after the second time step.

[0017] In one exemplary implementation, the task execution module is configured to execute: If the tool invocation instruction in the first task execution strategy information indicates a context management action, the second tool corresponding to the context management action is used to execute the context management action to obtain the managed collapsed interface state; Update the folding history record according to the historical folding instructions; Update the most recent step record based on the interface observation information, the action intent information, the tool call command, and the current step reasoning text; Based on the updated folding history, the updated recent step record, and the managed folding interface state, generate the second context information corresponding to the second time step; Based on the target task and the second context information, a second global state corresponding to the second time step is generated; The first interface screenshot and the second global state are input into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information. Based on the second task execution strategy information, the target task is executed at the third time step until the execution result of the target task is obtained. The third time step is the next time step after the second time step.

[0018] In one exemplary implementation, the history folding instruction includes a first parameter and a second parameter, the first parameter indicating the range of the compressed historical steps, and the second parameter indicating the generated summary; the task execution module is configured to execute: When the first parameter indicates single-step compression, a corresponding summary is generated based on the most recent historical step; when the first parameter indicates multi-step compression, a corresponding summary is generated based on the specified multi-step steps. The generated summary is stored in the second parameter, and the historical records corresponding to the compressed steps are replaced based on the second parameter.

[0019] In one exemplary embodiment, the folded interface state includes multiple memory items, each of which includes a memory item identifier, a corresponding memory item description, and the corresponding original interface content. The memory item is the smallest operational unit of the context management action.

[0020] In one exemplary implementation, the execution of the context management action using the second tool corresponding to the context management action includes performing at least one of the following operations on the plurality of memory items: deleting, modifying, adding, querying, merging, or saving as a persistent state.

[0021] In one exemplary implementation, the task execution module is configured to execute: Obtain basic tasks, which are standard interface interaction tasks with fixed scenarios, fixed operation objects, and fixed processes; The basic task is subjected to differential amplification processing to obtain a diversified task pool, which includes multiple sample tasks. The differential amplification processing includes at least one of entity replacement, memory operation enhancement, and task simplification. The task execution agent is trained based on the sample tasks in the diversified task pool.

[0022] In one exemplary implementation, the task execution module is configured to execute: For each of the sample tasks, the sample task is executed based on the teacher model, and the task execution strategy information output at each time step is recorded; the task execution strategy information is parsed to obtain the step execution state information corresponding to each of the time steps; and the interaction trajectory information is obtained based on the step execution state information. Filter the interaction trajectory information that meets the preset quality requirements to obtain the target interaction trajectory information; Each step in the target interaction trajectory information is labeled to obtain a labeled step sequence; Based on the labeled sequence of steps, a student model is trained, and the trained student model is used as the task execution agent.

[0023] According to a third aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the task execution method as described in any of the above embodiments.

[0024] According to a fourth aspect of the present disclosure, a computer storage medium is provided, which, when instructions in the computer storage medium are executed by a processor of an electronic device, causes the electronic device to perform the task execution method described in any of the above embodiments.

[0025] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the task execution method described in any of the above embodiments.

[0026] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: In the task execution method provided in this embodiment, by dynamically managing the context state through the context management strategy output by the task execution agent, the agent can not only refer to the current interface screenshot when making inference decisions at each step, but also combine the managed historical execution context, and automatically take both context management actions and interface actions as inference targets. This achieves the technical objective of making both context management and ordinary interface actions standard actions that can be output, executed, trained, and transferred by the task execution agent.

[0027] This design addresses the problem of accumulated contextual information leading to management failure or redundant information interfering with reasoning in long-term user interface (UI) tasks. By unifying action decision-making and context management within the reasoning scope of the task-executing agent, it achieves integrated and intelligent processing of both. It eliminates the need for separate context management rules or modules, allowing the agent to autonomously determine and execute appropriate context management operations and / or UI operations based on the current task execution situation. This adapts to UI interaction tasks of varying complexity and length, avoiding unnecessary waste of reasoning computational power on contextual information while retaining crucial historical information for current reasoning. This effectively improves the accuracy of action decisions for long-term UI interaction tasks, ensuring successful task execution and achieving the correct final result, thus guaranteeing high-quality and high-success-rate execution of long-term UI interaction tasks.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 This is a flowchart illustrating a task execution method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the execution of a target task at a second time step according to an exemplary embodiment. Figure 1 ; Figure 3 This is a schematic diagram illustrating the execution of a target task at a second time step according to an exemplary embodiment. Figure 2 ; Figure 4 This is a schematic diagram illustrating a task execution process framework according to an exemplary embodiment; Figure 5 This is a schematic diagram illustrating a sample generation process according to an exemplary embodiment; Figure 6 This is a schematic diagram illustrating the execution process of a long-running GUI task according to an exemplary embodiment; Figure 7 This is a schematic diagram illustrating a comparison between context length and task performance according to an exemplary embodiment; Figure 8 This is a block diagram of a task execution apparatus according to an exemplary embodiment; Figure 9 This is a structural block diagram of a computer device according to an exemplary embodiment. Figure 1 ; Figure 10 This is a structural block diagram of a computer device according to an exemplary embodiment. Figure 2 . Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0032] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] Current task execution agents are only capable of handling short-term tasks; once the task becomes longer or crosses applications, context management fails, and the task is easily interrupted. These task execution agents mainly fall into the following two categories: (a) ReAct-based end-to-end GUI (Graphical User Interface) task execution agent: Technical principle: The intelligent agent only infers a single interface action at each step, such as clicking or swiping; after executing the action, the entire step log is mechanically appended to the history, without any context management.

[0034] Technical defects: a. Passive linear expansion of context: The more steps there are, the longer the inference prompts become, and the higher the inference cost becomes; b. Mixed historical information: Task progress, key data, and invalid operations are piled together, and the agent cannot distinguish the information type; c. Key information is easily lost: Important information is buried in redundant logs or lost due to text truncation; d. The agent has no context management rights: it can only perform interface operations and cannot optimize the context; e. Cross-application tasks are easily interrupted: After switching applications, the original interface information disappears, and there is no separate storage mechanism.

[0035] (ii) Intelligent agent collaborative external independent memory module Technical principle: The agent is responsible for GUI action decisions, with additional independent external modules such as a memory bank, digester, retrieval unit, and planner to manage historical information, compress logs, and store data, thereby completely separating action decisions from context management.

[0036] Technical defects: a. Module fragmentation: External modules do not understand the agent's action intentions and cannot accurately implement context management; b. Complex architecture: Multiple modules are connected in series, resulting in high deployment, operation and maintenance, and troubleshooting costs, which is not conducive to end-to-end implementation; c. Rigid rules: Relying on fixed steps / fixed windows to compress history, it cannot be flexibly adjusted according to task progress and interface changes; d. Inconsistent training objectives: The objectives of the main agent and external modules are separated, making it impossible to conduct unified supervised training; e. Poor portability: Different components have different interfaces, resulting in high adaptation costs.

[0037] This disclosure differs from related technologies by proposing a context management-as-action (ConAct) technical solution. This allows the same task-executing agent to proactively, autonomously, and intelligently maintain the context while performing GUI operations. This gives context management actions and interface actions the same status, both belonging to the standard actions that the agent can output, execute, train, and transfer. This enables the integrated reasoning and execution of context management actions and interface actions.

[0038] Figure 1 This is a flowchart illustrating a task execution method according to an exemplary embodiment. The task execution method can be applied to an electronic device, which can be executed independently by a server or a terminal, or jointly by a terminal and a server. The terminal can be, but is not limited to, physical devices such as smartphones, tablets, laptops, desktop computers, smart speakers, smart wearable devices, digital assistants, augmented reality devices, and virtual reality devices, and can also include software such as applications running on the physical device. The server can be, but is not limited to, a standalone server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc. (Refer to...) Figure 1 As shown, the method includes the following steps.

[0039] In S110, the first context information corresponding to the target task and the first time step is obtained. The target task is an interface interaction task. The interface interaction task is a task that determines a first interaction target in the interface screenshot of a first application based on the task execution target, and determines a second interaction target after generating an interface screenshot of a second application through interaction with the first interaction target, and continues to interact with the second interaction target until the task execution target is reached. Both the first application and the second application are applications with a visual interface containing the interaction target. The first context information is used to indicate the context state managed by the context management strategy output by the task execution agent during the process of the target task being executed to the first time step.

[0040] In this disclosure, the task execution agent refers to an agent that automatically performs interface interaction tasks. For example, the agent can be an agent trained based on a multimodal large model (MLLM). This agent can automatically execute the target task according to the interface interaction requirements defined in the target task without human intervention. In this disclosure, G is used to refer to the target task.

[0041] For example, the target task might be "help me organize the sales data in the current WPS document and generate a bar chart to save to the desktop." This target task requires multiple interactive operations and spans multiple applications to complete, and belongs to the long-term interface interaction task that this disclosure is suitable for execution.

[0042] The interface interaction task disclosed herein specifically refers to a task that, based on the task execution objective, determines a first interaction objective in a screenshot of the interface of a first application, and after generating a screenshot of the interface of a second application through interaction with the first interaction objective, determines a second interaction objective, and continues to interact with the second interaction objective until the task execution objective is achieved; both the first application and the second application are applications with a visual interface that includes the interaction objective.

[0043] The method disclosed herein is primarily designed for long-term user interface interaction tasks. These tasks typically require multiple rounds of interaction (more than 3) to complete and cannot achieve the user-defined task execution goal through a single step. For example, the task execution goal could be various cross-application office requirements proposed by the user, or specific operational requirements that need to be implemented in an automated process. The core execution logic of the user interface interaction task revolves around interaction with the visual interface of the relevant application. During the task's progression, each interaction with the current interface's interaction goal may generate a new interface, necessitating the determination of the next new interaction goal. This process continues iteratively until the task execution goal is met or execution is confirmed as failed. The interaction goal refers to elements on the interface that can trigger interactive operations, such as buttons, input boxes, dropdown options, etc., and this disclosure does not limit this.

[0044] The first and second applications refer to any application with a visual interactive interface involved in the task execution process. These are relative concepts derived from the stages of task execution, not fixed categories. At different stages of the task, the same application can function as either the first or second application, or the first and second applications can be different. For example, if the task objective is to organize data in a WPS document, switch to Excel to generate a statistical chart, and then save it to the local desktop, then WPS at the beginning of the task is the first application. After completing the interaction within WPS, the opened Excel becomes the second application. If the generated statistical chart needs to be inserted into a PowerPoint presentation later, then Excel becomes the first application for the next stage, and the newly opened PowerPoint becomes the second application for that stage. This disclosure supports long-term interface interaction tasks across applications; therefore, the first and second applications can be different applications of the same type or different types of applications, adapting to cross-application operation scenarios.

[0045] A task-executing agent requires reasoning at multiple time steps, along with the execution of reasoning decisions, to complete a long-term target task. The reasoning result corresponding to each time step, once executed, can advance the execution progress of the target task. The first time step can be understood as any one of these multiple time steps.

[0046] For example, the first time step may need to identify the range of sales data in the current document, and the second time step may need to click the chart option in the insert function bar. Each step requires recording the previous execution status, which requires a context management strategy to organize and control the context information of the entire process.

[0047] The first context information is the context state that has been autonomously managed by the agent when the task progresses to the first time step. This context state may include the original task requirements, the records of completed operations, screenshots of the current software interface, and historical interaction information that is useful for the current task reasoning. It can record which operations have been completed, where the current interface is located, and which information is useful for subsequent decisions, avoiding redundant invalid information from interfering with subsequent reasoning decisions.

[0048] Context management strategy is the context processing rule for autonomous reasoning of the task execution agent. Unlike the mechanical management of context in related technologies, or the separate management from the task execution agent, the task execution agent of this disclosure may output a context management strategy at each time step. The context management strategy output before the first time step will affect the first context information corresponding to the current first time step.

[0049] In one exemplary implementation, the first context information corresponding to the first time step includes: (1) Folded history, used to indicate the history formed after compression since the start of the execution of the target task.

[0050] This disclosure uses This refers to the folded history in the first context information. The folded history refers to the simplified historical information obtained by compressing and organizing the historical execution steps of the target task from the start to the current first time step according to the historical folding instructions output by the task execution agent. It may not retain all the original step details, but only retain the core content that is valuable for subsequent reasoning and decision-making, thereby avoiding information redundancy and reducing the computational power consumption of reasoning.

[0051] (2) Folded interface state, used to record the interface records stored from the start of the execution of the target task.

[0052] This disclosure uses This refers to the collapsed interface state in the first context information. The collapsed interface state is a structured memory information obtained by the task-executing agent after managing various interface-related information generated during the interaction process according to context management actions. It stores key interface content that is valuable for subsequent task execution, rather than retaining all original interface screenshots, thus saving storage resources and avoiding redundant information from interfering with reasoning. For example, it can include interface screenshots, screenshot-related path information, text, numbers, and key image information in the screenshots.

[0053] In some implementations, it is also possible to It is broken down into memory of people, memory of goods, memory of places, and temporary memory for separate fine-grained management.

[0054] (3) Recent step record, used to indicate the task execution details corresponding to at least one recent historical time step.

[0055] This disclosure uses This refers to the most recent step record of at least one most recent historical time step in the first context information. The most recent step record specifically records the execution details of one or more recent time steps before the target task reached the first time step, facilitating subsequent reasoning by the agent.

[0056] In this disclosure, the folded history, folded interface state, and recent step record are all used as the first context information, allowing for structured hierarchical storage of context information: the folded history provides the core steps from the start of the task to the present, avoiding redundant information from occupying inference space; the folded interface state retains key interface information, facilitating the agent to call up key historical content as needed; and the recent step record fully retains the execution details of the most recent steps, ensuring that inference can accurately inherit the results of the preceding steps. This hierarchical structured context design not only meets the need for long-term tasks to retain core historical information but also avoids information redundancy and computational waste caused by full storage. Simultaneously, it ensures that the current step inference needs to obtain the preceding information, improving the accuracy of the agent's action decisions and the efficiency of inference, making the execution of long-term interface interaction tasks more stable and reliable.

[0057] In S120, a first global state corresponding to the first time step is constructed based on the target task and the first context information.

[0058] In this disclosure, the first global state corresponding to the first time step can be obtained through... Indicates, that is =(G, , , Where G represents the target task itself. It is the folded history in the first context information. It refers to the collapsed interface state in the first context information. It is the most recent step record in the first context information. After integrating the four types of information, the first global state of the first time step can be obtained. This can integrate the target task requirements and the sorted hierarchical context information into a unified state structure, providing a complete and clear decision basis for the reasoning and decision-making of the task execution agent, avoiding the lack of key reasoning basis, and avoiding interference from redundant and invalid information.

[0059] In S130, the screenshot of the first interface corresponding to the first time step is obtained.

[0060] In this disclosure, through This represents the first screenshot of the interface corresponding to the first time step. The first screenshot is the image information of the software interface corresponding to the current time step. It is the basic input for the task execution agent to perceive the current interface elements and determine the interactive positions. The task execution agent needs to combine the first screenshot to identify interactive elements such as buttons and input boxes on the interface, and then combine it with the global state to determine what operation should be performed.

[0061] In S140, the first interface screenshot and the first global state are input into the task execution agent to perform task execution reasoning, and the first task execution strategy information is obtained. The first task execution strategy information includes at least one of the interface execution strategy and the context management strategy.

[0062] The interface execution strategy is the operational strategy output by the task execution agent, used to directly interact with the current graphical user interface. It is the core execution action that propels the completion of the target task. The interface execution strategy can indicate the type of interactive operation to be performed, the corresponding interface location of the operation, and the execution intent of the operation. Common operation types include various interactive actions that can be performed on the software interface, such as clicking, dragging, inputting, and scrolling the wheel.

[0063] The context management strategy is also output by the task execution agent. It is used to manage the existing historical context information. It does not directly interact with the graphical interface to change the interface state, but it will update the context state needed for subsequent inference, and organize effective context information for the inference decision of the next time step.

[0064] Unlike the fixed and mechanical context management rules in traditional solutions, the context management strategy in this disclosure is obtained through autonomous reasoning by the task execution agent. The agent can autonomously output appropriate context management actions based on the actual situation of the current task progress, adapting to the context management needs of different tasks.

[0065] This disclosure allows a task execution agent to output only the interface execution strategy, only the context management strategy, or both strategies simultaneously at a given time step. This flexible output design adapts to different scenarios during task execution: when task progress requires direct manipulation of the interface without adjusting the context state, the agent can output only the interface execution strategy; when the current time step does not require changing the interface but only needs to organize the existing context information, the agent can output only the context management strategy; when manipulating the interface also requires updating the context state, the agent can output both strategies simultaneously. This design makes task execution and context management more flexible and autonomous, further enhancing the agent's adaptability to different scenarios.

[0066] In S150, based on the first task execution strategy information, the target task is executed at the second time step until the execution result of the target task is obtained. The second time step is the next time step after the first time step.

[0067] After obtaining the first task execution strategy information through task reasoning in the first time step, the corresponding actions can be executed according to the output strategy content, proceeding to the next time step, the second time step, to continue advancing the task. If the first task execution strategy information includes an interface execution strategy, the corresponding interactive operation will be completed directly on the current graphical interface according to the strategy requirements, changing the current interface state and propelling the task forward. If the first task execution strategy information includes a context management strategy, the existing context information will be managed according to the strategy requirements, generating new context information needed for the second time step reasoning. Subsequent iterations can continue until the task execution succeeds or fails, ensuring the accuracy and stability of task execution.

[0068] In the task execution method provided in this embodiment, by dynamically managing the context state through the context management strategy output by the task execution agent, the agent can not only refer to the current interface screenshot when making inference decisions at each step, but also combine the managed historical execution context, and automatically take both context management actions and interface actions as inference targets. This achieves the technical objective of making both context management and ordinary interface actions standard actions that can be output, executed, trained, and transferred by the task execution agent.

[0069] This design addresses the problem of accumulated contextual information leading to management failure or redundant information interfering with reasoning in long-term user interface (UI) tasks. By unifying action decision-making and context management within the reasoning scope of the task-executing agent, it achieves integrated and intelligent processing of both. It eliminates the need for separate context management rules or modules, allowing the agent to autonomously determine and execute appropriate context management operations and / or UI operations based on the current task execution situation. This adapts to UI interaction tasks of varying complexity and length, avoiding unnecessary waste of reasoning computational power on contextual information while retaining crucial historical information for current reasoning. This effectively improves the accuracy of action decisions for long-term UI interaction tasks, ensuring successful task execution and achieving the correct final result, thus guaranteeing high-quality and high-success-rate execution of long-term UI interaction tasks.

[0070] In one exemplary implementation, the first task execution strategy information includes: (1) The reasoning text in this step is used to explain the judgment result of the task execution agent on the execution status of the target task, as well as the reasoning process of the execution action.

[0071] In this disclosure, "Thinking" refers to the logical meaning of the reasoning text in this step, utilizing... This refers to the content of the reasoning text for this step. The reasoning text for this step is the text information output by the task-executing agent during the reasoning process of completing the current time step. It fully demonstrates its reasoning logic and records the reasoning judgments. It clearly shows how the agent combines the target task requirements, the global state, and the current interface screenshot information to deduce the first task execution strategy step by step. This provides a clear basis for reasoning in subsequent time steps, helps the agent continue the previous reasoning logic, avoids problems such as logical breaks or contradictory judgments, and improves the interpretability and accuracy of the task reasoning process.

[0072] (2) History folding command, used to specify how to fold and compress history records.

[0073] In this disclosure, "Folding" is used to refer to the logical meaning of the history folding instruction, utilizing... This refers to the content of the history folding instructions. History folding instructions are output by the task execution agent at the current time step, specifying concrete rules for how to fold and compress historical operations. They specify which completed operations, which have no direct impact on subsequent reasoning, need to be streamlined and merged; which invalid intermediate attempts can be omitted or deleted; and which core steps need to be retained in their entirety. This ensures that the folded and compressed history accurately preserves key steps in task progression while eliminating redundant and invalid content.

[0074] The history folding instructions in this disclosure are not fixed preset rules, but are generated autonomously by the task execution agent based on the current progress of the task. This can better adapt to the history management needs of tasks with different complexities, and ensure that the compressed history retains core information without occupying too much inference space.

[0075] (3) Tool call instruction, used to specify the tool and the action to be performed by the tool, wherein the action is at least one of interface action and context management action.

[0076] In this disclosure, "Tool_call" is used to refer to the logical meaning of tool invocation instructions, utilizing... This refers to the content of a tool invocation command. A tool invocation command is output by the task execution agent to invoke a pre-defined tool to complete a specified operation. It can explicitly specify the type of tool to be invoked and the specific action required from that tool. The action performed by the invoked tool can be a direct graphical interface action, a context management action to manage context information, or both. Introducing tool invocation commands encapsulates complex interactive actions and context management actions into standardized tools, treating them uniformly as the agent's inference targets. By simply explicitly invoking the corresponding tool and specifying the action requirements, the tool can complete the corresponding execution, simplifying the agent's inference output process and improving the standardization and accuracy of various action executions.

[0077] (4) Interface observation information, used to record the interface state generated under the current situation.

[0078] In this disclosure, ui_observation is used to refer to the logical meaning of interface observation information, utilizing... This refers to the content of interface observation information. Interface observation information is the record of interface states related to the progress of the current task, made by the task-executing agent at the current time step after perceiving and analyzing the current graphical user interface. This includes key interface information such as currently active windows, selected content areas, newly popped-up dialog boxes, and the status of tabs after clicks. This information facilitates the rapid acquisition of specific data from the interface during the inference process of subsequent time steps, helping the agent to more quickly grasp the actual situation of the current interface and improve the efficiency and accuracy of subsequent inference.

[0079] (5) Action intent information, used to indicate the purpose of the action and its subsequent effects. Among them, the history folding instruction, the interface observation information, and the action intent information belong to the context management strategy, and the tool call instruction belongs to the interface execution strategy or the context management strategy.

[0080] In this disclosure, action_intent is used to refer to the logical meaning of action intent information, utilizing... This refers to the content of action intent information. Action intent information is additional textual information added by the task-executing agent after outputting the action to be performed, explaining the purpose of the action and its subsequent impact on task progress and interface state. By storing action intent information as context, subsequent reasoning can quickly align the execution goal of each step, avoiding misunderstandings of the purpose of previous operations in later steps, and preventing issues such as duplicate operations or logical deviations from the goal. It also makes the entire task execution flow logic clearer and improves the interpretability of the task execution process.

[0081] The first task execution strategy information disclosed herein is the inference result of the first time step. This inference result forms a five-segment output structure, which clearly indicates the historical folding instructions, interface observation information, action intent information, tool invocation instructions, and the inference text for this step. An example of a five-segment output structure is as follows: <thinking> Based on the current screenshot and task objectives, the next step is to save the product prices.< / thinking> <folding> {range:[3,5],summary: Search for product A has been completed and the product details page has been accessed.}< / folding> <tool_call> {name:memory_add,arguments:{id:ml,description:Price of Product A,content:The current price of Product A is 129.99 yuan}< / tool_call> <ui_observation> The page displays product A, priced at 129.99 yuan, with specifications of 8GB+256GB.< / ui_observation> <action_intent> Save the price and specifications of product A for later comparison with product B. when<tool_call> When performing GUI actions, the agent changes the interface state by executing the GUI action and obtains the execution result and the next screenshot.<tool_call> When performing a context management action, the agent performs context management while the interface remains unchanged. This disclosure allows context management to be executed as a separate action, unlike related technologies that distinguish context management actions from interface actions.

[0082] The five-segment output structure disclosed herein makes the reasoning output of the task execution agent more standardized and organized. This facilitates the agent's output of reasoning results according to structured logic and allows subsequent execution modules to quickly break down and read different types of information, performing interface operations and context management separately, thus reducing the probability of errors in information reading and processing. It avoids redundant information consuming reasoning resources while accurately retaining the core content needed for task progression, thereby improving the overall stability and accuracy of long-term GUI task execution.

[0083] Please refer to Figure 2 This illustrates the execution of the target task at the second time step in this disclosure. Figure 1 The step of executing the target task at the second time step based on the first task execution strategy information includes: S210. If the tool call instruction in the first task execution strategy information indicates an interface action, the first tool corresponding to the interface action is used to execute the interface action to obtain the interface action execution result; based on the interface action execution result, a second interface screenshot corresponding to the second time step is obtained.

[0084] This disclosure defines the result of interface action execution as... The screenshot of the second interface is recorded as... In the task execution process, if the tool invocation command in the first task execution strategy information output in the first step specifies that a UI action needs to be performed, the corresponding preset first tool will be triggered to complete the specified UI interaction operation and obtain the corresponding UI action execution result. These tools generally accurately complete the specified UI interaction actions such as clicking, inputting, swiping, and jumping according to the requirements of the tool invocation command, ensuring that the action execution meets the command requirements.

[0085] After the interface action is executed, the state of the current graphical interface will change accordingly. At this time, a new screenshot of the current interface will be acquired and used as the second interface screenshot for the second time step inference. Explicitly recording the interface action execution result and the second interface screenshot can provide accurate basis for subsequent inference, making it convenient for the agent to grasp the actual interface state after the operation in a timely manner and ensuring the accuracy of the inference.

[0086] S220. Update the folding history record according to the history folding instruction.

[0087] This disclosure refers to the updated fold history as After executing the action at the current time step, the relevant historical records need to be folded and compressed according to the historical folding instructions output in the first task execution strategy information, and updated to obtain folded historical records suitable for the next inference step. The update process follows the folding rules output by the agent at the current time step, retaining key records that meet the rule requirements, simplifying redundant and invalid content, and finally generating a new folded historical record adapted to the next inference step. This method of updating the folded historical record in real time according to dynamic instructions ensures that the folded historical record always keeps pace with the progress of the current task, ensuring that the historical records used in the next inference step are concise and core, and avoiding the redundancy and expansion of historical records that affect inference efficiency and accuracy.

[0088] S230. Update the most recent step record based on the interface observation information, the action intention information, the tool call command, and the current step reasoning text.

[0089] This disclosure records the most recent updated steps as follows: The updated recent steps log allows the next time step inference to quickly retrieve the details of the recently completed operation, effectively improving the efficiency of obtaining the latest information in the next inference step and enhancing the accuracy of subsequent inferences.

[0090] For example, = ( , , ), , , These three parameters have been explained earlier and will not be repeated here. For example, when the agent receives information such as "price 129.99 yuan, color black, capacity 256GB" on a product details page,<ui_observation> This precise text can be saved; when the purpose of this step is to "save the product information for subsequent comparisons,"<action_intent> This purpose should be recorded. The next step for the intelligent agent can be based on... The `memory_add` context management action is issued to fold the relevant information for this step into... .

[0091] S240. Based on the updated folding history, the updated recent step record, and the folding interface state that has not been updated, generate second context information corresponding to the second time step; based on the target task and the second context information, generate a second global state corresponding to the second time step.

[0092] In this step, the second global state is denoted as... =(G, The updated folded history, the updated recent step record, and the folded interface state that does not yet need updating are combined to obtain complete second context information. This information is then combined with the original target task to integrate the second global state required for the second time step inference. This combination process preserves the core information of the historical task progression, the complete operation and reasoning details of the latest step, and maintains stable folded interface state information. This allows the task execution agent in the second time step to directly obtain a well-structured and complete global state input, enabling it to directly conduct the next stage of reasoning based on this integrated global state, effectively improving the efficiency and accuracy of the next stage of reasoning.

[0093] S250. Input the second interface screenshot and the second global state into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information; based on the second task execution strategy information, execute the target task at the third time step until the execution result of the target task is obtained, wherein the third time step is the next time step after the second time step.

[0094] After integrating the second global state, the newly acquired screenshot of the second interface, along with the integrated second global state, can be input into the task execution agent. The agent then performs inference in the second time step and outputs the corresponding second task execution strategy information. Following a similar execution flow to the second time step, the target task is advanced in the third time step. Each subsequent time step repeats this "action execution - context update - inference output" loop, advancing the task step by step until the entire target task is completed and the required execution result is obtained. This cyclical, multi-time-step execution framework ensures that each inference step is based on the latest and most accurate information, and that the context is updated promptly after each step. This adapts to the gradual advancement requirements of long-term GUI tasks, enabling the continuous and stable breakdown and completion of complex target tasks. It avoids information chaos or inference bias during task execution, ultimately improving the overall task completion quality.

[0095] Please refer to Figure 3 This illustrates the execution of the target task at the second time step in this disclosure. Figure 2 The step of executing the target task at the second time step based on the first task execution strategy information includes: S310. If the tool invocation instruction in the first task execution strategy information indicates a context management action, the second tool corresponding to the context management action is used to execute the context management action to obtain the managed collapsed interface state.

[0096] This disclosure records the managed collapsed interface state as: When the tool invocation command output in the first time step specifies the need to perform context management actions, it will trigger the corresponding preset second tool to adjust and manage the current collapsed interface state according to the command requirements, resulting in a new, organized collapsed interface state. These context management tools can perform standardized management operations such as merging, filtering, and updating collapsed interface states according to commands, fulfilling the agent's need to organize interface state information. The new collapsed interface state obtained after management will serve as the basis for subsequent inference, ensuring that the interface information used in subsequent inference always meets the specific requirements for task progression.

[0097] S320. Update the folding history record according to the history folding instruction.

[0098] This step is the same as S220 above, so I will not repeat it here.

[0099] S330. Update the most recent step record based on the interface observation information, the action intention information, the tool call command, and the current step reasoning text.

[0100] This step is the same as S330 mentioned above, so I will not repeat it here.

[0101] S340. Based on the updated folding history, the updated recent step record, and the managed folding interface state, generate second context information corresponding to the second time step; based on the target task and the second context information, generate a second global state corresponding to the second time step.

[0102] In this step, the second global state is denoted as... =(G, Similar to the logic of generating the global state in the previous scenario, this step, because the collapsed interface state has been adjusted through context management actions, results in a new managed collapsed interface state. Therefore, the updated collapsed history and the updated recent step record are combined with the newly organized collapsed interface state to generate the second context information. This information is then combined with the target task to obtain the corresponding second global state. This adjustment adapts to the state changes after the context management actions, ensuring that the second global state accurately contains all the core information that has just been updated, making the input information for subsequent reasoning accurate and reliable.

[0103] S350. Input the first interface screenshot and the second global state into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information; based on the second task execution strategy information, execute the target task at the third time step until the execution result of the target task is obtained, wherein the third time step is the next time step after the second time step.

[0104] In this scenario, since no new interface actions are performed, the current state of the graphical interface remains unchanged. The first screenshot obtained in the previous step can still be used as input material for reasoning, eliminating the need to acquire a new screenshot and saving unnecessary screenshot operations. Subsequent reasoning and task progression remain consistent with the aforementioned process and will not be elaborated upon further.

[0105] In one exemplary implementation, the history folding instruction includes a first parameter and a second parameter, the first parameter indicating the range of compressed historical steps, and the second parameter indicating the generated summary; updating the folding history according to the history folding instruction includes: When the first parameter indicates single-step compression, a corresponding summary is generated based on the most recent historical step; when the first parameter indicates multi-step compression, a corresponding summary is generated based on the specified multi-step steps. The generated summary is stored in the second parameter, and the historical records corresponding to the compressed steps are replaced based on the second parameter.

[0106] Illustratively, the historical folding instruction may be expressed as phi_t=([s_t,t],z_t), where [s_t,t] is a first parameter, and z_t is a second parameter. When s_t=t, only the most recent step is compressed into a summary. For example: "Step 5: Open the Settings app and enter the Wi-Fi page". When s_t<t, multiple completed steps are compressed into a summary. For example: "Steps 3 to 7: Search for product A in the shopping application, enter the detail page and record the price and specifications". The compressed summary is used as z_t to replace the compressed steps, and this mechanism significantly reduces the growth rate of context length and reduces noise interference. Both the folding range of the historical folding instruction and the compressed summary are dynamically inferred by the intelligent agent, rather than being determined by a fixed window, a fixed period or an independent summarizer, so that better intelligence is achieved.

[0107] In an exemplary embodiment, the folded interface state includes a plurality of memory items, any one of the memory items includes a memory item identifier, a corresponding memory item description and corresponding original interface content, and the memory item is the minimum operation unit of the context management action.

[0108] Illustratively, a memory item may be expressed as m=(id,d,c), where id is a unique identifier of the memory item, d is a brief description, and c is complete content. c shall store complete facts, such as detailed content of complete prices, complete phone numbers, complete addresses, complete product specifications, complete verification codes or complete comparison conclusions, rather than referential content.

[0109] In some embodiments, the memory item may further include a source step, a source screenshot, a confidence level, a corresponding application, an expiration condition, a time stamp, a visible area coordinate or the like.

[0110] In this embodiment, each memory item includes an independent identifier, a description and complete original content, and by using it as the minimum operation unit of the context management action, the context management action can accurately locate the single memory item that specifically needs to be adjusted, which makes the management operation more accurate and flexible; the independent and clear memory item structure also facilitates the intelligent agent to quickly retrieve and call the required original interface content, and improves the information calling efficiency.

[0111] In an exemplary embodiment, the performing the context management action by using a second tool corresponding to the context management action comprises performing at least one of the following operations on the plurality of memory items: deleting, modifying, adding, querying, merging, or saving as a persistent state.

[0112] When context management actions are required, the corresponding second tool can directly operate on individual memory items as the unit of operation. Based on the action instructions output by the agent, it performs corresponding operations on the memory items in the collapsed interface state: if it is new interface information, it performs an add operation; if it is expired old interface information, it performs a delete operation; if the information has changed, it performs a modify operation; if it is necessary to call existing information, it performs a query operation; if multiple related information can be integrated, it performs a merge operation; and if it is necessary to retain key information across steps for a long time, it performs a save as a persistent state.

[0113] The rich variety of operation types can meet various needs for managing interface state information. It can adjust the content stored in the collapsed interface state in a timely manner according to the actual changes in the progress of the task, so that the collapsed interface state always fits the current actual interface situation. It can better leverage the advantages of dynamic context management and ensure the accuracy and conciseness of context information.

[0114] This implementation method is applicable to the following scenarios: 1. The current interface displays precise information such as price, number, address, verification code, and product specifications, and this information will be used in subsequent steps.

[0115] 2. The agent is about to switch applications or return to the previous screen, and the current screen information may not be visible in the future.

[0116] 3. Previous memory items have been corrected by the new interface and need to be updated.

[0117] 4. The single step corresponding to a certain memory item has been completed, and it is no longer necessary to retain the corresponding memory item.

[0118] This implementation method can perform autonomous context management in these scenarios, ensuring that the agent always plans its next action based on the latest and correct information.

[0119] Please refer to Figure 4 This diagram illustrates the task execution process framework of this disclosure. First, this disclosure explains the meaning of each term in the diagram; please refer to the table below:

[0120] In general, the core process of this diagram can be summarized into the following main steps:

[0121] In one exemplary embodiment, the task execution agent is trained using the following method: S410. Obtain the basic task, wherein the basic task is a standard interface interaction task with a fixed scene, fixed operation object and fixed process.

[0122] Basic tasks are training samples used to enable task-executing agents to learn basic GUI interaction logic. These tasks typically have clear scenarios, fixed objects of operation, and clear and standardized execution processes. They do not require complex multi-step reasoning adjustments and allow agents to first master the basic rules of interface interaction, the logic of calling common tools, and the reasoning methods of basic tasks, laying the foundation for handling complex long-term tasks.

[0123] S420. Perform differential amplification processing on the basic task to obtain a diversified task pool, the diversified task pool including multiple sample tasks, and the differential amplification processing including at least one of entity replacement, memory operation enhancement, and task simplification.

[0124] Entity replacement refers to replacing the interactive entities in the original basic task with other similar entities. For example, if the original task involved setting up a Wi-Fi option within an application, the expanded version could replace it with a Bluetooth option, a display settings option, or a function button from another application. The overall interaction logic and operational framework remain unchanged; only the specific entity content is adjusted. This expansion method can quickly generate a large number of different training samples without altering the core inference logic. This allows the agent to learn the universal interaction methods of similar entities in different scenarios, preventing the agent from only remembering the operations of specific entities and being unable to transfer them to interaction tasks with other entities. This effectively improves the generalization ability of the trained agent, allowing it to adapt to the GUI interaction requirements of different interfaces and applications.

[0125] Memory operation augmentation refers to adding additional steps requiring context management operations to the original basic task. For example, adding nodes during task progression that require adding key information, merging related memory items, or correcting old memory items, thus including more explicit context management training scenarios in the samples. This type of augmentation can specifically improve the agent's ability to judge and execute context management actions, allowing the agent to learn when to adjust memory items and how to execute different memory operations, thereby strengthening the agent's dynamic context management capabilities.

[0126] Task simplification refers to breaking down a basic task with a long process and many steps into multiple sub-tasks with shorter steps and a single goal, thus decomposing complex multi-step tasks into multiple simple single-step training samples. This augmentation process helps the agent learn gradually from easy to difficult, first mastering the correct reasoning and operation methods for single steps, and then gradually learning the coherent task progression of multiple steps. This avoids training the agent on complex and long tasks from the beginning, which would prevent the agent from learning the correct basic logic, and gradually builds the ability to process tasks from simple to complex.

[0127] S430. Train the task execution agent based on each of the sample tasks in the diversified task pool.

[0128] After obtaining a diverse task pool covering different scenarios and difficulties, each sample task can be sequentially input into the task execution agent to be trained. Based on the sample tasks and initial interface information, the agent gradually outputs inference results and execution actions. Then, the agent's output is compared with the standard results of the sample tasks, the corresponding loss value is calculated, and the agent's model parameters are updated in reverse. This allows the agent to gradually adjust its inference logic and learn GUI interaction and context management methods that meet the requirements. This training method based on an expanded, diverse task pool covers both basic interaction capabilities and strengthens the core capability of dynamic context management. It enables the trained task execution agent to possess both basic interface interaction capabilities and long-term task context management capabilities, better adapting to the execution needs of complex real-world GUI tasks.

[0129] In one exemplary implementation, training the task execution agent based on each of the sample tasks in the diversified task pool includes: S431. For each of the sample tasks, execute the sample task based on the teacher model, and record the task execution strategy information output at each time step; parse the task execution strategy information to obtain the step execution status information corresponding to each of the time steps; and obtain the interaction trajectory information based on the step execution status information.

[0130] In this disclosure, both the teacher and student models can be built upon a larger model. The teacher model has mastered mature GUI task reasoning and context management capabilities. When executed by the teacher model, it can output reasoning processes and operational strategies that conform to standard logic. The decision logic and execution results of each time step are completely recorded. Then, from the recorded task execution strategy information of each time step, the interface state, memory operation, action selection, and other step execution state information of that step are parsed. Finally, the state information of all time steps is integrated according to the execution order to obtain complete interaction trajectory information. These interaction trajectories can provide clear and accurate learning references for the subsequent training of the task execution agent, allowing the task execution agent to directly learn the correct reasoning and decision-making paths, reducing the exploration cost during the training process, and improving the training efficiency and the decision quality of the final model.

[0131] S432. Filter the interaction trajectory information that meets the preset quality requirements to obtain the target interaction trajectory information.

[0132] Some trajectories may contain errors in reasoning logic, non-standard operating procedures, or inadequate context management. Directly using such low-quality trajectories for training could mislead the learning direction of the task-executing agent, affecting the final model's performance. Therefore, it is necessary to screen and verify all generated interaction trajectories according to preset quality requirements, filtering out low-quality trajectories that do not meet the standards, and retaining only the target interaction trajectory information with clear reasoning logic, correct operating procedures, and standardized context management.

[0133] S433. Each step in the target interaction trajectory information is labeled to obtain a labeled step sequence.

[0134] The target interaction trajectory generated by the teacher model only records the actual execution process. It's also necessary to clearly label the decision-making basis, operation type, and expected goal of each step, allowing the task-execution agent to clearly learn "why this action is performed in the current step" and "what type of operation this action belongs to." The labeling typically includes core information such as the task goal of the current step, the current interface observation results, the type of action to be performed, whether memory items need adjustment, and what operations need to be performed on which memory items. Arranging all the labeled steps sequentially yields a sequence of labeled steps suitable for supervised training. Clear and explicit single-step labeling transforms implicit standard reasoning logic into explicit training labels, enabling the task-execution agent to accurately align with standard decision-making logic and quickly learn correct reasoning and operational habits during supervised learning.

[0135] S434. Based on the labeled sequence of steps, train the student model, and use the trained student model as the task execution agent.

[0136] Based on labeled standard step sequences, supervised training can be conducted on the student model, which is the initial task execution agent to be trained. The student model autonomously outputs the decision result of the current step, and then the output result is compared with the labeled standard actions and decision logic. The corresponding training loss is calculated, and the model parameters of the student model are iteratively updated through backpropagation, gradually aligning the student model's decision logic with the labeled standard logic. This distillation-style training method based on teacher model demonstration can efficiently transfer the mature reasoning and context management capabilities already mastered by the teacher model to the student model. This not only accelerates the training convergence speed but also ensures that the final task execution agent possesses the required decision-making capabilities, with clearer output action logic and more standardized context management, better adapting to the execution needs of actual GUI tasks.

[0137] This disclosure does not limit the student model, which can be a visual language model, a large language model with tool-calling capabilities, a multi-model cascade system, a combination of a small edge model and a large cloud model, or a lightweight model that has been distilled. It also does not limit the training method, such as using full-parameter SFT, LoRA SFT, distillation, preference optimization, reinforcement learning, human feedback training, or multi-stage training.

[0138] Please refer to Figure 5 The diagram illustrates the sample generation process of this disclosure. The entire pipeline can be divided into five core stages: Expanded MemGUI Tasks, MemGUIAVD / Emulator, Worker Progress, Teacher Model Generation Interaction, and Trajectory Evaluation and Selection. The final output is the MemGUI-3K dataset (a sample set obtained from a diversified task pool), containing 2,956 interactive trajectories and 64,430 effective steps for training student models.

[0139] Next, the terms in the diagram will be explained. Please refer to the table below:

[0140] This disclosure can provide multiple virtual devices (Device1 to Device) n This involves a parallel simulation environment that mimics the GUI interface of a real mobile phone / tablet, allowing the intelligent agent to perform tasks, observe the interface, and output operations within it. The technical terminology for this stage is explained below:

[0141] This process involves scheduling multiple devices to run simultaneously through task parallelism, recording screenshots and operation logs during the interaction process. The technical terminology for this step is explained below:

[0142] The teacher model generates reasonable instructions for the next step, translates them into actionable actions that are then fed back to the simulator. The technical terminology for this step is explained below:

[0143] The interaction trajectories generated by the teacher model are automatically evaluated, and high-quality data with reasonable steps and task completion are selected to ultimately construct the MemGUI-3K dataset. The technical terminology for this step is explained below:

[0144] visible, Figure 5The sample generation process illustrated involves expanding the original baseline task during the task preparation phase to generate diverse GUI interaction tasks of varying difficulty. Through parallel execution, the teacher model drives the task execution across multiple virtual devices, recording interface screenshots and operation logs. Self-evaluation assesses the rationality of the generated trajectories and the degree of task completion, filtering for high-quality data. In the data output phase, reasonable trajectories are retained to construct the MemGUI-3K dataset for subsequent training of lightweight GUI agents. A failure retry mechanism ensures that unreasonable trajectories are re-executed, guaranteeing the quality of the final dataset.

[0145] Combination Figure 5 This disclosure describes the entire training process, which includes the following basic steps: Step T101: Task Extension 1. Basic Input Based on basic tasks, which are standardized GUI interaction tasks with fixed scenarios, fixed operation objects, and fixed processes, this dataset covers various mobile GUI operation scenarios such as shopping, travel, tool query, and parameter comparison. It serves as the basic prototype for the diversity of dataset tasks.

[0146] 2. Specific Implementation and Handling To address the issues of limited original seed task scenarios and severe sample homogeneity, three differentiated amplification strategies are employed to generate entirely new tasks in batches, thus constructing a diversified extended task pool. The specific implementation methods are as follows: (1) Entity replacement: The core interactive objects and target entities in the task are precisely replaced while retaining the original GUI operation logic and step framework. For example, "filter mobile phone products" is replaced with "filter Bluetooth headphone products", and "query car routes" is replaced with "query bus routes". Without changing the task operation process, the task scenario and interactive content are enriched, and the model is avoided from overfitting to fixed entity tasks.

[0147] (2) Enhanced memory operations: To address the weakness of weak memory dependence in the original task, long-term memory requirements such as cross-page information extraction, multi-step data comparison, and historical operation association judgment are added. This forces the agent to retain, call up, and compare historical interface information during the interaction process, adapting to the memory dependence characteristics of real complex GUI scenarios, and improving the task difficulty and model generalization ability.

[0148] (3) Task simplification: Complex and redundant composite tasks are broken down into independent and standardized basic sub-tasks. After the breakdown, the single task objectives are more focused and the operation steps are simpler, which can cover the core operations of basic GUI such as clicking, sliding, inputting, and filtering, and fill the training sample gaps in simple interaction scenarios.

[0149] 3. Final Output Generate a diverse task pool with large scale, multiple scenarios, and varying levels of difficulty.

[0150] Step T102: Introduction of the Teacher Model 1. Basic Input Based on a diversified task pool and a virtual device execution environment, it provides task materials and operating platforms for automated task execution.

[0151] 2. Specific Implementation and Handling Based on a high-precision teacher model and combined with the five-stage output proposed in this publication, a fully automated task rollout (trajectory extrapolation execution) is completed. The complete implementation process is as follows: The virtual device is initialized and the corresponding sample task is loaded. The simulator captures observation information such as GUI interface screenshots, device status, and historical operation records in real time. The teacher model performs five-stage reasoning and automatically plans and executes the sample task, retaining the results of the reasoning and the data generated during the task execution process to construct mutual trajectories.

[0152] Step T103: Trajectory-level evaluation 1. Basic Input The input is the interactive trajectory generated by T102.

[0153] 2. Specific Implementation and Handling A global trajectory-level coarse screening is conducted, and quality verification is performed on a complete task-by-task basis. The main criteria for judging the two core indicators are: first, whether the task is complete and closed-loop and successfully achieves the preset goal; and second, whether the operation process of the entire trajectory conforms to the scenario logic, has no invalid operations, no logical conflicts, and no process jump errors.

[0154] For trajectories where the task is fully completed and the overall process is compliant, they are deemed qualified and proceed to the next stage; for unqualified trajectories where the task is interrupted, the objective is not achieved, or the logic throughout is chaotic, they are deemed unqualified and a retry mechanism is triggered, returning to T102 to regenerate the trajectory. This step only performs overall screening and does not interfere with the detailed judgment of individual steps.

[0155] 3. Final Output The set of qualified overall trajectories after filtering out failed, incomplete, and illegal trajectories.

[0156] Step T104: Step labeling 1. Basic Input Each independent operation in the trajectory is taken as the smallest processing unit.

[0157] 2. Specific Implementation and Handling Conduct fine-grained, single-step precision annotation and quality assessment: (1) Binary labeling of step rationality: Each step in the trajectory is reviewed one by one to determine whether the operation is in line with the current GUI scenario, whether it fits the task progression logic, and whether it is a valid and necessary operation, and the "rational / unreasonable" label is marked. Problematic steps such as accidental clicks, invalid swipes, repeated operations, and logical inversions are eliminated.

[0158] (2) Step impact polarity labeling: All reasonable steps are further refined and classified, and the impact attributes of a single step operation on the overall task are labeled, divided into three categories: positive, negative, and neutral. Positive steps are key operations that drive the task progress, negative steps are operations that slightly hinder the task (which can be corrected), and neutral steps are auxiliary operations that do not affect the task progress.

[0159] 3. Final Output A high-quality collection of materials that has undergone a series of screening, labeling, and purification processes.

[0160] Step T105: Sample Transformation 1. Basic Input High-quality, logically structured data.

[0161] 2. Specific Implementation and Handling The single-step GUI interaction data is standardized and converted into chat-style SFT supervised samples adapted for fine-tuning of multimodal models, unifying the model input and output formats. The specific reconstruction rules are as follows: Sample input construction: Four types of core information are integrated as model input, namely general system prompt words (defining the GUI agent's task identity and capability requirements), original user task instructions (clarifying the interaction goal), structured context data (historical operations, page status, and memory information), and current GUI interface screenshot (visual input), to construct a complete and multi-dimensional model input context.

[0162] Sample output construction: Based on the five-stage reasoning logic of the teacher model, standardized structured responses are generated to unify the output paradigm of all samples.

[0163] 3. Final Output High-quality SFT supervised samples with unified format, standardized logic, and adaptability for fine-tuning training, namely MemGUI-3K core training samples.

[0164] Step T106: Student Model Training 1. Basic Input Training is performed using standardized SFT training samples and an open-source multimodal pedestal model.

[0165] 2. Specific Implementation and Handling Two mainstream fine-tuning schemes are used for model training, which can be selected as needed: one is full parameter SFT (supervised fine-tuning), which iteratively optimizes all parameters of the model; the other is LoRA SFT lightweight fine-tuning, which trains only part of the model to adapt the parameters, balancing training efficiency and effect.

[0166] During training, the five-stage reasoning output of the teacher model and the standardized GUI operation logic are used as supervision signals to enable the base model to learn the ability of GUI interface understanding, task reasoning, step planning, and precise interaction, thus fitting the behavioral paradigm of a high-quality teacher model and avoiding the negative guidance of inferior samples, resulting in a lightweight student GUI agent model with independent GUI interaction capabilities.

[0167] In one specific embodiment, starting from 128 long-range mobile GUI seed tasks, 7,303 candidate sample tasks are generated through three strategies: entity replacement, memory operation enhancement, and task simplification. 5,293 sample tasks are then selected for the teacher model's rollout. The teacher model executes the tasks in the mobile environment using a complete five-stage output protocol. After task-level evaluation and filtering of the trajectories, 2,956 successful trajectories are obtained, and these are divided into training and testing sets at 90% and 10% respectively. Further extraction of appropriate steps yields 64,430 supervised training samples, of which 57,951 are used for training and 6,479 for testing.

[0168] Each supervised sample can include: prompts, task descriptions, structured context states, current screenshots, and the gold standard five-segment output. Through these samples, the student model not only learns GUI actions such as clicking, inputting, and swiping, but also learns various inference objectives such as memory triggering, memory content generation, historical folding range selection, summary generation, interface observation generation, and action intent generation.

[0169] The trained agent does not require an additional external memory agent during the inference phase; instead, it can independently perform step-by-step reasoning until the task is completed. If the output format is invalid, it can automatically perform format validation, retry, rollback to a waiting state, or terminate the current task and log the error. If a GUI action fails, the failure result can be written to... And in the next step The system prompts the model to restore it. If the remembered action references a non-existent ID, the interface memory management module can refuse to update and return an error result for the next step of model correction.

[0170] To give a specific example, Figure 6 This diagram illustrates the execution process of a long-running GUI task. The user requests an agent to compare the specifications of two products in a shopping application and write the comparison results to a notes application. The execution process of this task is as follows: Receive user task target G, initialize , and .

[0171] Get a screenshot of the shopping app's homepage and put and Input task execution agent.

[0172] The task-executing agent outputs the GUI action of clicking the search box, and simultaneously outputs interface observation information and action intent information.

[0173] Perform a click and input action to navigate to the search results or details page for product A.

[0174] When the page displays the full price and specifications of product A, the agent...<ui_observation> Record precisely visible content and in<tool_call> Output memory_add, which writes the price and specifications of product A to the memory. .

[0175] Continue searching for product B, and save the price and specifications of product B using memory_add.

[0176] After the product search is complete, <folding>Specify the historical range, collapse the multi-step search process into a summary of "Information collection for Product A and Product B completed", and write it into the file. .

[0177] Switch to the notes application. The original shopping page will no longer be visible, but... The facts of the two goods are still preserved. Save the progress of completed searches.

[0178] Model reading Then enter the comparison results in the note-taking application.

[0179] When the task is completed, the model outputs "terminate", indicating that the task has ended.

[0180] In this embodiment, This solves the problem of lengthy, multi-step historical processes. This resolves the issue of accurately preserving facts after leaving the original interface. This solves the problem of reusing the purpose of the most recent observation and action.

[0181] To give another concrete example, a user requests an agent to find contact information from a social media app and send a specified message via a messaging app. The agent first opens the contacts page in the social media app, reads the contact's name, nickname, or phone number, and writes it using `memory_add`. Then switch to the Contacts or Messages app, and the smart agent uses... The system uses the stored complete number and name to perform a search, select a recipient, and enter the SMS message content. If the interface displays a contact number update, the model can output a `memory_add` correction. The system already contains memory entries; once the SMS message is sent, the model can output `memory_add` to delete the temporary number that is no longer needed, reducing subsequent contextual interference.

[0182] This implementation demonstrates that the present disclosure is applicable not only to product comparison, but also to cross-application information migration, contact retrieval, calendar recording, form filling, message sending, and other GUI automation tasks that require long-term memory.

[0183] Please refer to Figure 7 This diagram illustrates a comparison between context length and task effectiveness. The following is a description of the diagram: This diagram illustrates the two core advantages of this disclosure in long-running GUI (Graphical User Interface) tasks: lower context cost and higher task success rate. Regarding Lower Context Cost: This graph illustrates how the average number of input tokens varies with the number of steps when an agent performs a GUI task.

[0184] The horizontal axis (Step) represents the number of steps required to complete the task, ranging from 0 to 150. The "Long-horizon region" indicates that the test involves a complex GUI task that requires multiple interactive steps.

[0185] The vertical axis (Average Input Tokens) represents the number of tokens the agent inputs each time. More tokens mean higher context costs, higher computation / reasoning costs, and slower speed.

[0186] Blue line (React): In the traditional React strategy, the number of input tokens grows almost linearly and rapidly as the number of steps increases, and the later the step, the heavier the context burden.

[0187] Purple line (Context): The proposed scheme in this disclosure results in a very gradual increase in the number of tokens, which even tends to stabilize in the later stages, achieving "Flatter token growth".

[0188] At step 150, the proposed scheme (referred to as Context) reduces the average number of input tokens by 1.5k compared to traditional React strategies. This means that in long-cycle tasks, the pressure on the context window is significantly reduced, and the inference cost is lowered; the requirement for context length is reduced, making it suitable for more small and medium-sized models; and the response speed is faster, avoiding the efficiency bottleneck caused by token explosion.

[0189] Regarding Higher Success Rate: This section compares the success rates of different agents in two authoritative GUI task benchmark tests (the Pass@3 metric, which is the percentage of tasks successfully completed within three attempts).

[0190] 1. MemGUI-Bench: A cross-application long-cycle GUI task benchmark, consisting of 128 tasks, with an average of 35.2 steps to complete, and cross-application tasks accounting for 78.17%.

[0191] 2. MobileWorld: Mobile GUI task benchmark, with a total of 117 tasks, requiring an average of 27.8 steps to complete, and cross-application tasks accounting for 62.2%.

[0192] Light gray bars (Baselines): Baseline performance of different mainstream agents, including: Light purple bars (Trained on MemGUI-3K): Agent performance after fine-tuning on the MemGUI dataset.

[0193] Deep purple bars (Zero-shot with Context): Zero-shot performance of the MemGUI-Agent (with Context mechanism) disclosed in this publication.

[0194] On MemGUI-Bench, the baseline model achieved a maximum success rate of approximately 49.2%, while MemGUI-Agent achieved a zero-sample success rate of 62.5%, an improvement of +13.3% over the baseline and +26.6% over the fine-tuned MemG3 (HR-SFT, 35.9%).

[0195] On MobileWorld, the baseline model achieved a maximum success rate of approximately 26.3% (Dual-1.5 / UI-TARS), while MemGUI-Agent achieved a zero-sample success rate of 29.1%, an improvement of +14.6% over the baseline and +11.2% over the finely tuned MemG3 (HR-SFT, 17.9%).

[0196] As can be seen, the MemGUI-Agent disclosed herein has achieved state-of-the-art results among open-source models in long-cycle GUI task benchmarks. It solves the context explosion problem in long-cycle tasks: through more efficient context management, the number of input tokens no longer increases linearly with each step, significantly reducing costs; while optimizing costs, the task success rate is significantly improved: it surpasses existing baseline models on two authoritative benchmarks, and even outperforms the fine-tuned model, achieving the dual advantages of "low cost + high performance".

[0197] Based on the figure, it can be seen that this disclosure has achieved at least the following beneficial effects: (1) Reduce context bloat in long-term tasks. Through the history folding mechanism, completed multi-step sub-tasks can be compressed into a summary, so that the context growth rate is lower than that of mechanically appending history.

[0198] (2) Improve the fidelity of key facts. Through Preserving the complete interface information allows you to continue using accurate information such as price, number, address, and specifications across applications, after returning to the previous page, or after a long delay.

[0199] (3) Improve the success rate of cross-application tasks. Allow agents to actively save key information before switching applications and directly read and use it in subsequent applications.

[0200] (4) Reduce context-induced hallucinations. By separating recent step records, history folding, and interface fact memory, it is less likely for users to confuse completed progress, visible facts, and speculative content.

[0201] (5) Simplify system structure. Context maintenance actions and interface actions are output by the same agent, eliminating the need to deploy complex external memory agents, retrieval agents, or rule summarizers separately.

[0202] (5) Improve trainability. The five-stage output protocol makes folding, memorization, observation and action intention explicit, so that context management behavior can be trained as a supervised target.

[0203] (6) Supports the ability to manage the learning context of smaller-scale models.

[0204] (7) Improved context management sub-skills. Memory-triggered F1 increased from 19.9% ​​to 48.0%, deep folding ratio increased from 8.8% to 26.1%, and format compliance rate reached 99.9%.

[0205] (8) Reduce the number of failures. Compared to ReAct, the total number of failures decreased from 99 to 58, a reduction of about 41%, with a significant decrease in process hallucinations and output hallucinations.

[0206] Figure 8 This is a block diagram illustrating a task execution apparatus according to an exemplary embodiment. The apparatus includes: The basic information acquisition module 810 is configured to acquire the target task and the first context information corresponding to the first time step. The target task is an interface interaction task, which is a task that, based on the task execution target, determines the first interaction target in the interface screenshot of the first application, and determines the second interaction target after generating the interface screenshot of the second application through interaction with the first interaction target, and continues to interact with the second interaction target until the task execution target is achieved. Both the first application and the second application are applications with a visual interface containing the interaction target. The first context information is used to indicate the context state managed by the context management strategy output by the task execution agent during the execution of the target task to the first time step. The global state construction module 820 is configured to construct a first global state corresponding to the first time step based on the target task and the first context information. The screenshot acquisition module 830 is configured to acquire the first interface screenshot corresponding to the first time step; The task execution module 840 is configured to input the first interface screenshot and the first global state into the task execution agent to perform task execution reasoning and obtain first task execution strategy information, which includes at least one of interface execution strategy and context management strategy; based on the first task execution strategy information, the target task is executed at a second time step until the execution result of the target task is obtained, where the second time step is the next time step after the first time step.

[0207] In one exemplary implementation, the first task execution strategy information includes: The reasoning text in this step is used to explain the judgment result of the task execution agent on the execution status of the target task, as well as the reasoning process of the execution action; The history folding command specifies how to fold and compress historical records; A tool invocation instruction is used to specify a tool and the action to be performed by the tool, wherein the action is at least one of an interface action and a context management action; Interface observation information is used to record the interface data generated in the current situation; Action intent information, used to indicate the purpose of the action and its subsequent effects; The history folding instruction, the interface observation information, and the action intent information belong to the context management strategy, while the tool call instruction belongs to the interface execution strategy or the context management strategy.

[0208] In one exemplary implementation, the first context information corresponding to the first time step includes: Collapsed history, used to indicate the history formed after compression since the start of the execution of the target task; The collapsed interface state is used to record the interface records stored since the start of the execution of the target task; Recent step record, used to indicate the task execution details corresponding to at least one recent historical time step.

[0209] In one exemplary embodiment, the task execution module 840 is configured to execute: If the tool invocation instruction in the first task execution strategy information indicates an interface action, the first tool corresponding to the interface action is used to execute the interface action to obtain the interface action execution result; based on the interface action execution result, a second interface screenshot corresponding to the second time step is obtained; Update the folding history record according to the historical folding instructions; Update the most recent step record based on the interface observation information, the action intent information, the tool call command, and the current step reasoning text; Based on the updated folding history, the updated recent step record, and the folded interface state that has not been updated, the second context information corresponding to the second time step is generated; Based on the target task and the second context information, a second global state corresponding to the second time step is generated; The screenshot of the second interface and the second global state are input into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information. Based on the second task execution strategy information, the target task is executed at the third time step until the execution result of the target task is obtained. The third time step is the next time step after the second time step.

[0210] In one exemplary embodiment, the task execution module 840 is configured to execute: If the tool invocation instruction in the first task execution strategy information indicates a context management action, the second tool corresponding to the context management action is used to execute the context management action to obtain the managed collapsed interface state; Update the folding history record according to the historical folding instructions; Update the most recent step record based on the interface observation information, the action intent information, the tool call command, and the current step reasoning text; Based on the updated folding history, the updated recent step record, and the managed folding interface state, generate the second context information corresponding to the second time step; Based on the target task and the second context information, a second global state corresponding to the second time step is generated; The first interface screenshot and the second global state are input into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information. Based on the second task execution strategy information, the target task is executed at the third time step until the execution result of the target task is obtained. The third time step is the next time step after the second time step.

[0211] In one exemplary implementation, the history folding instruction includes a first parameter and a second parameter, the first parameter indicating the range of the compressed history steps, and the second parameter indicating the generated summary; the task execution module 840 is configured to execute: When the first parameter indicates single-step compression, a corresponding summary is generated based on the most recent historical step; when the first parameter indicates multi-step compression, a corresponding summary is generated based on the specified multi-step steps. The generated summary is stored in the second parameter, and the historical records corresponding to the compressed steps are replaced based on the second parameter.

[0212] In one exemplary embodiment, the folded interface state includes multiple memory items, each of which includes a memory item identifier, a corresponding memory item description, and the corresponding original interface content. The memory item is the smallest operational unit of the context management action.

[0213] In one exemplary implementation, the execution of the context management action using the second tool corresponding to the context management action includes performing at least one of the following operations on the plurality of memory items: deleting, modifying, adding, querying, merging, or saving as a persistent state.

[0214] In one exemplary embodiment, the task execution module 840 is configured to execute: Obtain basic tasks, which are standard interface interaction tasks with fixed scenarios, fixed operation objects, and fixed processes; The basic task is subjected to differential amplification processing to obtain a diversified task pool, which includes multiple sample tasks. The differential amplification processing includes at least one of entity replacement, memory operation enhancement, and task simplification. The task execution agent is trained based on the sample tasks in the diversified task pool.

[0215] In one exemplary embodiment, the task execution module 840 is configured to execute: For each of the sample tasks, the sample task is executed based on the teacher model, and the task execution strategy information output at each time step is recorded; the task execution strategy information is parsed to obtain the step execution state information corresponding to each of the time steps; and the interaction trajectory information is obtained based on the step execution state information. Filter the interaction trajectory information that meets the preset quality requirements to obtain the target interaction trajectory information; Each step in the target interaction trajectory information is labeled to obtain a labeled step sequence; Based on the labeled sequence of steps, a student model is trained, and the trained student model is used as the task execution agent.

[0216] Regarding the apparatus in the above embodiments, the specific manner of each step has been described in detail in the embodiments of the foregoing method, and will not be elaborated here.

[0217] Please refer to Figure 9 It illustrates the structural block of a computer device provided in an exemplary embodiment of this disclosure. Figure 1 The computer device may be a terminal. This computer device is used to implement the task execution method provided in the above embodiments. Specifically: Typically, computer device 900 includes a processor 901 and a memory 902.

[0218] Processor 901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In an exemplary embodiment, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In an exemplary embodiment, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0219] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In an exemplary embodiment, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, at least one program, code set, or instruction set, configured to be executed by one or more processors to implement the task execution method described above.

[0220] In one exemplary embodiment, the computer device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a touch display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

[0221] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the computer device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0222] Please refer to Figure 10 It illustrates the structural block of a computer device provided in another exemplary embodiment of this disclosure. Figure 2 The computer device can be a server to perform the above-described task execution method. Specifically: Computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including random access memory (RAM) 1002 and read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. Computer device 1000 also includes a basic input / output system (I / O system) 1006 that facilitates information transfer between various devices within the computer, and a mass storage device 1007 for storing the operating system 1013, application programs 1014, and other program modules 1011.

[0223] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1100 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1100 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1100 also provides output to a display screen, printer, or other types of output devices.

[0224] Mass storage device 1007 is connected to central processing unit 1001 via a mass storage controller (not shown) connected to system bus 1005. Mass storage device 1007 and its associated computer-readable media provide non-volatile storage for computer device 1000. That is, mass storage device 1007 may include computer-readable media (not shown) such as hard disk or CD-ROM (Compact Disc Read-Only Memory) drive.

[0225] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1004 and mass storage device 1007 described above can be collectively referred to as memory.

[0226] According to various embodiments of this disclosure, the computer device 1000 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1012 via a network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer systems (not shown).

[0227] The aforementioned memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the aforementioned task execution method.

[0228] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is executed by a processor to implement the task execution method.

[0229] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0230] In an exemplary embodiment, a computer-readable storage medium including program code is also provided, such as a memory including program code, which can be executed by a processor to perform the above-described task execution method. Optionally, the computer-readable storage medium may be read-only memory (ROM), random access memory (RAM), read-only optical disc (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0231] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the task execution method described above.

[0232] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0233] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.< / folding>

Claims

1. A task execution method characterized by, The method includes: The system acquires first context information corresponding to a target task and a first time step. The target task is a user interface interaction task, which is determined based on the task execution target. A first interaction target is identified in a screenshot of a first application's interface, and a second interaction target is identified after an interface screenshot of a second application is generated through interaction with the first interaction target. Interaction with the second interaction target continues until the task execution target is achieved. Both the first and second applications are applications with visual interfaces containing interaction targets. The first context information indicates the context state managed by the context management strategy output by the task execution agent during the execution of the target task up to the first time step. Based on the target task and the first context information, construct the first global state corresponding to the first time step; Obtain the screenshot of the first interface corresponding to the first time step; The first interface screenshot and the first global state are input into the task execution agent to perform task execution reasoning, and the first task execution strategy information is obtained. The first task execution strategy information includes at least one of the interface execution strategy and the context management strategy. Based on the first task execution strategy information, the target task is executed at the second time step until the execution result of the target task is obtained. The second time step is the next time step after the first time step.

2. The method of claim 1, wherein, The first task execution strategy information includes: The reasoning text in this step is used to explain the judgment result of the task execution agent on the execution status of the target task, as well as the reasoning process of the execution action; The history folding command specifies how to fold and compress historical records; A tool invocation instruction is used to specify a tool and the action to be performed by the tool, wherein the action is at least one of an interface action and a context management action; Interface observation information is used to record the interface data generated in the current situation; Action intent information, used to indicate the purpose of the action and its subsequent effects; The history folding instruction, the interface observation information, and the action intent information belong to the context management strategy, while the tool call instruction belongs to the interface execution strategy or the context management strategy.

3. The method of claim 2, wherein, The first context information corresponding to the first time step includes: Collapsed history, used to indicate the history formed after compression since the start of the execution of the target task; The collapsed interface state is used to record the interface records stored since the start of the execution of the target task; Recent step record, used to indicate the task execution details corresponding to at least one recent historical time step.

4. The method of claim 3, wherein, The step of executing the target task at the second time step based on the first task execution strategy information includes: If the tool invocation instruction in the first task execution strategy information indicates an interface action, the first tool corresponding to the interface action is used to execute the interface action to obtain the interface action execution result; based on the interface action execution result, a second interface screenshot corresponding to the second time step is obtained; Update the folding history record according to the historical folding instructions; Update the most recent step record based on the interface observation information, the action intent information, the tool call command, and the current step reasoning text; Based on the updated folding history, the updated recent step record, and the folded interface state that has not been updated, the second context information corresponding to the second time step is generated; Based on the target task and the second context information, a second global state corresponding to the second time step is generated; The screenshot of the second interface and the second global state are input into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information. Based on the second task execution strategy information, the target task is executed at the third time step until the execution result of the target task is obtained. The third time step is the next time step after the second time step.

5. The method according to claim 3, characterized in that, The step of executing the target task at the second time step based on the first task execution strategy information includes: If the tool invocation instruction in the first task execution strategy information indicates a context management action, the second tool corresponding to the context management action is used to execute the context management action to obtain the managed collapsed interface state; Update the folding history record according to the historical folding instructions; Update the most recent step record based on the interface observation information, the action intent information, the tool call command, and the current step reasoning text; Based on the updated folding history, the updated recent step record, and the managed folding interface state, generate the second context information corresponding to the second time step; Based on the target task and the second context information, a second global state corresponding to the second time step is generated; The first interface screenshot and the second global state are input into the task execution agent to continue task execution reasoning and obtain the second task execution strategy information. Based on the second task execution strategy information, the target task is executed at the third time step until the execution result of the target task is obtained. The third time step is the next time step after the second time step.

6. The method according to claim 4 or 5, characterized in that, The history folding instruction includes a first parameter and a second parameter, wherein the first parameter indicates the range of the compressed history steps and the second parameter indicates the generated summary. The step of updating the folding history record according to the historical folding instruction includes: When the first parameter indicates single-step compression, a corresponding summary is generated based on the most recent historical step; when the first parameter indicates multi-step compression, a corresponding summary is generated based on the specified multi-step steps. The generated summary is stored in the second parameter, and the historical records corresponding to the compressed steps are replaced based on the second parameter.

7. The method according to claim 5, characterized in that, The folded interface state includes multiple memory items. Each memory item includes a memory item identifier, a corresponding memory item description, and the corresponding original interface content. The memory item is the smallest operational unit of the context management action.

8. The method according to claim 7, characterized in that, The execution of the context management action using the second tool corresponding to the context management action includes performing at least one of the following operations on the plurality of memory items: deletion, modification, addition, query, merging, or saving as a persistent state.

9. The method according to claim 2, characterized in that, The task execution agent is trained using the following method: Obtain basic tasks, which are standard interface interaction tasks with fixed scenarios, fixed operation objects, and fixed processes; The basic task is subjected to differential amplification processing to obtain a diversified task pool, which includes multiple sample tasks. The differential amplification processing includes at least one of entity replacement, memory operation enhancement, and task simplification. The task execution agent is trained based on the sample tasks in the diversified task pool.

10. The method according to claim 9, characterized in that, The step of training the task execution agent based on each of the sample tasks in the diversified task pool includes: For each of the sample tasks, the sample task is executed based on the teacher model, and the task execution strategy information output at each time step is recorded; the task execution strategy information is parsed to obtain the step execution state information corresponding to each of the time steps; and the interaction trajectory information is obtained based on the step execution state information. Filter the interaction trajectory information that meets the preset quality requirements to obtain the target interaction trajectory information; Each step in the target interaction trajectory information is labeled to obtain a labeled step sequence; Based on the labeled sequence of steps, a student model is trained, and the trained student model is used as the task execution agent.

11. A task execution device, characterized in that, The device includes: The basic information acquisition module is configured to acquire the target task and the first context information corresponding to the first time step. The target task is an interface interaction task, which is a task that, based on the task execution target, determines the first interaction target in the interface screenshot of the first application, and determines the second interaction target after generating the interface screenshot of the second application through interaction with the first interaction target, and continues to interact with the second interaction target until the task execution target is achieved. Both the first application and the second application are applications with a visual interface containing the interaction target. The first context information is used to indicate the context state managed by the context management strategy output by the task execution agent during the execution of the target task to the first time step. The global state construction module is configured to construct the first global state corresponding to the first time step based on the target task and the first context information. The screenshot acquisition module is configured to acquire the first interface screenshot corresponding to the first time step; The task execution module is configured to input the first interface screenshot and the first global state into the task execution agent to perform task execution reasoning and obtain first task execution strategy information, which includes at least one of interface execution strategy and context management strategy; based on the first task execution strategy information, the target task is executed at a second time step until the execution result of the target task is obtained, where the second time step is the next time step after the first time step.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the task execution method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device performs the task execution method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the device to perform the task execution method as described in any one of claims 1 to 10.

Images