Task execution method, device and system

Abstract

The application discloses a task execution method, device and system, relates to the technical field of terminals, and can respond to user instructions in various scenes, including user instructions in complex scenes, in a timely, accurate and reliable manner, and improves the use experience of users. In the scheme, the system agent can combine the historical records of the historical tasks related to the execution of the task to perform fusion analysis and reasoning when determining the task corresponding to the user instruction, and finally determine the task corresponding to the user instruction. On the one hand, the user does not need to input overly complex instructions, and on the other hand, the task corresponding to the user instruction can be accurately identified, and the user instruction can be responded to in a timely, accurate and reliable manner.

Description

[0001] This application claims priority to Chinese patent application filed on June 20, 2024, with application number 202410808507.3 and entitled "Task Execution Method, Apparatus and System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to a task execution method, device and system. Background Technology

[0003] Currently, language model-based intelligent assistants are widely used in human-computer interaction (HCI) processes. These assistants can simplify user operations and accurately respond to user commands. For example, an intelligent assistant can analyze and reason about user commands based on a language model, create corresponding tasks, and execute those tasks to accurately respond to user commands. However, conventional intelligent assistants can only perform simple tasks. For complex scenarios, such as those involving multiple cross-application calls, complex reasoning processes, or situations requiring clarification, they cannot accurately respond to user commands. Summary of the Invention

[0004] This application provides a task execution method, device, and system that can respond to user commands in various scenarios in a timely, accurate, and reliable manner, including user commands in complex scenarios, thereby improving the user experience.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a task execution method is provided, which can be applied to a system including a system agent and at least one application control agent. The method includes: the system agent acquiring a user instruction; the system agent determining a task corresponding to the user instruction based on the user instruction and first memory information, the first memory information including a historical record of historical tasks related to the execution of the aforementioned task; and the system agent instructing the application control agent to execute the aforementioned task.

[0007] The solution provided in the first aspect above allows the system agent to perform fusion analysis and reasoning by combining the historical records of the tasks related to the execution of the aforementioned tasks when determining the task corresponding to the user command. This approach eliminates the need for users to input overly complex commands and accurately identifies the tasks corresponding to user commands, enabling timely, accurate, and reliable responses to user commands.

[0008] As one possible implementation, the above method further includes: during the execution of the task by the application control agent, the system agent interacts with the user based on the task clarification request from the application control agent to obtain the clarification result; the system agent instructs the application control agent to continue executing the task based on the clarification result. In this way, through one or more rounds of interaction, certain issues regarding the task can be clarified and / or certain details about the task can be supplemented, achieving clear, timely, accurate, and reliable task execution. While simplifying user operations, it intelligently responds to user commands in different scenarios, including both simple and complex scenarios.

[0009] As one possible implementation, the aforementioned system agent interacts with the user based on the task clarification request from the application control agent to obtain the clarification result. This includes: the system agent interacting with the user based on the task clarification request from the application control agent to obtain user input information; the system agent instructing the application control agent to execute the clarification task based on the user input information; and obtaining the clarification result based on the execution result of the clarification task. In this way, through one or more rounds of interaction, certain issues regarding the task can be clarified and / or certain details about the task can be supplemented, achieving clear, timely, accurate, and reliable task execution. While simplifying user operations, it intelligently responds to user commands in different scenarios, including both simple and complex scenarios.

[0010] As one possible implementation, the method further includes: the application control agent executing the task based on the task and the application control agent's second memory information, the second memory information including historical records of tasks related to the application control agent's execution of the task. In this way, task supplementation can be aided by historical records, achieving clear, timely, accurate, and reliable task execution.

[0011] As one possible implementation, the method further includes: the system agent breaking down the task into multiple subtasks, including a first subtask and a second subtask; the system agent determining a first application control agent and a second application control agent to execute the first subtask, and determining a second application control agent to execute the second subtask; the system agent instructing the application control agents to execute the task, including: the system agent distributing the first subtask to the first application control agent and distributing the second subtask to the second application control agent. In this way, user commands can be intelligently responded to in different scenarios, especially complex scenarios, while simplifying user operations.

[0012] As one possible implementation, the system agent instructs the application control agent to perform the aforementioned tasks, including: the system agent distributing the tasks to a third application control agent. This simplifies user operations while intelligently responding to user commands in different scenarios, including simple ones.

[0013] As one possible implementation, the system agent determines the task corresponding to the user instruction based on the user command and the first memory information, including: the system agent determines the task corresponding to the user instruction based on a language model. In this way, fusion analysis and reasoning can be performed based on the language model to accurately determine the task corresponding to the user instruction.

[0014] As an example, the language models mentioned above include at least one of the following: large-scale language model (LLM) and multimodal large-scale language model (MLLM).

[0015] As one possible implementation, the system is deployed on a server, and the system agent obtains user instructions, including: the system agent receiving user instructions from the electronic device; the method also includes: the application control agent invoking the execution engine, tools, or interfaces on the electronic device to execute the task. Thus, the task execution method provided in this application can be used in an edge-cloud collaborative architecture, improving the applicability of this solution.

[0016] As one possible implementation, the system is deployed on an electronic device, and the method further includes: an application control agent invoking an execution engine, tool, or interface on the electronic device to execute the task. This allows the task execution method provided in this application to be used in a single-device architecture, improving the applicability of the solution.

[0017] As one possible implementation, the aforementioned system agent is a system intelligent agent, and the application manipulation agent is a domain intelligent agent. This allows the task execution method provided in this application to be used within a multi-agent architecture, improving the applicability of the proposed solution.

[0018] In a second aspect, an electronic device is provided, comprising: a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device in implementing the methods of any possible implementation of the first aspect.

[0019] Thirdly, a server is provided, comprising: a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device in implementing the methods of any possible implementation of the first aspect.

[0020] Fourthly, a task execution system is provided, comprising: electronic devices and / or servers, the electronic devices and / or servers being used to support the task execution system in implementing the methods as described in any possible implementation of the first aspect.

[0021] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to implement the method as described in any possible implementation of the first aspect.

[0022] Sixthly, a chip system is provided, comprising processing circuitry and a storage medium storing computer program instructions; when executed by the processor, the computer program instructions implement the method as described in any possible implementation of the first aspect. The chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0023] Figure 1 A schematic diagram of a multi-layered proxy architecture provided in this application embodiment;

[0024] Figure 2 A schematic diagram of a multi-layered agent architecture provided in an embodiment of this application;

[0025] Figure 3 Two task execution architecture diagrams provided in the embodiments of this application;

[0026] Figure 4 These are schematic diagrams illustrating the two system architectures provided in the embodiments of this application.

[0027] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0028] Figure 6A An interactive diagram of a task execution process provided in an embodiment of this application;

[0029] Figure 6B Another task execution process interaction diagram provided in this application embodiment;

[0030] Figure 7 This is a schematic diagram of a task execution interaction process provided in an embodiment of this application;

[0031] Figure 8 This is a schematic diagram illustrating the task execution process in response to user instructions under a specific application scenario provided in this application embodiment. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0033] In the following text, the terms "first," "second," etc., are used only to distinguish different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. For example, if the described object is a "field," then the ordinal numbers before "field" in "first field" and "second field" do not limit the position or order of the "fields." "First" and "second" do not limit whether the "fields" they modify are in the same message, nor do they restrict the order of "first field" and "second field." Similarly, if the described object is a "level," then the ordinal numbers before "level" in "first level" and "second level" do not limit the priority of the "levels." Furthermore, the quantity of described objects is not limited by ordinal numbers and can be one or more; for example, in "first device," the number of "devices" can be one or more. In addition, objects modified by different prefixes can be the same or different. For example, if the described object is "device," then "first device" and "second device" can be devices of the same type or different types. Similarly, if the described object is "information," then "first information" and "second information" can be information with the same content or information with different content. In summary, the use of ordinal numbers and other prefixes used to distinguish the described objects in the embodiments of this application does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0034] Furthermore, in the embodiments of this application, "connection" can be a direct connection or an indirect connection; in addition, it can refer to an electrical connection or a communication connection; for example, the connection of two electrical components A and B can refer to A and B being directly connected, or it can refer to A and B being indirectly connected through other electrical components or connection media, or it can refer to A and B being indirectly connected through other communication devices or communication media, as long as it enables communication between A and B.

[0035] As described in the background section, conventional smart assistants can only perform some simple tasks and cannot accurately respond to user commands in complex scenarios.

[0036] As an example, when a smart assistant performs tasks based on user commands, it can use a language model to analyze and reason about the user commands (such as prompts), determine the tasks to be performed and the tools used to execute them, call the tools to perform the corresponding tasks, and then feed back the execution results (observation) to the language model for adjustments, thereby improving the reliability and accuracy of the language model's analysis and reasoning results. In this process, the smart assistant's reasoning mode is mainly reactive (ReAct), involving only one interaction, reasoning, and execution process, making the process relatively simple and thus limiting the types of tasks it can handle. Furthermore, the determined tasks rely heavily on the analysis and reasoning of user commands through the language model; therefore, the accurate and reliable execution of the tasks strongly depends on the accuracy and completeness of the user commands, thus requiring high precision in the user input.

[0037] To handle complex tasks in challenging scenarios, as an example, when executing tasks based on user commands, the intelligent assistant can follow an inner and outer loop mechanism. This mechanism uses a language model to analyze and reason about user commands level by level, determining the tasks to be executed and the tools used to perform them, and then invoking those tools to execute the corresponding tasks. Within this inner and outer loop mechanism, the outer loop is primarily used for high-level task management, such as identifying tasks based on user commands and dividing them into multiple operable subtasks. The inner loop is mainly used for executing low-level tasks, such as handling fine-grained subtasks. This inner and outer loop mechanism improves the reliability and accuracy of analysis and reasoning results, enhances the reliability and efficiency of task execution, and is easily integrated and compatible with external tools. However, the intelligent assistant's reasoning mode in this solution is primarily task execution (Action), still involving only one interaction, reasoning, and execution process. This process is relatively simple, and therefore the types of tasks it can handle are also relatively limited. Furthermore, in this scheme, the tasks to be executed mainly rely on the analysis and reasoning of user instructions through language models. Therefore, the accurate and reliable execution of the tasks also heavily depends on the accuracy and completeness of the user instructions, and thus the requirements for user input instructions are also high.

[0038] As an example, when a smart assistant performs tasks based on user commands, it can use a language model to analyze and reason about the user's instructions (such as prompts) to complete the tasks. It then employs a multimodal agent architecture to interact with the operating system, navigating across multiple applications to complete the task. This multimodal agent architecture includes an application agent (Appagent) and an execution agent (Actagent). The application agent breaks down the task into multiple sub-tasks and distributes them to the corresponding execution agents (Actagents). The execution agents then execute these sub-tasks. This approach is similar to the internal and external loop mechanisms described above. While it improves the reliability and accuracy of analysis and reasoning results, enhances the reliability and efficiency of task execution, and increases the possibility of integration and compatibility with external tools, it still involves only one interaction, reasoning, and execution process. The process is relatively simple, and therefore, the types of tasks it can handle are also relatively limited. Furthermore, since the tasks that need to be performed mainly rely on the analysis and reasoning of user instructions through language models, the accurate and reliable execution of the tasks also heavily depends on the accuracy and completeness of the user instructions. Therefore, the requirements for user input instructions are also relatively high.

[0039] As the examples above demonstrate, conventional smart assistants often involve relatively simple and fixed reasoning patterns, thus limiting their ability to perform only simple tasks. They struggle to accurately respond to user commands in complex scenarios involving multiple cross-application calls, intricate reasoning processes, or situations requiring clarification. Furthermore, conventional smart assistants heavily rely on the accuracy and completeness of user commands when determining the tasks to be performed, thus placing high demands on the accuracy of user input.

[0040] With the emergence of speech models with higher computational power and accuracy, such as large-scale language models (LLM) and multimodal large-scale language models (MLLM), the advantages of multi-layer agent architectures based on language models such as LLM and MLLM are becoming increasingly significant. For example, intelligent assistants built on multi-layer agent architectures can rely on language models such as LLM and MLLM as super entry points, and provide natural dialogue understanding, multi-turn dialogue fusion, multimodal analysis and reasoning, hierarchical management of historical records, multi-layer task management and isolation, and task switching through multi-layer agent division of labor and cooperation. This helps users handle various types of more complex tasks, such as task-oriented, casual conversation, question-and-answer, and AI-generated content (AIGC).

[0041] In the context of a multi-layered proxy architecture, this application provides a task execution method based on this architecture. This method can respond to user commands in different scenarios, including both simple and complex scenarios, in a timely, accurate, and reliable manner. For example, based on the task execution method provided in this application, a smart assistant can rely on a multi-layered proxy architecture including system agents and application control agents to perform related processing layer by layer. This includes determining the task through the system agent based on user commands and historical records of historical tasks, and splitting the task into multiple sub-tasks and distributing them to different application control agents. The application control agents then complete the corresponding sub-tasks. Furthermore, when the application control agents execute sub-tasks, they can supplement the sub-tasks based on historical records of related historical tasks. This enables more timely, accurate, and reliable task execution in response to user commands. Exemplarily, historical records may include, but are not limited to, session context.

[0042] Furthermore, when the system agent needs to clarify certain issues or supplement certain details about a task, the intelligent assistant can engage in one or more rounds of interaction with the user to clarify the relevant task. After clarification, it switches back to the system agent to update or supplement the user's instructions based on the clarification results and continue task identification, task planning, task distribution, etc.; and / or, when the application control agent needs to clarify certain issues or supplement certain details about a task, the intelligent assistant can also engage in one or more rounds of interaction with the user to clarify the relevant task. After clarification, it switches back to the application control agent to update or supplement the sub-tasks based on the clarification results and continue the execution of the sub-tasks, and so on, until the sub-tasks are successfully completed. Thus, through natural dialogue understanding, multi-turn dialogue fusion, multimodal analysis and reasoning, hierarchical management of historical records, multi-layered task management and isolation, and task switching, this multi-layered agent architecture can support user instruction responses in complex scenarios involving multiple cross-application calls, complex reasoning processes, and scenarios requiring clarification.

[0043] As an example, please refer to Figure 1 , Figure 1 A schematic diagram of a multi-layered proxy architecture provided in an embodiment of this application is shown. Figure 1 As shown, the multi-layered proxy architecture includes a system proxy 101 and multiple application manipulation proxies 102. Figure 1 (Taking three application manipulation agents 102 as an example).

[0044] in, Figure 1The system agent 101 shown is responsible for one or more of the following: user command analysis and reasoning, task identification, task planning, task distribution, and task clarification. User command analysis and reasoning may involve using language models such as LLM or MLLM to analyze and reason about user commands; task identification may involve analyzing and identifying the user's control intent through user commands, primary memory information, and system context to determine the corresponding task; task planning may involve planning the identified tasks, such as splitting tasks and determining the application control agent to execute the task; and task distribution may involve distributing the task to the corresponding application control agent 102 for subsequent execution. The primary memory information includes the historical record of tasks executed by the system agent 101, such as the session context of the historical tasks executed by the system agent 101, and is not limited to any specific instance.

[0045] As an example, when the system agent 101 determines the task of the user instruction, it can also rewrite the user instruction by combining a language model and a knowledge base, and determine the corresponding task based on the rewritten user instruction. For example, the system agent 101 can rewrite the user instruction through processes such as system context loading, first memory information retrieval or loading, named entity recognition, and hot word retrieval.

[0046] In some embodiments, during the task identification process of the system agent 101, if it is necessary to clarify certain issues about the task or supplement certain details about the task, the system agent 101 may also conduct one or more rounds of interaction with the user to clarify the task, and after the task clarification is completed, switch back to the system agent to update or supplement the user's instructions according to the clarification results and continue to perform task identification, task planning, task distribution, etc.

[0047] In some embodiments, if the user's control intent involves multiple tasks, the system agent 101 is further configured to split the identified task into multiple sub-tasks, determine the application control agent 102 for executing each sub-task, and distribute each sub-task to the corresponding application control agent 102 for subsequent execution. Figure 1 Taking the example of system agent 101 splitting the identified task into subtask 1, subtask 2, and subtask 3, and distributing them to different application control agents 102 respectively. System agent 101 may distribute subtask 1, subtask 2, and subtask 3 asynchronously to different application control agents 102, or it may distribute subtasks one by one and distribute the next subtask only after receiving a response indicating successful execution of a subtask; this is not limited and depends on the specific circumstances.

[0048] In some embodiments, when distributing each subtask to the corresponding application control agent 102, the system agent 101 may also instruct each application control agent 102 to execute each subtask in the order in which they execute each subtask. For example, each application control agent 102 may need to execute each subtask synchronously, or it may execute each subtask sequentially in the specified execution order.

[0049] In some embodiments, during the process of the application control agent 102 executing a task (such as a subtask) according to the instructions of the system agent 101, if it is necessary to clarify certain issues about the task or supplement certain details about the task, the system agent 101 may also conduct one or more rounds of interaction with the user to clarify the task according to the request of the application control agent 102, and after the task clarification is completed, instruct the relevant application control agent 102 to update or supplement the subtask according to the clarification result and continue to execute the subtask, and so on, until the execution of the subtask is successfully completed.

[0050] In some embodiments, during one or more rounds of interaction between system agent 101 and user for task clarification, system agent 101 may also interact with user based on task clarification requests from application control agent 102, obtain user input information, and analyze the user input information to obtain clarification results. For example, system agent 101 can analyze information input by user in one or more rounds of interaction to clarify questions and obtain clarification results.

[0051] In some embodiments, during one or more rounds of interaction between the system agent 101 and the user to clarify a task, the system agent 101 may also instruct the application control agent 102 to execute the clarification task based on the information input by the user, and then obtain the clarification result based on the execution result of the clarification task. For example, the system agent 101 may obtain the clarification result based on the user's clarification instructions in one or more rounds of interaction, through processes such as clarification task planning and clarification task distribution corresponding to the clarification instructions.

[0052] in, Figure 1 The multiple application control agents 102 shown are used to execute specific sub-tasks according to the instructions of the system agent 101.

[0053] In this application embodiment, the application control agent may include, but is not limited to, control agents for any of the following applications: calls, instant messaging, video, audio, shopping, news, information, finance, wealth management, weather, search, navigation, email, clock, memo, calendar, map, office, artificial intelligence, etc., without limitation.

[0054] In some embodiments, during the execution of a specific subtask by the application control agent 102, if the user needs to clarify certain questions about the task or supplement certain details about the task, the application control agent 102 may also request the system agent 101 to interact with the user to clarify the task. After the task clarification is completed, the subtask is updated or supplemented according to the clarification result and the execution of the subtask continues. This process continues until the execution of the subtask is successfully completed.

[0055] In other words, based on the multi-layered agent architecture provided in this application embodiment, the system agent 101 can interact with the user in one or more rounds depending on the execution of sub-tasks by the application control agent 102, until the sub-tasks of each application control agent 102 can be clearly, accurately, and reliably executed. Therefore, this architecture can not only accurately determine the user's operational intent and corresponding task based on user instructions and first memory information, but also flexibly perform task decomposition and distribution of the resulting sub-tasks to the application control agent 102 to support the reliable execution of each sub-task. Furthermore, it can flexibly interact with the user in one or more rounds depending on the specific execution of the sub-task to clarify certain questions about the task and / or supplement certain details about the task, achieving clear, timely, accurate, and reliable execution of each sub-task. Ultimately, it ensures the timely, accurate, and reliable execution of tasks, intelligently responding to user instructions in different scenarios, including simple and complex scenarios, while simplifying user operations. Especially in complex scenarios involving multiple cross-application calls, complex reasoning processes, and scenarios requiring clarification, it can complete related tasks in a timely, accurate, and reliable manner, improving the user experience.

[0056] In some embodiments, Figure 1 The multi-layered proxy architecture shown also includes a memory module 103 for maintaining memory information (such as first memory information, second memory information, etc.) and system context information. Figure 1 The architecture shown is merely an example. In actual applications, system agent 101 may include a module for maintaining first memory information, and each application control agent 102 may include a module for maintaining its own second memory information, etc., without limitation. The second memory information of application control agent 102 includes the historical record of tasks executed by application control agent 102, such as the session context of tasks executed by application control agent 102, etc., without limitation.

[0057] For example, the multi-layered agent architecture described in this application embodiment is a multi-agent architecture based on language models such as LLM and MLLM, wherein the agent uses language models such as LLM and MLLM as its core computing engine, including a system agent and a domain agent. The system agent is used for user instruction analysis, task identification, task planning, task distribution, and task clarification; the domain agent is used to execute specific sub-tasks.

[0058] It should be noted that in a multi-agent architecture, there may be multiple domain agents responsible for executing different types of subtasks.

[0059] As an example, please refer to Figure 2 , Figure 2 This diagram illustrates a multi-layered agent architecture provided in an embodiment of this application. Figure 2 As shown, a multi-layered agent architecture can include a system agent (also known as a system agent) 201 and multiple domain agents (also known as domain agents) 202. Figure 2 (Taking a system agent 202 as an example) and a memory module 203. For an introduction to the system agent 201, please refer to the description of the system agent 101 above; for an introduction to the domain agent 202, please refer to the description of the application manipulation agent 102 above; and for an introduction to the memory module 203, please refer to the description of the memory module 103 above. These will not be repeated here.

[0060] In some embodiments, the task execution method provided in this application can be applied to the architecture of edge-cloud interaction.

[0061] For example, this method can be applied to Figure 3 The architecture shown in (a) includes electronic device 310 and server 320 (such as a cloud server). Figure 3 The electronic device 310 shown in (a) is responsible for interacting with the user, recognizing user commands based on user actions, and sending user commands to the server 320. Figure 3 The server 320 shown in (a) is responsible for performing fusion analysis and reasoning based on user instructions and first memory information to identify tasks, and for subsequent task planning, task execution, and task clarification. The first memory information includes historical records of tasks related to the execution of user instructions.

[0062] Optionally, in Figure 3After the server 320 and electronic device 310 shown in (a) complete the execution of the task, the electronic device 310 can also output the response results to the user's instructions, including but not limited to interface display, result reminders, etc.

[0063] Optionally, Figure 3 The server 320 shown in (a) is also used to instruct the electronic device 310 to engage in one or more rounds of interaction with the user to clarify certain issues regarding the task or to supplement certain details about the task when clarification is needed. After the task clarification is completed, the server 320 updates or supplements the user's instructions based on the clarification results and continues with task identification, task planning, task execution, etc., or updates or supplements sub-tasks based on the clarification results and continues with the execution of sub-tasks. Correspondingly, Figure 3 The electronic device 310 shown in (a) is also used to interact with the user again to clarify the task according to the instructions of the server 320, and send the clarification result to the server 320, and so on, until the execution of the subtask is successfully completed.

[0064] In some embodiments, the task execution method provided in this application can be applied to a single-device architecture.

[0065] For example, this method can be applied to Figure 3 In the electronic device 310 shown in (b), the electronic device 310 is specifically used to interact with the user and perform user instruction recognition, task recognition, task planning, task execution and task clarification based on user operations.

[0066] Optionally, in Figure 3 After the electronic device 310 shown in (b) completes the task, it can also output the response results to the user's instructions, including but not limited to interface display and result reminders.

[0067] Optionally, Figure 3 The electronic device 310 shown in (b) is also used to conduct one or more rounds of interaction with the user to clarify the task when it is necessary to clarify certain issues about the task or supplement certain details about the task. After the task clarification is completed, the user instructions are updated or supplemented according to the clarification results and the task identification, task planning, task execution, etc. are continued. Alternatively, after the task clarification is completed, the sub-tasks are updated or supplemented according to the clarification results and the sub-tasks are executed. This process continues until the sub-tasks are successfully executed.

[0068] The specific architecture of the task execution method provided in the embodiments of this application is not specifically limited, and can be determined according to the specific device type, device function, device computing power and application scenario.

[0069] As an example, please refer to Figure 4, Figure 4 respectively Figure 3 (a) and Figure 3 Taking the architecture shown in (b) as an example, a schematic diagram of the structure of two system architectures provided in the embodiments of this application is shown.

[0070] in, Figure 4 Image (a) shows a schematic diagram of an edge-cloud collaborative architecture. Figure 4 As shown in (a), the electronic device 310 may include a human-computer interaction module 310-1 and a communication module 310-2; the server 320 may include a communication module 320-1, a system agent module 320-2, and multiple application control agent modules 320-3. Figure 4 (Taking the three application manipulation agent modules 320-3 as an example).

[0071] Figure 4 The human-computer interaction module 310-1 of the electronic device 310 shown in (a) is used to interact with the user, such as receiving user input, receiving user touch operation, receiving user voice instructions and other user operations, and recognizing user instructions based on user operations. The specific way the user interacts with the electronic device 310 is not limited in the embodiments of this application.

[0072] Figure 4 The communication module 310-2 of the electronic device 310 shown in (a) is used to send the recognized user instructions to the server 320. In some embodiments, the communication module 310-2 of the electronic device 310 is also used to interact with the server 320 in one or more rounds to clarify the task and send the clarification results to the server 320, clarifying certain questions about the task and / or supplementing certain details about the task.

[0073] In some embodiments, the communication module 310-2 of the electronic device 310 is also used to send the system context obtained by the smart assistant to the system agent. The system context of the smart assistant may include, but is not limited to, information of the electronic device (such as device identifier, device name, login username, etc.), running application A (such as foreground application, background application, etc.), location information (such as location coordinates collected by the global positioning system (GPS), etc.). The system context sent by the smart assistant is used by the system agent for reference when determining the task corresponding to the user instruction.

[0074] Figure 4The communication module 320-1 of the server 320 shown in (a) is used to receive user instructions from the electronic device 310. In some embodiments, the communication module 320-1 of the server 320 is also used to interact with the electronic device 310 in one or more rounds to clarify tasks, clarify certain questions about the tasks and / or supplement certain details about the tasks.

[0075] Figure 4 The system agent module 320-2 of the server 320 shown in (a) is responsible for analyzing user commands to perform task identification, task planning, and task distribution. In some embodiments, when the system agent module 320-2 or the application control agent module 320-3 of the server 320 needs to clarify certain issues about the task or supplement certain details about the task, the system agent module 320-2 of the server 320 is also used to trigger one or more rounds of interaction with the user through the electronic device 310 to clarify the task.

[0076] Figure 4 In (a) shown, the multiple application control agent modules 320-3 of server 320 are used to execute their respective subtasks according to the instructions of system agent module 320-2 of server 320. In some embodiments, when it is necessary to clarify certain issues about the task or supplement certain details about the task, the multiple application control agent modules 320-3 of server 320 are also used to request system agent module 320-2 of server 320 to clarify the task.

[0077] Figure 4 (b) shows a schematic diagram of a single-device architecture. Figure 4 As shown in (b), the electronic device 310 may include a human-computer interaction module 310-1, a system agent module 310-3, and an application control agent module 310-4.

[0078] in, Figure 4 The human-computer interaction module 310-1 of the electronic device 310 shown in (b) is used to interact with the user, such as receiving user input, receiving user touch operation, receiving user voice instructions and other user operations, and is used to identify user commands based on user operations. The specific way the user interacts with the electronic device 310 is not limited in the embodiments of this application.

[0079] Figure 4 The system agent module 310-3 of the electronic device 310 shown in (b) is responsible for analyzing user commands to perform task identification, task planning and task distribution.

[0080] Figure 4The multiple application control agent modules 310-4 of the electronic device 310 shown in (b) are used to execute their respective sub-tasks according to the instructions of the system agent module 310-3 of the electronic device 310.

[0081] In some embodiments, when clarification of certain issues regarding the task or supplementation of certain details about the task is required, the application control agent module 310-4 of the electronic device 310 is further configured to request the system agent module 310-3 of the electronic device 310 to perform task clarification, in order to clarify certain issues regarding the task and / or supplement certain details about the task. Correspondingly, the system agent module 310-3 of the electronic device 310 is further configured to trigger one or more rounds of interaction with the user for task clarification.

[0082] It needs to be explained that, Figure 4 (a) and Figure 4 (b) is merely an example of two system architectures for executing tasks according to user instructions. This application does not limit the specific architecture or the specific functional module settings in electronic devices or servers. Figure 4 (a) or Figure 4 In (b), multiple modules may be integrated together, or the function of one module may be accomplished by multiple sub-modules, depending on the specific circumstances.

[0083] The electronic devices described in this application's embodiments may include, but are not limited to, any electronic device capable of displaying images. These include, but are not limited to, smartphones, netbooks, tablets, smart drawing tablets, handwriting tablets, smartwatches, smart bracelets, phone watches, smart glasses, smart cameras, handheld computers, in-vehicle computers, personal computers (PCs), personal digital assistants (PDAs), portable multimedia players (PMPs), augmented reality (AR) / virtual reality (VR) devices, smart TVs, projection devices, or motion-sensing game consoles in human-computer interaction scenarios. Alternatively, the electronic device may also be other types or structures of electronic devices capable of displaying images; this application is not limited to these categories.

[0084] As an example, please refer to Figure 5 , Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown.

[0085] like Figure 5As shown, the electronic device may include a processor 510, a memory (including an external memory interface 520 and an internal memory 521), a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna 1, an antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, a headphone jack 570D, a sensor module 580, buttons 590, a motor 591, an indicator light 592, a camera 593, a display screen 594, etc.

[0086] The sensor module 580 may include, but is not limited to, one or more of the following: touch sensor, pressure sensor, temperature sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, distance sensor, proximity sensor, fingerprint sensor, ambient light sensor, bone conduction sensor, etc.

[0087] Processor 510 may include one or more processing units. For example, processor 510 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a flight controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0088] The processor 510 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. This memory can store instructions or data that the processor 510 has just used or that are used repeatedly. If the processor 510 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.

[0089] In some embodiments of this application, the processor 510 can send the application interface, memory images, etc. to the display screen 594 for display via an interface such as a DSI interface.

[0090] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, modem processor and baseband processor, etc.

[0091] In some embodiments of this application, the electronic device can communicate with other devices through antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, modem processor and baseband processor, etc., such as sending user instructions, information for clarifying problems, clarification instructions, clarification results, etc. to the server; and receiving execution engine, tool or interface call instructions from the server.

[0092] Electronic devices implement display functions through GPUs, displays 594, and APs. A GPU is a microprocessor for image processing, connected to both the display 594 and the AP. The GPU performs mathematical and geometric calculations for drawing, rendering, or compositing graphics. Processor 510 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0093] In some embodiments of this application, the electronic device can implement application interface and / or memory image processing through AP, GPU, etc., and display the application interface and / or memory image through display screen 594.

[0094] Electronic devices can achieve shooting functions through ISP, camera 593, video codec, GPU, display 594 and AP, etc.

[0095] The external memory interface 520 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 510 through the external memory interface 520 to perform data storage.

[0096] Internal memory 521 can be used to store computer executable program code. Exemplarily, the computer program may include an operating system program and application programs. The executable program code includes instructions. Processor 510 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 521. Internal memory 521 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. Furthermore, internal memory 521 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 510 executes various functional applications and data processing of the electronic device by running instructions stored in internal memory 521 and / or instructions stored in memory disposed within the processor.

[0097] Electronic devices can implement audio functions through audio modules 570, speakers 570A, receivers 570B, microphones 570C, and access points (APs), such as music playback and recording.

[0098] about Figure 5 The descriptions of the charging management module 540, power management module 541, battery 542, motor 591, indicator light 592, etc. shown can be found in conventional technology and will not be elaborated here.

[0099] It is understood that this application Figure 5 The illustrated structure does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0100] The task execution method provided in the embodiments of this application will be described in detail below.

[0101] The multi-layered proxy architecture provided in this application can respond to user commands in various scenarios in a timely, accurate, and reliable manner, including user commands in various complex scenarios. Taking a proxy architecture that includes system proxies and application manipulation proxies as an example, the multi-layered proxy architecture can execute the tasks in the following scenarios 1-3:

[0102] Scenario 1: Single-task scenario

[0103] In single-task scenarios, such as when the system agent determines the task corresponding to a user instruction based on the user command and the historical records of related tasks (i.e., first memory information), no task splitting is required; a single task execution is sufficient to respond to the user command. In this case, after obtaining the user command corresponding to the user operation and determining the corresponding task based on the user command and first memory information, the system agent can determine the application control agent to execute the task and instruct that application control agent to perform the corresponding task in response to the user command.

[0104] For example, a single-task scenario such as a task execution (Action) instruction scenario.

[0105] For example, taking the user instruction "set an alarm for 8 o'clock" as an example, the system agent determines the task corresponding to the user instruction as "set an alarm for 8 o'clock tomorrow in the clock application" based on the user instruction and the first memory information. For example, the system agent can instruct the clock control agent to execute the task "set an alarm for 8 o'clock tomorrow in the clock application" through the following instruction 1:

[0106] Command 1: CreateAlarm(time = "Tomorrow at 8 o'clock"), LastAction: True.

[0107] In instruction 1, "CreateAlarm(time = "tomorrow at 8 o'clock")" is used to instruct setting an alarm for 8 o'clock tomorrow, and "LastAction: True" indicates that the response to the user's instruction will end once the task is successfully executed.

[0108] Alternatively, for example, if the user instruction is "Buy one serving of celery dumplings and deliver to address C", the system agent determines the corresponding task as "Buy one serving of celery dumplings from store B in application A and deliver to address C" based on the user instruction and initial memory information (such as the user's frequent order records from store B in application A). Executing this task will then terminate the response to the user instruction.

[0109] Scenario 2: Multitasking Scenario

[0110] As an example, multi-task scenarios include multi-task scenarios with one-time planning and multi-task scenarios with multiple planning steps.

[0111] In a multi-task scenario planned in one step, after determining the task corresponding to a user instruction based on the user's command and initial memory information, the system agent breaks down the task into multiple subtasks and distributes them to the corresponding application control agents. In this case, the system agent can obtain the user command corresponding to the user's operation, determine the task based on the user command and initial memory information, break down the task into multiple subtasks, and then distribute the subtasks to the designated application control agents for execution in response to the user command. This includes scenarios where the application control agents can automatically switch tasks based on instructions from the system agent while executing their respective subtasks.

[0112] For example, a multi-task scenario planned at one time, such as a ReAct instruction scenario.

[0113] For example, taking the user instruction as "find Zhang San's phone number and then send it to Li Si via instant messaging application X", the system agent determines the task corresponding to the user instruction as "find Zhang San's phone number in the address book application and then send it to the contact Li Si via instant messaging application X" based on the user instruction and the first memory information. Then, the system agent splits the task into a first subtask and a second subtask. The first subtask is "find Zhang San's phone number in the address book application", and the second subtask is "send Zhang San's phone number to the contact Li Si via instant messaging application X".

[0114] For example, the process of a system agent performing task planning might be as follows:

[0115] plugins [Find Zhang San's phone number in the address book application, and send Zhang San's phone number to contact Li Si via instant messaging application X].

[0116] For example, the system agent can instruct the address book control agent to perform the first subtask "find Zhang San's phone number in the address book application" using the following instruction 2, and instruct the address book control agent to perform the second subtask "send Zhang San's phone number to contact Li Si via instant messaging application X" using the following instruction 3:

[0117] Command 2: CheckPhone(contract="Zhang San"), LastAction: False;

[0118] Command 3: SendMail(App = "Instant Messaging Application X", Receiver = "Li Si"), LastAction: True.

[0119] In instruction 2, "CheckPhone(contract="Zhang San")" is used to instruct the user to search for Zhang San's phone number in the address book application, "SendMail(App="Instant Messaging Application X", Receiver="Li Si")" is used to instruct the user to send Zhang San's phone number to the contact Li Si through the instant messaging application X, "LastAction: False" indicates that other tasks need to be executed after this task is successfully executed, and "LastAction: True" indicates that the response to the user's instruction will end after this task is successfully executed.

[0120] Alternatively, for example, if the user instruction is "buy another serving of dumplings", the system agent determines the task corresponding to the user instruction as "buy a serving of celery dumplings from store B in application A and deliver it to address C" based on the user instruction and the first memory information. Then, the system agent splits the task into a first subtask, a second subtask, and a third subtask. The first subtask is "open application A", the second subtask is "search for store B in application A", and the third subtask is "order a serving of celery dumplings from store B and deliver it to address C".

[0121] It should be noted that the scenarios and task examples mentioned above are for illustrative purposes only. In actual applications, the user instructions, task planning, and specific execution details of each subtask are not limited and can be determined according to the specific circumstances.

[0122] In multi-task scenarios where multiple plans are made, if the system agent finds that the execution of the task cannot successfully respond to the user instruction or cannot accurately respond to the user instruction after determining the task corresponding to the user instruction based on the user instruction and the first memory information, the system agent will re-plan the task based on the above task and respond to the user instruction by executing the first sub-task and the second sub-task.

[0123] For example, taking the user instruction as "send Zhang San's phone number to Li Si through communication application X", the system agent determines the task corresponding to the user instruction as "send Zhang San's phone number to Li Si through communication application X" based on the user instruction and the first memory information. However, since the system agent does not store Zhang San's phone number, the system agent will further perform task planning based on the task through analysis and reasoning to obtain the first subtask and the second subtask. The first subtask is "get Zhang San's phone number", and the second subtask is "send Zhang San's phone number to Li Si through communication application X".

[0124] For example, the process of a system agent performing task planning might be as follows:

[0125] plugins[send Zhang San's phone number to Li Si via communication application X]->plan(subtask1: get Zhang San's phone number; subtask2: send the obtained phone number to Li Si via communication application X).

[0126] In the example above, "subtask1: Get Zhang San's phone number" is the first subtask mentioned above, and "subtask2: Send the obtained phone number to Li Si through the communication application X" is the second subtask mentioned above.

[0127] For example, the system agent can instruct the address book control agent to perform the first subtask "get Zhang San's phone number" using the following instruction 4, and instruct the address book control agent to perform the second subtask "send Zhang San's phone number to contact Li Si via instant messaging application X" using the following instruction 5:

[0128] Command 4: CheckPhone(contract="Zhang San"), LastAction: False;

[0129] Command 5: SendMail(App = "Instant Messaging Application X", Receiver = "Li Si"), LastAction: True.

[0130] In instruction 4, "CheckPhone(contract="Zhang San")" is used to retrieve Zhang San's phone number, and "LastAction: False" indicates that other tasks need to be executed after this task is successfully completed. In instruction 5, "SendMail(App="Instant Messaging Application X", Receiver="Li Si")" indicates that Zhang San's phone number will be sent to the contact Li Si through instant messaging application X, and "LastAction: True" indicates that the response to the user's instruction will end after this task is successfully completed.

[0131] Alternatively, for example, if the user instruction is "buy another serving of dumplings", the system agent determines the corresponding task as "buy a serving of dumplings from store B in application A" based on the user instruction and the first memory information. However, since the system agent does not store what kind of dumplings the user likes, nor does it store the user's delivery address, the system agent will further plan the task based on the task through analysis and reasoning to obtain a first subtask and a second subtask. The first subtask is "get the user's order information from store B in application A", and the second subtask is "buy a serving of dumplings from store B in application A based on the obtained order information".

[0132] Scenario 3: Task clarification scenario.

[0133] As an example, a task clarification scenario might be where a system agent, when determining a task corresponding to a user instruction, needs to clarify certain issues or supplement certain details about the task. In this scenario, the system agent can trigger task clarification to clarify certain issues or supplement certain details about the task through one or more rounds of interaction with the user.

[0134] For example, if the user instruction is "I want to buy a ticket to Beijing", when the system agent is determining the task corresponding to the user instruction, since it is not certain whether the user wants to buy a plane ticket, a train ticket or other type of ticket, the system agent can trigger task clarification to determine which type of ticket the user wants to buy through one or more rounds of interaction with the user.

[0135] Alternatively, for example, if the user instruction is "I want to buy a plane ticket to Beijing", when the system agent is determining the task corresponding to the user instruction, since it is not certain which day the user wants to buy the ticket, the system agent can trigger task clarification to determine which day the user wants to buy the ticket through one or more rounds of interaction with the user.

[0136] As an example, a task clarification scenario might be where an application control agent needs to clarify certain issues or supplement certain details about a task during its execution. In this scenario, the application control agent can request the system agent to trigger task clarification, thereby clarifying certain issues or supplementing certain details about the task through one or more rounds of interaction with the user. For example, the system agent could interact with the user through a smart assistant to clarify certain issues or supplement certain details about the task.

[0137] For example, if the user instruction is "buy another serving of dumplings", and the system agent determines the corresponding task as "buy a serving of dumplings from store B in application A" based on the user instruction and the first memory information, and the resulting first, second, and third subtasks are "open application A", "search for store B in application A", and "order a serving of dumplings from store B and deliver it to address C", respectively, after application A controls the agent to open application A and search for and enter store B in application A, during the execution of the third subtask "order a serving of dumplings from store B and deliver it to address C", since application A controls the agent not knowing what kind of dumplings the user wants to buy, application A controls the agent to request the system agent to trigger task clarification, so as to determine what kind of dumplings the user wants to buy through one or more rounds of interaction with the user.

[0138] Alternatively, for example, if the user instruction is "set an alarm for 8 o'clock", and the system agent determines the task corresponding to the user instruction as "set an alarm for 8 o'clock in the clock application" based on the user instruction and the first memory information, when the application control agent of the clock application opens the clock application to set the alarm, since the application control agent of the clock application is not sure whether the user wants to set an alarm for 8 am or 8 pm, the application control agent of the clock application requests the system agent to trigger task clarification, so as to determine whether the user wants to set an alarm for 8 am or 8 pm through one or more rounds of interaction with the user.

[0139] As one possible implementation, during one or more rounds of interaction with the user, the system agent can use an intelligent assistant to analyze the information entered by the user in one or more rounds of interaction to clarify the issue and obtain the clarification result.

[0140] For example, during the execution of the third subtask "order a serving of dumplings from store B and deliver it to address C" by the application A controlling agent, since the application A controlling agent is unsure what kind of dumplings the user wants to buy, it requests the system agent to trigger task clarification. This clarification is done through one or more rounds of interaction with the user to determine the desired filling. For instance, the system agent can ask the user through a smart assistant, "What kind of dumplings would you like?" In response to the user's answer, "celery filling," the system agent can determine that the user wants to buy celery-filled dumplings and send this clarification result to the application A controlling agent. After receiving this clarification result, the application A controlling agent updates or supplements the third subtask to "order a serving of celery-filled dumplings from store B and deliver it to address C" and continues executing the third subtask.

[0141] As one possible implementation, during one or more rounds of interaction with the user, the system agent can obtain the clarification result through the intelligent assistant based on the user's clarification instructions in one or more rounds of interaction, through processes such as clarification task planning and clarification task distribution corresponding to the clarification instructions.

[0142] For example, during the execution of the third subtask "order a serving of dumplings from store B and deliver it to address C" by the application A control agent, since the application A control agent is unsure what kind of dumplings the user wants to buy, it requests the system agent to trigger task clarification. This clarification is done through one or more rounds of interaction with the user to determine the desired dumpling filling. For instance, the system agent can ask the user through a smart assistant, "What kind of dumplings do you want?" If the user instructs, "Call Dad and ask," the system agent can determine the clarification task as "Call Dad and ask what kind of dumplings he wants," and break this task down into a first clarification subtask, "Get Dad's phone number," and a second clarification subtask, "Call Dad and ask what kind of dumplings he wants," and distribute these subtasks to the corresponding application control agents. If the system agent ultimately determines that the user wants to buy celery-filled dumplings by executing the first and second clarification subtasks, it can confirm this and send the clarification result to the application A control agent. Upon receiving the clarification result, application A controls the agent to update or supplement the third subtask to "order a serving of celery dumplings from store B and deliver it to address C" and continues to execute the third subtask.

[0143] In some embodiments, scenario 3 may overlap with scenario 1 or scenario 2. For example, in a single-task scenario (i.e., scenario 1), a task clarification scenario (i.e., scenario 3) may also be involved; in a multi-task scenario (i.e., scenario 2), a task clarification scenario (i.e., scenario 3) may also be involved.

[0144] As an example, please refer to Figure 6A , Figure 6A Taking a single-task scenario as an example, an interaction diagram of a task execution process provided in an embodiment of this application is shown. Figure 6A As shown, the intelligent assistant can rely on a multi-layered agent architecture including a system agent and at least one application control agent to perform tasks in response to user commands through S601A-S605A. The relevant functions of the intelligent assistant may be performed by electronic devices, servers, or a combination of electronic devices and servers, without specific limitations.

[0145] S601A: The system agent obtains the user instructions corresponding to the user operation.

[0146] As an example, user action refers to actions taken by a user towards a smart assistant on an electronic device. For instance, a smart assistant may include, but is not limited to, a voice assistant.

[0147] User operations may include, but are not limited to, at least one of the following: voice commands, gestures, body language, touch input, and information input (such as text input). For example, taking a voice assistant as an example, user operations may include voice commands from the user within the voice assistant. This application does not limit the specific form of user operations.

[0148] As an example, smart assistants are deployed on electronic devices.

[0149] As one possible implementation, after an electronic device receives a user operation, such as an operation performed by the user on a smart assistant within the device, the electronic device can first determine the input event corresponding to the user operation, and then perform analysis and reasoning based on the input event to identify the corresponding user instruction. For example, after receiving a user operation, the electronic device can first convert the user operation into a specific input event, and then distribute the input event to its corresponding smart assistant window. The smart assistant can then perform analysis and reasoning based on the input event to identify the corresponding user instruction.

[0150] As an example, if a user inputs the voice command "order another serving of dumplings" into a smart assistant, the electronic device can first convert the voice command into a specific input event, and then distribute the input event to its corresponding smart assistant window. The voice assistant can then analyze and reason based on the input event to identify the corresponding user command as "buy another serving of dumplings".

[0151] As one possible implementation, after the user's instructions are determined, the smart assistant can call the system agent to execute the subsequent S602A.

[0152] As an example, if a system agent is deployed on an electronic device, the smart assistant can call the relevant interface to send the specified user instructions to the system agent.

[0153] As an example, if the system agent is deployed on a server, the smart assistant can send specific user commands to the system agent through a communication connection with the server.

[0154] In some embodiments, in addition to sending user commands to the system agent, the smart assistant can also send the system context obtained by the smart assistant to the system agent. The system context of the smart assistant may include, but is not limited to, information about the electronic device (such as device identifier, device name, login username, etc.), the running application A (such as foreground application, background application, etc.), and location information (such as location coordinates collected by GPS). The system context sent by the smart assistant is used by the system agent for reference when determining the task corresponding to the user command.

[0155] In some embodiments, the system agent can also rewrite user commands by combining language models and knowledge bases, and determine the corresponding task based on the rewritten user commands. For example, the system agent can rewrite user commands through processes such as system context loading, first memory information retrieval or loading, named entity recognition, and hot word retrieval.

[0156] S602A: The system agent determines the task corresponding to the user instruction based on the user instruction and the first memory information.

[0157] As one possible implementation, the system agent can use user commands and primary memory information as input to a language model. The language model then performs fusion analysis and reasoning on the user commands and primary memory information to determine the task corresponding to the user command. For example, the language model may include, but is not limited to, at least one of the following: LLM, MLLM, etc.

[0158] As an example, the first memory information includes a historical record of tasks related to the system agent's execution of the task. For instance, the first memory information includes a historical record of historical tasks during the historical interactions between the system agent and the application control agent, specifically the historical tasks related to the user instructions mentioned above during these interactions. Exemplarily, the historical record may include, but is not limited to, at least one of the following: the application control agent's response to the historical task, the system agent's judgments and rewrites during the execution of the historical task, and the actions and results performed by the system agent during the execution of the historical task.

[0159] As an example, system agents and application manipulation agents can interact in accordance with prescribed communication specifications.

[0160] For example, this communication specification defines the session context for each historical dialogue shown in Table 1 below when the system agent and application manipulation agent interact:

[0161] Table 1

[0162]

[0163]

[0164] In some embodiments, as shown in Table 2 below, the communication specifications followed between the system agent and the application manipulation agent, in addition to specifying the session context specifications for each historical dialogue shown in Table 1, also specify the specifications for the system agent and the application manipulation agent to generate their respective memory information:

[0165] Table 2

[0166]

[0167] The multi-layered proxy architecture provided in this application allows each layer to interact by following a prescribed communication standard. This not only reduces the analysis and reasoning costs at each layer but also ensures the execution efficiency, reliability, and accuracy of tasks when the multi-layered proxy architecture is applied to various complex scenarios.

[0168] As an example, taking the user instruction "Buy another serving of dumplings" as an example, and the initial memory information including the session context of the system agent and application control agent interacting to execute the historical task "Buy one serving of celery dumplings from store B in application A and deliver it to address C," the system agent, based on the fusion analysis and reasoning of the user instruction and the initial memory information, determines that the task corresponding to the user instruction is "Buy one serving of celery dumplings from store B in application A and deliver it to address C." It can be seen that in this example, the user instruction "Buy another serving of dumplings" does not specify which application, which store, what type of dumplings, or the delivery address. Through the fusion analysis and reasoning of the user instruction and the initial memory information, the system agent can intelligently determine the target application, target store, target dish, and delivery address, without requiring the user to input overly complex instructions, and can promptly, accurately, and reliably identify the task corresponding to the user instruction.

[0169] In some embodiments, if the system agent receives a system context, it can also refer to that system context when determining the task corresponding to the user instruction. For example, the system agent can use the prompt information determined based on the user instruction, first memory information, and system context as input to a language model, and then use the language model to perform fusion analysis and reasoning on the first prompt information to determine the task corresponding to the user instruction.

[0170] S603A: The system agent determines the application control agent used to execute the task corresponding to the user instruction.

[0171] As one possible implementation, the system agent can determine which application control agent will execute the task based on the application involved in the task corresponding to the user instruction.

[0172] For example, if the user command corresponds to the task of "setting an alarm for 8 o'clock", the system agent determines that the application control agent of the clock application will execute the task based on the application A involved in the task.

[0173] S604A: The system agent instructs the application control agent to perform the above tasks.

[0174] As an example, the system agent can instruct the application manipulation agent to perform the tasks described above through an interface with the application manipulation agent.

[0175] S605A: The application control agent executes tasks according to the instructions of the system agent.

[0176] As a possible example, upon receiving instructions from a system agent, the application control agent makes calls to relevant execution engines, tools, or interfaces to perform a corresponding task. For instance, if the task is an operation within the application, the application control agent can use UI-based understanding to call execution engines, tools, or interfaces to perform relevant control over the application.

[0177] For example, if the user command corresponds to the task of "setting an alarm for 8 o'clock tomorrow", after receiving the instruction from the system agent, the application control agent of the clock application can, based on the UI understanding, call the execution engine (such as a smart assistant), tools or interfaces to start the clock and set the alarm for 8 o'clock tomorrow in the clock application.

[0178] In some embodiments, the application control agent can execute the aforementioned task based on the task corresponding to the user instruction and the application control agent's second memory information. The second memory information includes a historical record of tasks executed by the application control agent, such as a historical record of tasks related to the task corresponding to the user instruction executed by the application control agent. For example, during the process of the application control agent executing a task according to the instructions of the system agent, the application control agent can also supplement the task based on the second memory information and execute the task according to the supplemented task.

[0179] In some embodiments, while the application control agent is executing tasks according to the instructions of the system agent, the application control agent can also supplement the tasks based on the system context. Based on this, the application control agent executes the tasks according to the supplemented subtasks.

[0180] As an example, if an application control agent is deployed on an electronic device, the application control agent can call the relevant execution engine, tools, or interfaces of the electronic device to perform corresponding tasks.

[0181] As an example, if the application control agent is deployed on a server, the application control agent can invoke the communication connection between the server and the electronic device to send the invocation instructions of the execution engine, tools, or interfaces to the electronic device; correspondingly, the electronic device invokes the relevant execution engine, tools, or interfaces to perform the corresponding tasks according to the invocation instructions of the application control agent.

[0182] As an example, please refer to Figure 6B , Figure 6B An interaction diagram illustrating a task execution process provided in an embodiment of this application is shown in a multi-tasking scenario. For example... Figure 6B As shown, intelligent assistants can rely on system agents and at least one application control agent. Figure 6BTaking the first application control agent, the second application control agent, and the third application control agent as examples, a multi-layered agent architecture is used to execute tasks based on user commands through S601B-S605B. The relevant functions of the intelligent assistant may be completed by electronic devices, servers, or a combination of electronic devices and servers, without specific limitations.

[0183] S601B: The system agent obtains user instructions corresponding to user operations.

[0184] For a detailed introduction to S601B, please refer to the description of S601A above, which will not be repeated here.

[0185] S602B: The system agent determines the task corresponding to the user instruction based on the user instruction and the first memory information.

[0186] For a detailed introduction to S602B, please refer to the description of S602A above, which will not be repeated here.

[0187] S603B: The system agent breaks down the task corresponding to the user command into multiple subtasks and determines one or more application manipulation agents to execute the multiple subtasks respectively.

[0188] The system agent may break down the task corresponding to a user command into at least two subtasks. For example, the system agent may break down the task corresponding to a user command into a first subtask and a second subtask. Or, the system agent may break down the task corresponding to a user command into a first subtask, a second subtask, and a third subtask.

[0189] As one possible implementation, the system agent can select one or more application control agents capable of executing each subtask from multiple application control agents based on the subtasks obtained after breaking down the task corresponding to the user command. Different subtasks may be executed by different application control agents, or they may be executed by the same application control agent, depending on the specific content of the subtask, and there is no limitation on this.

[0190] As a possible example, the system agent may split the task corresponding to the user command into multiple subtasks, which may need to be executed sequentially in a certain order.

[0191] For example, taking the system agent splitting the task corresponding to the user command into a first subtask, a second subtask, and a third subtask as an example, the execution order of the first subtask, the second subtask, and the third subtask may be: first subtask → second subtask → third subtask. That is, when executing multiple subtasks, the subtasks need to be executed in the order of first executing the first subtask, then executing the second subtask, and finally executing the third subtask.

[0192] For example, taking the user instruction corresponding to the task "Purchase a serving of celery dumplings from store B in application A and deliver it to address C", the system agent can analyze and break down this task into the following three sub-tasks:

[0193] First subtask: Open application A;

[0194] Second subtask: Search for store B in application A;

[0195] Third sub-task: Order a serving of celery dumplings from store B and have it delivered to address C.

[0196] For example, taking the user instruction corresponding to the task "Buy a serving of celery dumplings from store B in application A and deliver it to address C", the system agent can analyze and break down this task into the following three sub-tasks:

[0197] First subtask: Open application A;

[0198] Second subtask: Retrieve historical orders for "Purchase a serving of celery dumplings from store B in app A and deliver it to address C";

[0199] The third sub-task: Go to the celery dumpling product page through historical orders, place an order for one serving of celery dumplings, and have it delivered to address C.

[0200] In the example where the task corresponding to the two user instructions above is "to purchase a serving of celery dumplings from store B in application A and deliver them to address C", the first subtask, the second subtask, and the third subtask need to be executed in the following order: first subtask, then second subtask, and finally third subtask.

[0201] It should be noted that the above examples of splitting tasks corresponding to user commands are only two examples. In actual applications, how to split tasks depends on the specific scenario and the specific task corresponding to the user command, and no specific limitations are made.

[0202] In addition, in practical applications, the multiple subtasks obtained by the system agent from splitting the task corresponding to the user command may not need to be executed in a certain order, depending on the specific situation.

[0203] For example, taking the system agent splitting the task corresponding to the user command into a first subtask, a second subtask, and a third subtask as an example, there are no execution order restrictions for the first subtask, the second subtask, and the third subtask. That is, when executing multiple subtasks, the multiple subtasks can be performed synchronously and independently.

[0204] S604B: The system agent distributes multiple subtasks to one or more designated application manipulation agents.

[0205] As one possible implementation, the system agent can distribute multiple subtasks to different application control agents.

[0206] For example, taking multiple subtasks including a first subtask, a second subtask, and a third subtask, the system agent can distribute the first subtask to a first application control agent, the second subtask to a second application control agent, and the third subtask to a third application control agent.

[0207] As one possible implementation, the system agent can distribute at least two of the multiple subtasks to the same application manipulation agent.

[0208] For example, taking multiple subtasks including a first subtask, a second subtask, and a third subtask as an example, the system agent can distribute the first subtask and the second subtask to the first application control agent, and distribute the third subtask to the third application control agent.

[0209] As one possible implementation, the system agent may synchronously distribute multiple subtasks to one or more designated application manipulation agents.

[0210] For example, when multiple subtasks do not need to be executed in a specific order, the system agent can synchronously distribute multiple subtasks to one or more designated application control agents.

[0211] For example, when multiple subtasks need to be executed in a certain order, the system agent can instruct one or more application control agents to execute the subtasks in a specific order when distributing the subtasks to them.

[0212] As another possible implementation, the system agent may synchronously distribute multiple subtasks to one or more designated application manipulation agents in a certain order.

[0213] For example, when multiple subtasks need to be executed sequentially in a certain order, the system agent can distribute the multiple subtasks to one or more designated application control agents in a certain order.

[0214] For example, taking multiple subtasks including a first subtask, a second subtask, and a third subtask, with the execution order of the first subtask, the second subtask, and the third subtask being: first subtask → second subtask → third subtask, the system agent can first distribute the first subtask to the first application control agent. After the first application control agent completes the execution of the first subtask, it sends the second subtask to the second application control agent. After the second application control agent completes the execution of the second subtask, it sends the third subtask to the third application control agent.

[0215] There are no restrictions on the specific methods and approaches for the system agent to distribute multiple sub-tasks; they can be determined based on the specific scenario and the specific task.

[0216] S605B: The above one or more application control agents execute the corresponding subtasks.

[0217] As a possible example, upon receiving a subtask dispatched by a system agent, one or more application manipulation agents invoke relevant execution engines, tools, or interfaces to execute the corresponding subtask. For instance, if the subtask is an operation within the application, the application manipulation agent can use UI-based understanding to invoke execution engines, tools, or interfaces to perform relevant application manipulations.

[0218] For example, taking the first application control agent as executing the first subtask, the second application control agent as executing the second subtask, and the third application control agent as executing the third subtask as an example, the first application control agent calls the first execution engine to execute the first subtask, the second application control agent calls the second execution engine to execute the second subtask, and the third application control agent calls the third execution engine to execute the third subtask.

[0219] As a possible example, when the system agent distributes multiple subtasks to one or more designated application manipulation agents, it instructs the one or more application manipulation agents to execute the multiple subtasks in a specific order. In this case, the one or more application manipulation agents execute the corresponding subtasks in the order specified by the system agent.

[0220] For example, taking the example of a first application control agent executing a first subtask, a second application control agent executing a second subtask, and a third application control agent executing a third subtask, with the execution order of the first, second, and third subtasks being: first subtask → second subtask → third subtask, S605B includes: the first application control agent calling the first execution engine to execute the first subtask; after the first subtask is completed, the second application control agent calling the second execution engine to execute the second subtask; after the second subtask is completed, the third application control agent calling the third execution engine to execute the third subtask.

[0221] For example, if the first application control agent is used to execute the first subtask and the second subtask, and the third application control agent is used to execute the third subtask, and the execution order of the first subtask, the second subtask and the third subtask is: first subtask → second subtask → third subtask, S605B includes: the first application control agent calls the first execution engine to execute the first subtask, and after the first subtask is completed, it calls the second execution engine to execute the second subtask; after the second subtask is completed, the third application control agent calls the third execution engine to execute the third subtask.

[0222] In the above example, any two of the first execution engine, the second execution engine, and the third execution engine may be the same execution engine or different engines, without specific limitations.

[0223] In some embodiments, while each application control agent is executing its own subtask according to the instructions of the system agent, each application control agent may also supplement the subtask based on its own historical records (i.e., second memory information) of the relevant historical tasks, and each application control agent executes the task according to its supplemented subtask.

[0224] As an example, if one or more application control agents are deployed on an electronic device, the aforementioned one or more application control agents can call the relevant execution engine, tools, or interfaces of the electronic device to execute corresponding subtasks.

[0225] As an example, if one or more application control agents are deployed on a server, the one or more application control agents can invoke the communication connection between the server and the electronic device to send their respective execution engine invocation instructions to the electronic device; correspondingly, the electronic device invokes the relevant execution engine to execute multiple subtasks according to the execution engine invocation instructions of the one or more application control agents.

[0226] based on Figure 6A and Figure 6B The task execution scheme shown allows the system agent to perform fusion analysis and reasoning based on historical records of relevant tasks (such as session context) when determining the task corresponding to a user command. This eliminates the need for users to input overly complex commands and accurately identifies the task corresponding to the user command, enabling timely, accurate, and reliable responses to user commands.

[0227] And, based on Figure 6A and Figure 6B The task execution scheme shown avoids the inefficiencies, inaccuracies, and reliability caused by excessively long session contexts in the analysis, reasoning, or task execution process by using communication protocols between the system agent and application control agent layers and managing session contexts at both levels. This assists each layer in performing relevant analysis, reasoning, or task execution, further improving the timeliness, accuracy, and reliability of task execution, as well as increasing the isolation between layers in the multi-layer agent architecture, ensuring the safe and stable operation of the system.

[0228] Furthermore, based on Figure 6A and Figure 6BThe task execution scheme shown uses a two-layer agent architecture, based on language models such as LLM and MLLM, to achieve user instruction understanding, multi-turn dialogue fusion, multimodal analysis and reasoning, multi-layer task management and isolation, and task switching in a layered manner from easy to difficult. This not only saves reasoning costs but also improves reasoning efficiency and the timeliness of user instruction response.

[0229] In addition, based on Figure 6B The task execution scheme shown in the multi-task scenario ensures smooth human-computer interaction and timely, accurate, and reliable task execution in complex scenarios through flexible, timely, and accurate switching between multiple tasks.

[0230] In some embodiments, during the execution of a task by the application control agent, if clarification of certain issues or supplementation of certain details regarding the task is required, the application control agent may request the system agent to interact with the user to clarify the task. After clarification, the system agent updates or supplements the task based on the clarification result and continues task execution. For example, during the execution of a task by the application control agent, the system agent may interact with the user based on the application control agent's task clarification request, obtain the clarification result, and instruct the application control agent to continue executing the task based on the clarification result, and so on, until the task is successfully completed.

[0231] As an example, the above task clarification can be... Figure 6A The task can be performed during its execution or in [the following context]. Figure 6B This is done during the execution of the task.

[0232] exist Figure 6A During the task execution process, while the application control agent executes S605A and performs the task according to the system agent's instructions, if it is necessary to clarify certain issues or supplement certain details about the task, the application control agent sends a task clarification request to the system agent. This request carries relevant information about the issues to be clarified or the details to be supplemented. Upon receiving the task clarification request from the application control agent, the system agent interacts with the user through the human-computer interaction module of the electronic device, such as through a smart assistant, in one or more rounds to clarify the relevant issues or obtain relevant details. After obtaining the clarification result, the system agent sends the clarification result back to the application control agent. The application control agent updates or supplements the task based on the clarification result and the second memory information, and continues the task execution; this process continues until the application control agent successfully completes the task execution.

[0233] exist Figure 6BDuring the task execution process, while one or more application control agents are executing S605B and their respective sub-tasks, if an application control agent needs to clarify certain issues or supplement certain details about a sub-task, it sends a task clarification request to the system agent. This request carries information about the issues requiring clarification or the details requiring supplementation. Upon receiving the task clarification request from the application control agent, the system agent interacts with the user through the human-computer interaction module of the electronic device, such as through a smart assistant in the electronic device, in one or more rounds to clarify the relevant issues or obtain the relevant details. After obtaining the clarification result, the system agent sends the clarification result back to the application control agent. The application control agent updates or supplements the sub-task based on the clarification result and the second memory information, and then continues the execution of the sub-task. This process continues until the application control agent successfully completes the execution of the sub-task.

[0234] As an example, please refer to Figure 7 , Figure 7 Taking the execution of a task as an example, which requires the participation of a first application control agent and a second application agent, and where the execution of the task by the first application control agent involves a task clarification process, a schematic diagram of a task execution interaction process provided in an embodiment of this application is shown. Figure 7 As shown, task execution based on user commands can be achieved through S701-S715:

[0235] S701: System agent obtains user commands.

[0236] S702: The system agent obtains the first memory information.

[0237] As an example, the system agent can... Figure 7 The memory module shown acquires first memory information. This memory module stores the first memory information, which includes historical records of tasks executed by the system agent. For example, the first memory information includes historical records of tasks related to user instructions executed by the system agent.

[0238] As an example, the system agent can also obtain first-memory information from the intelligent assistant, which may be from... Figure 7 The memory module shown acquires the first memory information.

[0239] For example, Figure 7 The memory module shown may include a first memory module, which may be located in the system agent or outside the system agent, without specific limitations.

[0240] S703: The system agent determines the task corresponding to the user instruction based on the first memory information.

[0241] S704: The system agent splits the task corresponding to the user instruction into a first subtask and a second subtask, and determines a first application control agent and a second application control agent to execute the first subtask and the second subtask, respectively.

[0242] For example, the execution order of the first subtask and the second subtask is: first subtask → second subtask.

[0243] Of course, in some embodiments, the first subtask and the second subtask may not need to be executed in a certain order, depending on the specific circumstances. Figure 7 This is just one example.

[0244] S705: The system agent distributes the first subtask to the first application control agent.

[0245] S706: The first application control agent obtains the second memory information of the first application control agent.

[0246] As an example, the first application can manipulate the agent from Figure 7 The memory module shown acquires the second memory information of the first application control agent. The memory module stores the second memory information of the first application control agent, which includes historical records of tasks executed by the first application control agent. For example, the second memory information of the first application control agent includes historical records of tasks related to the execution of the first subtask.

[0247] For example, Figure 7 The memory module shown may include a second memory module, which may be located in the first application control agent or outside the first application control agent, without specific limitations.

[0248] S707: The first application control agent calls the execution engine to start executing the first subtask based on the second memory information of the first application control agent.

[0249] S708: During the execution of the first subtask, the first application control agent sends a task clarification request to the system agent.

[0250] The task clarification request is used to request clarification of the first subtask.

[0251] As an example, a task clarification request may carry information about issues that need clarification or details that need to be added to the first subtask.

[0252] S709: The system agent interacts with the user based on the task clarification request to obtain the clarification result regarding the first subtask.

[0253] As an example, the system agent can interact with the user through the human-computer interaction module of an electronic device, such as through a smart assistant in the electronic device, in one or more rounds to clarify questions or obtain relevant details about the first subtask, and finally obtain the clarification result about the first subtask.

[0254] S710: The system agent sends the clarification results regarding the clarification of the first subtask to the first application control agent.

[0255] S711: The first application control agent continues to execute the first subtask based on the clarification result.

[0256] As an example, the first application manipulation agent can update or supplement the first subtask based on the clarification results and continue to execute the first subtask.

[0257] S712: After completing the execution of the first subtask, the first application control agent sends the first response to the system agent.

[0258] The first response is used to indicate that the first subtask has been completed.

[0259] S713: The system agent distributes the second subtask to the second application control agent.

[0260] S714: The second application control agent obtains the second memory information of the second application control agent.

[0261] As an example, the second application can manipulate the agent from Figure 7 The memory module shown acquires the second memory information of the second application control agent. The memory module is also used to store the second memory information of the second application control agent, which includes historical records of tasks executed by the second application control agent. For example, the second memory information of the second application control agent includes historical records of tasks related to the execution of a second subtask.

[0262] For example, Figure 7 The memory module shown may include a third memory module, which may be located in the second application control agent or outside the second application control agent, without specific limitations.

[0263] S715: The second application control agent calls the execution engine to start executing the second subtask based on the second memory information of the second application control agent.

[0264] about Figure 7 For details regarding S701, please refer to the descriptions of S601A or S601B above; regarding... Figure 7For details regarding S702-S703, please refer to the explanations of S602A or S602B above; regarding Figure 7 For details regarding S704, please refer to the explanation of S603B above; regarding... Figure 7 For details regarding S705 and S713, please refer to the explanation of S604B above; regarding Figure 7 The relevant descriptions of S706-S707 and S714-S715 shown can be found in the description of S605B above, and will not be repeated here.

[0265] based on Figure 7 The task execution scheme shown allows the application control agent to request the system agent to trigger task clarification during the execution of subtasks when it is necessary to clarify certain issues or supplement certain details about the subtasks. This clarification, achieved through one or more rounds of interaction with the user, further ensures the accurate and reliable execution of tasks based on user instructions.

[0266] The following will use the user command "order another serving of dumplings" as an example to illustrate the specific implementation process of the task execution method provided in this application embodiment, combined with a specific scenario. For example, please refer to... Figure 8 , Figure 8 This illustration shows a schematic diagram of the task execution process in response to user instructions in a specific application scenario provided by an embodiment of this application.

[0267] like Figure 8 As shown in S801, after receiving the user's query "Order me another serving of dumplings", the smart assistant obtains the system context. This includes, but is not limited to, information about the electronic device (such as device identifier, device name, login username, etc.), the running application A (such as foreground application, background application, etc.), and location information (such as location coordinates collected by GPS).

[0268] like Figure 8 As shown in S802, the intelligent assistant obtains the first memory information of the system agent, such as the first memory information indicating that the user likes dumplings from family B, and indicating that the user's home address is address C.

[0269] like Figure 8 As shown in S803, the system agent determines the first prompt information (such as the first prompt) as input to the language model based on the user instruction, the first memory information, and the system context. The language model then performs fusion analysis and reasoning on the first prompt information to determine the task corresponding to the user instruction as "Application A control agent controls the A application interface to purchase a serving of dumplings and deliver it to address C". After determining the task, the system agent distributes the task to the A application control agent.

[0270] like Figure 8 As shown in S804 and S805, the A application control agent determines the second prompt information (such as the second prompt) as the input of the language model based on the task indicated by the system agent and the second memory information of the A application control agent. The language model performs fusion analysis and reasoning on the second prompt information, determines and calls the execution engine (such as the intelligent assistant) to execute the task instruction, such as: search (dumpling), LastAction:False, AgentState:continue, and the execution engine (such as the intelligent assistant) executes the task instruction.

[0271] like Figure 8 As shown in S806-S807, since the application A control agent is unsure what kind of dumplings the user wants to buy, the application A control agent requests the system agent to trigger a task clarification to ask the user, "What kind of dumplings do you want?"

[0272] like Figure 8 As shown in S808, after receiving the message "Help me call and ask my dad," the smart assistant obtains the current system context, including but not limited to information about the electronic device (such as device identifier, device name, login username, etc.), the running application A (such as foreground application, background application, etc.), location information (such as location coordinates collected by GPS), and interface information of application A; and sends this information to the system agent.

[0273] like Figure 8 As shown in S809-S810, the system agent determines the task as "call (dad)" based on "call Dad for me", the first memory information and the system context, and calls the execution engine (such as a smart assistant) to execute the task.

[0274] like Figure 8 As shown in S811 and S812, after the call is completed, the smart assistant determines that the user wants celery dumplings and sends this information, the current system context, and the first memory information to the system agent.

[0275] like Figure 8 As shown in S813 and S814, the system agent determines the clarification result "I want celery dumplings" based on the above information from the smart assistant, and instructs the A application control agent to continue executing "The A application control agent controls the A application interface to purchase a serving of celery dumplings and deliver it to address C".

[0276] like Figure 8 As shown in S815, the A application control agent continues to execute the task according to the instructions of the system agent, and completes the task "The A application control agent controls the A application interface to purchase a serving of celery dumplings and deliver them to address C".

[0277] based on Figure 8 In the specific application scenario shown, regarding the task execution process of user commands, when determining the task corresponding to the user command "Order me another serving of dumplings," the system agent can combine the system agent's first memory information, such as the running application A, the user's favorite store B, and the user's home address, to perform fusion analysis and reasoning. Ultimately, it determines the task corresponding to the user command as "Application A controls the agent to control the interface of application A to purchase a serving of dumplings and deliver it to address C." Through one or more rounds of interaction with the user, it clarifies what kind of dumplings the user wants. On the one hand, it eliminates the need for the user to input overly complex commands, and on the other hand, it can accurately identify the task corresponding to the user command, responding to the user command in a timely, accurate, and reliable manner.

[0278] And, based on Figure 8 The task execution scheme shown, through interaction between the system agent and application manipulation agent layers following prescribed communication specifications, and through hierarchical management of memory information (such as first and second memory information) at both levels, can avoid the inefficiencies, low accuracy, and low reliability caused by excessively long historical records (such as session context) in the analysis, reasoning, or task execution process. It assists each layer in performing relevant analysis, reasoning, or task execution, further improving the timeliness, accuracy, and reliability of task execution, and enhancing the isolation between different layers in the multi-layered agent architecture, ensuring the secure and stable operation of the system. Furthermore, based on… Figure 8 The task execution scheme shown employs a two-layer agent architecture, based on language models such as LLM and MLLM, to achieve user command understanding, multi-turn dialogue fusion, multimodal analysis and reasoning, multi-layer task management and isolation, and task switching in a hierarchical manner from easy to difficult. This not only saves reasoning costs but also improves reasoning efficiency and the timeliness of user command response. Furthermore, based on... Figure 8 The task execution scheme shown in the multi-task scenario ensures smooth human-computer interaction and timely, accurate, and reliable task execution in complex scenarios through flexible, timely, and accurate switching between multiple tasks.

[0279] It should be understood that the various solutions in the embodiments of this application can be used in a reasonable combination, and the explanations or descriptions of the various terms appearing in the embodiments can be referenced or explained to each other in the various embodiments, without limitation.

[0280] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0281] It is understood that, in order to implement the functions of any of the above embodiments, electronic devices or servers include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0282] This application embodiment can divide electronic devices or servers into functional modules. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0283] It should also be understood that the various modules in an electronic device or server can be implemented in software and / or hardware, without specific limitations. In other words, the electronic device or server is presented in the form of functional modules. Here, "module" can refer to application-specific integrated circuits (ASICs), circuits, processors and memory that execute one or more software or firmware programs, integrated logic circuits, and / or other devices that can provide the above functions.

[0284] In an alternative approach, when data transmission is implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disk (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0285] The steps of the methods or algorithms described in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, portable hard disk, CD-ROM, or any other form of storage medium well known in the art. One exemplary embodiment couples a storage medium to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Furthermore, the ASIC can reside in an electronic device or server. Alternatively, the processor and storage medium can exist as discrete components.

[0286] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

Claims

1. A task execution method, characterized in that, The method includes: The system agent obtains user commands; The system agent determines the target task corresponding to the user instruction based on the user instruction and the first memory information, wherein the first memory information includes a history of historical tasks related to the execution of the target task; The system agent instructs at least one application control agent corresponding to the target task to execute the target task; During the execution of the target task by the at least one application control agent according to the instructions of the system agent, the system agent receives a task clarification request from the at least one application control agent. The task clarification request is used to indicate questions or details about the target task that need to be clarified. In response to the task clarification request, the system agent interacts with the user; The system agent obtains a clarification result based on the interaction, and the clarification result is used to answer the question or supplement the details; and the system agent instructs the at least one application manipulation agent to continue executing the target task based on the clarification result. The system agent obtains the clarification result based on the interaction, including: The system agent acquires the information input by the user during the interaction; The system agent obtains the clarification result by analyzing the input information; or; The system agent obtains clarification results based on the interaction, including: The system agent acquires the information input by the user during the interaction; The system agent plans clarification tasks based on the input information; The system agent distributes the clarification task to the application manipulation agent corresponding to the clarification task; The system agent obtains the clarification result based on the execution result of the clarification task.

2. The method according to claim 1, characterized in that, The system agent instructs the at least one application manipulation agent to continue executing the target task based on the clarification result, including: The system agent instructs the at least one application manipulation agent to update or supplement the subtask based on the clarification result and continue the execution of the subtask.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The at least one application control agent executes the target task based on the target task and the second memory information of the at least one application control agent, wherein the second memory information includes a historical record of historical tasks related to the execution of the target task by the at least one application control agent.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The system agent breaks down the target task into multiple subtasks, including a first subtask and a second subtask; the system agent determines a first application control agent for executing the first subtask and a second application control agent for executing the second subtask. The system agent instructs at least one application manipulation agent corresponding to the target task to execute the target task, including: The system agent distributes the first subtask to the first application control agent and the second subtask to the second application control agent.

5. The method according to any one of claims 1-4, characterized in that, The system agent determines the target task corresponding to the user instruction based on the user instruction and the first memory information, including: The system agent determines the target task corresponding to the user instruction based on the user instruction and the first memory information, using a language model.

6. The method according to any one of claims 1-5, characterized in that, The system agent is deployed on a server, and the system agent obtains user instructions, including: The system agent receives the user instructions from the electronic device; The method further includes: The at least one application control agent invokes the execution engine, tools, or interfaces on the electronic device to execute the target task.

7. The method according to any one of claims 1-5, characterized in that, The system agent is deployed on an electronic device, and the method further includes: The at least one application control agent invokes the execution engine, tools, or interfaces on the electronic device to execute the target task.

8. The method according to any one of claims 1-7, characterized in that, The system agent is a system intelligent agent, and the application control agent is a domain intelligent agent.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store computer program instructions; A processor for executing the computer program instructions to support the electronic device in implementing the method as described in any one of claims 1-5 and 7-8.

10. A server, characterized in that, The server includes: Memory is used to store computer program instructions; A processor for executing the computer program instructions to support the server in implementing the method as described in any one of claims 1-6 and 8.

11. A task execution system, characterized in that, The task execution system includes: Electronic devices and / or servers are used to support the task execution system in implementing the method as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processing circuit, implement the method as described in any one of claims 1-8.

13. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-8.

Images