Task execution method, device and system

Abstract

The invention discloses a task execution method, task execution equipment and a task execution system, relates to the technical field of terminals, and aims to timely, accurately and reliably respond to user instructions in various scenes, including user instructions in complex scenes, so that the use experience of a user is improved. According to the scheme, when the task corresponding to the user instruction is determined, the system agent can perform fusion analysis and reasoning in combination with the historical records of the historical tasks related to execution of the task, and finally the task corresponding to the user instruction is determined, so that on one hand, a user does not need to input too complex instructions; and on the other hand, the task corresponding to the user instruction can be accurately identified, and the user instruction can be timely, accurately and reliably responded.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 20, 2024, with application number 202410808507.3 and application name “Task Execution Method, Device and System”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a task execution method, device and system. Background Art

[0003] At present, intelligent assistants based on language models are widely used in the process of human-computer interaction. Intelligent assistants can simplify user operations and accurately respond to user instructions during human-computer interaction. For example, intelligent assistants can analyze and reason about user instructions based on language models, create corresponding tasks and execute the tasks to accurately respond to user instructions. However, conventional intelligent assistants can only perform simple tasks. For some complex scenarios, such as scenarios involving multiple cross-application calls, scenarios with complex reasoning processes, scenarios that require clarification of issues, etc., they cannot accurately respond to user instructions. Summary of the invention

[0004] The present application provides a task execution method, device and system that can promptly, accurately and reliably respond to user instructions in various scenarios, including user instructions in complex scenarios, thereby improving the user experience.

[0005] In order to achieve the above purpose, this application adopts the following technical solutions:

[0006] In a first aspect, a task execution method is provided, which can be applied to a system including a system agent and at least one application control agent, the method comprising: the system agent obtains a user instruction; the system agent determines the task corresponding to the user instruction based on the user instruction and first memory information, the first memory information including a historical record of historical tasks related to the execution of the above task; the system agent instructs the application control agent to execute the above task.

[0007] In the solution provided in the first aspect above, the system agent can, when determining the task corresponding to the user instruction, combine the historical records of historical tasks related to the execution of the above task for fusion analysis and reasoning, and finally determine the task corresponding to the user instruction. On the one hand, there is no need for the user to input overly complex instructions, and on the other hand, it can also accurately identify the task corresponding to the user instruction and respond to the user instruction in a timely, accurate and reliably manner.

[0008] As a possible implementation, the method further includes: during the process of the application control agent executing the task, the system agent interacts with the user according to the task clarification request of the application control agent to obtain the clarification result; the system agent instructs the application control agent to continue to execute the task according to the clarification result. In this way, through one or more rounds of interaction, some questions about the task can be clarified and / or some details about the task can be supplemented, so as to realize the clear, timely, accurate and reliable execution of the task, and intelligently respond to user instructions in different scenarios, including simple and complex scenarios, while simplifying user operations.

[0009] As a possible implementation method, the system agent interacts with the user according to the task clarification request of the application control agent to obtain the clarification result, including: the system agent interacts with the user according to the task clarification request of the application control agent to obtain the information input by the user; the system agent instructs the application control agent to perform the clarification task according to the information input by the user, and obtains the clarification result according to the execution result of the clarification task. In this way, through one or more rounds of interaction, some questions about the task can be clarified and / or some details about the task can be supplemented, so as to achieve clear, timely, accurate and reliable execution of the task, and intelligently respond to user instructions in different scenarios, including simple and complex scenarios, while simplifying user operations.

[0010] As a possible implementation, the method further includes: the application control agent performs the task according to the task and the second memory information of the application control agent, and the second memory information includes the historical records of the application control agent performing the task. In this way, the task can be supplemented according to the historical records to achieve clear, timely, accurate and reliable execution of the task.

[0011] As a possible implementation, the method further includes: the system agent splits the task into multiple subtasks, including a first subtask and a second subtask; the system agent determines a first application control agent and a second application control agent for executing the first subtask, and determines a second application control agent for executing the second subtask; the system agent instructs the application control agent to execute the task, including: the system agent distributes the first subtask to the first application control agent, and distributes the second subtask to the second application control agent. In this way, the user's instructions in different scenarios, especially complex scenarios, can be intelligently responded to under the premise of simplifying the user's operation.

[0012] As a possible implementation, the system agent instructs the application control agent to perform the task, including: the system agent distributes the task to the third application control agent. In this way, it is possible to intelligently respond to user instructions in different scenarios, including simple scenarios, while simplifying user operations.

[0013] As a possible implementation, the system agent determines the task corresponding to the user instruction according to the user instruction and the first memory information, including: the system agent determines the task corresponding to the user instruction based on the language model according to the user instruction and the first memory information. In this way, fusion analysis and reasoning can be performed based on the language model to accurately determine the task corresponding to the user instruction.

[0014] As an example, the above-mentioned language model includes at least one of the following: a large-scale language model (LLM) and a multimodal large-scale language model (MLLM).

[0015] As a possible implementation method, the above system is deployed on a server, and the above system agent obtains user instructions, including: the system agent receives user instructions from the electronic device; the above method also includes: the application control agent calls the execution engine, tool or interface on the electronic device to perform the above task. In this way, the task execution method provided by this application can be used under the end-cloud collaborative architecture to improve the applicability of this solution.

[0016] As a possible implementation, the system is deployed on an electronic device, and the method further includes: the application control agent calls the execution engine, tool or interface on the electronic device to perform the task. In this way, the task execution method provided by the present application can be used in a single device architecture to improve the applicability of the solution.

[0017] As a possible implementation, the system agent is a system agent, and the application control agent is a domain agent. In this way, the task execution method provided by this application can be used in a multi-agent architecture to improve the applicability of this solution.

[0018] In a second aspect, an electronic device is provided, comprising: a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device to implement a method as in any possible implementation of the first aspect.

[0019] In a third aspect, a server is provided, comprising: a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device to implement a method as in any possible implementation of the first aspect.

[0020] In a fourth aspect, a task execution system is provided, which includes: an electronic device and / or a server, wherein the electronic device and / or the server are used to support the task execution system to implement a method as in any possible implementation of the first aspect.

[0021] According to a fifth aspect, a computer program product comprising instructions is provided. When the computer program product is run on a computer, the computer implements the method in any possible implementation of the first aspect.

[0022] In a sixth aspect, a chip system is provided, the chip system comprising a processing circuit and a storage medium, wherein the storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the method in any possible implementation of the first aspect is implemented. The chip system may be composed of a chip, or may include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a multi-layer proxy architecture provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a multi-layer agent architecture provided in an embodiment of the present application;

[0025] Figure 3 Two task execution architecture diagrams provided for embodiments of the present application;

[0026] Figure 4 A schematic diagram of the structure of two system architectures provided in the embodiments of the present application;

[0027] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0028] Fig. 6A An interactive diagram of a task execution process provided in an embodiment of the present application;

[0029] Figure 6B Another task execution process interaction diagram provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of a task execution interaction process provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of a task execution process for a user instruction in a specific application scenario provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0033] In the following, the terms "first", "second", etc. are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. For example, if the described object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the described object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of described objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes can be the same or different. For example, if the described object is a "device", the "first device" and the "second device" can be the same type of device or different types of devices. For another example, if the described object is "information", the "first information" and the "second information" can be information of the same content or information of different contents. In summary, the use of prefixes such as ordinal numbers to distinguish the described objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes.

[0034] Furthermore, in the embodiments of the present application, "connection" may be a direct connection or an indirect connection; in addition, it may refer to an electrical connection or a communication connection; for example, the connection between two electrical components A and B may refer to a direct connection between A and B, or may refer to an indirect connection between A and B through other electrical components or connecting media, or may refer to an indirect connection between A and B through other communication devices or communication media, as long as communication between A and B can be achieved.

[0035] As described in the background art, conventional intelligent assistants can only perform some simple tasks and cannot accurately respond to user instructions in complex scenarios.

[0036] As an example, when the smart assistant performs corresponding tasks based on user instructions, the smart assistant can use the language model to analyze and reason about the user instructions (such as prompt), determine the tasks to be performed and the tools used to perform the tasks, call the tools to perform the corresponding tasks, and after completing the tasks, feed back the execution results (observation) to the language model for corresponding language model adjustments to improve the reliability and accuracy of the analysis and reasoning results of the language model. In this process, the reasoning mode of the smart assistant is mainly to react (ReAct), which only involves one interaction, reasoning and execution process. The process is relatively simple, so the tasks that can be handled are also relatively simple. Moreover, in this process, the tasks that need to be performed mainly rely on the analysis and reasoning of user instructions through the language model. Therefore, the accurate and reliable execution of the tasks strongly depends on the accuracy and completeness of the user instructions, so the instructions input by the user are required to be high.

[0037] In order to handle more complex tasks in some complex scenarios, as an example, when executing corresponding tasks based on user instructions, the intelligent assistant can follow the internal and external loop mechanism, analyze and reason the user instructions step by step through the language model, determine the tasks to be performed and the tools used to perform the tasks, and call the tools to perform the corresponding tasks. Among them, in the internal and external loop mechanism, the outer loop part is mainly used for high-level task management, such as identifying tasks based on user instructions and dividing tasks into multiple operable subtasks; the inner loop part is mainly used for the execution of underlying tasks, such as responsible for the execution of fine-grained subtasks. The internal and external loop mechanism followed by this solution can improve the reliability and accuracy of the analysis and reasoning results, improve the reliability and execution efficiency of task execution, and is easy to integrate and compatible with external tools. However, the reasoning mode of the intelligent assistant in this solution is mainly task execution (Action), which still only involves one interaction, reasoning and execution process. The process is relatively simple, so the tasks that can be handled are still relatively simple. Moreover, in this solution, the tasks to be performed are mainly determined by analyzing and reasoning the user instructions through the language model. Therefore, the accurate and reliable execution of the tasks is also strongly dependent on the accuracy and completeness of the user instructions. Therefore, the requirements for the instructions input by the user are also high.

[0038] As an example, when the intelligent assistant performs the corresponding task based on the user's instructions, the intelligent assistant can use the language model to complete the analysis and reasoning recognition tasks based on the user's instructions (such as prompt), and use the multimodal agent architecture to interact with the operating system to navigate in multiple applications to complete the task. Among them, the multimodal agent architecture includes an application agent (Appagent) and an execution agent (Actagent). The application agent (Appagent) is used to decompose the task into multiple subtasks and distribute the multiple subtasks to the corresponding execution agent (Actagent); the execution agent (Actagent) is used to execute the subtasks. This scheme is similar to the internal and external loop mechanism described above. Although it can improve the reliability and accuracy of the analysis and reasoning results, improve the reliability and execution efficiency of task execution, and improve the possibility of integration and compatibility with external tools, it still only involves one interaction, reasoning and execution process. The process is relatively simple, so the tasks that can be handled are still relatively simple. In addition, since the tasks that need to be performed are mainly determined by analyzing and reasoning about user instructions through language models, the accurate and reliable execution of tasks is also strongly dependent on the accuracy and completeness of user instructions, so the requirements for user input instructions are also high.

[0039] From the above examples, we can see that conventional intelligent assistants mostly involve relatively simple and fixed reasoning modes, so they can only perform relatively simple tasks. They cannot accurately respond to user instructions in complex scenarios such as scenarios involving multiple cross-application calls, scenarios with complex reasoning processes, and scenarios that require clarification of issues. In addition, conventional intelligent assistants rely heavily on the accuracy and completeness of user instructions when determining the tasks to be performed, so they also have high requirements for user input instructions.

[0040] With the emergence of speech models with higher computing power and accuracy, such as large-scale language model (LLM) and multimodal large-scale language model (MLLM), the advantages of multi-layer agent architecture based on language models such as LLM and MLLM are becoming more and more obvious. For example, smart assistants built on multi-layer agent architecture can rely on language models such as LLM and MLLM as super entrances, and provide natural dialogue understanding, multi-round dialogue fusion, multimodal analysis and reasoning, historical record hierarchical management, multi-layer task management and isolation, task switching, etc. through multi-layer agent division of labor and cooperation to help users handle various types of more complex tasks such as task-based, chat-based, question-and-answer-based, and AI-generated content (AIGC)-based tasks.

[0041] In the context of a multi-layer proxy architecture, an embodiment of the present application provides a task execution method based on a multi-layer proxy architecture, which can respond to different scenarios in a timely, accurate and reliable manner, including user instructions in simple and complex scenarios. For example, based on the task execution method provided by the embodiment of the present application, the smart assistant can rely on a multi-layer proxy architecture including a system agent and an application control agent to perform relevant processing layer by layer, such as determining a task through a system agent based on user instructions and historical records of historical tasks recorded by the system agent, and splitting the task into multiple subtasks and distributing them to different application control agents, and then completing the corresponding subtasks through the application control agent. When the application control agent executes the subtask, it can also assist in supplementing the relevant subtasks based on the historical records of the relevant historical tasks recorded by the application control agent. In this way, more timely, accurate and reliable task execution for user instructions can be achieved. Exemplarily, historical records may include, but are not limited to, session contexts, etc., without limitation.

[0042] In addition, the smart assistant can also interact with the user for one or more rounds to clarify the relevant tasks when the system agent needs to clarify certain issues about the task or supplement certain details about the task, and switch back to the system agent after the clarification to update or supplement the user instructions according to the clarification results and continue to perform task identification, task planning, task distribution, etc.; and / or, the smart assistant can also interact with the user for one or more rounds to clarify the relevant tasks when the application control agent needs to clarify certain issues about the task or supplement certain details about the task, and switch back to the application control agent after the clarification to update or supplement the subtask according to the clarification results and continue to perform the subtask, and so on, until the execution of the subtask is successfully completed. In this way, through natural dialogue understanding, multi-round dialogue fusion, multimodal analysis and reasoning, historical record hierarchical management, multi-layer task management and isolation, task switching and other processes, the multi-layer agent architecture can support user instruction responses in complex scenarios such as scenarios involving multiple cross-application calls, scenarios with complex reasoning processes, and scenarios where clarification of issues is required.

[0043] As an example, see Figure 1 , Figure 1 A schematic diagram of a multi-layer proxy architecture provided by an embodiment of the present application is shown. Figure 1 As shown, the multi-layer proxy architecture includes a system proxy 101 and multiple application control agents 102 ( Figure 1 (For example, three application manipulation agents 102 are included).

[0044] in, Figure 1The system agent 101 shown is responsible for one or more of user instruction analysis and reasoning, task identification, task planning, task distribution and task clarification. User instruction analysis and reasoning include using language models such as LLM and MLLM to analyze and reason user instructions; task identification includes analyzing and identifying the user's control intention through user instructions, first memory information, system context, etc., and then determining the corresponding task; task planning includes planning the identified tasks, such as splitting the tasks, determining the application control agent for executing the tasks, etc.; task distribution includes distributing the tasks to the corresponding application control agent 102 for subsequent execution. Among them, the first memory information includes the historical records of the system agent 101 executing historical tasks, for example, the first memory information includes the session context of the system agent 101 executing historical tasks, etc., without limitation.

[0045] As an example, when the system agent 101 determines the task of the user instruction, the system agent 101 can also rewrite the user instruction in combination with the language model and the knowledge base, and determine the corresponding task based on the rewritten user instruction. For example, the system agent 101 can rewrite the user instruction through processes such as system context loading, first memory information retrieval or loading, named entity recognition, and hot word retrieval.

[0046] In some embodiments, during the process of task identification by the system agent 101, if it is necessary to clarify certain questions about the task or supplement certain details about the task, the system agent 101 may also conduct one or more rounds of interaction with the user to clarify the task, and after completing the task clarification, switch back to the system agent to update or supplement the user instructions based on the clarification results and continue with task identification, task planning, task distribution, etc.

[0047] In some embodiments, if the user's control intention involves multiple tasks, the system agent 101 is also used to split the identified task into multiple subtasks, determine the application control agent 102 for executing each subtask, and distribute each subtask to the corresponding application control agent 102 for subsequent execution. Figure 1 Take the example that the system agent 101 splits the identified task into subtask 1, subtask 2 and subtask 3, and distributes them to different application control agents 102. The system agent 101 may distribute subtask 1, subtask 2 and subtask 3 to different application control agents 102 asynchronously, or distribute subtasks one by one and distribute the next subtask after receiving a response about the successful execution of the subtask, which is not limited and depends on the specific situation.

[0048] In some embodiments, when distributing each subtask to the corresponding application control agent 102, the system agent 101 may also instruct each application control agent 102 on the order in which to execute each subtask. For example, each application control agent 102 may need to execute each subtask synchronously, or may execute its own subtask in sequence according to a specified execution order.

[0049] In some embodiments, during the process of the application control agent 102 executing a task (such as a subtask) according to the instructions of the system agent 101, if it is necessary to clarify certain issues about the task or supplement certain details about the task, the system agent 101 may also conduct one or more rounds of interaction with the user according to the request of the application control agent 102 to clarify the task, and after completing the task clarification, instruct the relevant application control agent 102 to update or supplement the subtask according to the clarification result and continue to execute the subtask, and so on, until the execution of the subtask is successfully completed.

[0050] In some embodiments, during one or more rounds of interaction between the system agent 101 and the user to clarify the task, the system agent 101 may also interact with the user according to the task clarification request of the application manipulation agent 102, obtain information input by the user, and analyze the information input by the user to obtain a clarification result. For example, the system agent 101 may analyze the information input by the user for clarifying the question in one or more rounds of interaction to obtain a clarification result.

[0051] In some embodiments, during one or more rounds of interaction between the system agent 101 and the user to clarify the task, the system agent 101 may also instruct the application control agent 102 to perform the clarification task according to the information input by the user, and then obtain the clarification result according to the execution result of the clarification task. For example, the system agent 101 may obtain the clarification result according to the clarification instruction of the user in one or more rounds of interaction, through the process of clarification task planning and clarification task distribution corresponding to the clarification instruction.

[0052] in, Figure 1 The multiple application manipulation agents 102 are used to execute specific subtasks according to the instructions of the system agent 101 .

[0053] Among them, in the embodiments of the present application, the application control agent may include but is not limited to the control agent of any of the following applications: calls, instant messaging, video, audio, shopping, news, information, finance, financial management, weather, search, navigation, mailbox, clock, memo, calendar, map, office, artificial intelligence, etc., without limitation.

[0054] In some embodiments, during the process of the application control agent 102 executing a specific subtask, if the user needs to clarify certain questions about the task or supplement certain details about the task, the application control agent 102 can also request the system agent 101 to interact with the user to clarify the task, and after completing the task clarification, update or supplement the subtask according to the clarification result and continue to execute the subtask, and so on, until the execution of the subtask is successfully completed.

[0055] That is to say, based on the multi-layer proxy architecture provided by the embodiment of the present application, the system agent 101 can perform one or more rounds of interaction with the user according to the situation of the application control agent 102 executing the subtask, until the subtasks of each application control agent 102 can be clearly, accurately and reliably executed. Therefore, the architecture can not only accurately determine the user's operation intention according to the user instruction and the first memory information and then accurately determine the corresponding task, but also flexibly split the task, and distribute the subtasks obtained by splitting to the application control agent 102 to support the reliable execution of each subtask, and can also flexibly perform one or more rounds of interaction with the user according to the specific situation of the subtask execution to clarify certain issues about the task and / or supplement certain details about the task, so as to achieve clear, timely, accurate and reliable execution of each subtask, and finally ensure the timely, accurate and reliable execution of the task, and intelligently respond to different scenarios, including user instructions in simple and complex scenarios, on the premise of simplifying user operations. In particular, in complex scenarios such as scenarios involving multiple cross-application calls, scenarios with complex reasoning processes, and scenarios where problems need to be clarified, relevant tasks can be completed in a timely, accurate and reliable manner to improve the user experience.

[0056] In some embodiments, Figure 1 The multi-layer proxy architecture shown also includes a memory module 103, which is used to maintain memory information (such as first memory information, second memory information, etc.) and system context information. Figure 1 The architecture shown is only an example. In actual applications, the system agent 101 may include a module for maintaining the first memory information, and each application control agent 102 may include a module for maintaining its own second memory information, etc., without limitation. The second memory information of the application control agent 102 includes the history records of the application control agent 102 executing historical tasks, for example, the second memory information of the application control agent 102 includes the session context of the application control agent 102 executing historical tasks, etc., without limitation.

[0057] Exemplarily, the multi-layer agent architecture described in the embodiment of the present application is a multi-agent architecture based on language models such as LLM and MLLM, wherein the agent uses language models such as LLM and MLLM as its core computing engine, including a system agent and a domain agent. The system agent is used to perform user instruction analysis, task identification, task planning, task distribution and task clarification, etc.; the domain agent is used to perform specific subtasks.

[0058] It should be noted that in a multi-agent architecture, there may be multiple domain agents responsible for the execution of different types of subtasks.

[0059] As an example, see Figure 2 , Figure 2 FIG. 1 shows a schematic diagram of a multi-layer agent architecture provided by an embodiment of the present application. Figure 2 As shown, the multi-layer agent architecture may include a system agent (also called system agent) 201, multiple domain agents (also called domain agents) 202 ( Figure 2 The example includes three domain agents 202) and a memory module 203. For the introduction of the system agent 201, please refer to the description of the system agent 101 above, for the introduction of the domain agent 202, please refer to the description of the application control agent 102 above, and for the introduction of the memory module 203, please refer to the description of the memory module 103 above, which will not be repeated here.

[0060] In some embodiments, the task execution method provided in the embodiments of the present application can be applied to the architecture of end-cloud interaction.

[0061] For example, this method can be applied to Figure 3 The structure shown in (a) includes an electronic device 310 and a server 320 (such as a cloud server). Figure 3 The electronic device 310 shown in (a) is responsible for interacting with the user, identifying the user command based on the user's operation, and sending the user command to the server 320. Figure 3 The server 320 shown in (a) is responsible for performing fusion analysis and reasoning based on the user instruction and the first memory information to perform task identification, and perform subsequent task planning, task execution and task clarification, etc. The first memory information includes historical records of historical tasks related to the execution of the user instruction.

[0062] Optionally, in Figure 3After the server 320 and the electronic device 310 shown in (a) complete the execution of the task, the electronic device 310 can also output a response result to the user's instruction to the user, including but not limited to interface display, result reminder, etc.

[0063] Optionally, Figure 3 The server 320 shown in (a) is also used to instruct the electronic device 310 to perform one or more rounds of interaction with the user to clarify the task when it is necessary to clarify certain issues about the task or supplement certain details about the task, and after completing the task clarification, update or supplement the user instructions according to the clarification results and continue to perform task identification, task planning, task execution, etc., or after completing the task clarification, update or supplement the subtask according to the clarification results and continue to execute the subtask. Correspondingly, Figure 3 The electronic device 310 shown in (a) is also used to interact with the user again according to the instruction of the server 320 to clarify the task, and send the clarification result to the server 320, and so on, until the execution of the subtask is successfully completed.

[0064] In some embodiments, the task execution method provided by the embodiments of the present application can be applied to a single device architecture.

[0065] For example, this method can be applied to Figure 3 In the electronic device 310 shown in (b), the electronic device 310 is specifically used to interact with the user and perform user instruction recognition, task recognition, task planning, task execution and task clarification based on user operations.

[0066] Optionally, in Figure 3 After the electronic device 310 shown in (b) completes the execution of the task, the electronic device 310 can also output a response result to the user's instruction to the user, including but not limited to interface display, result reminder, etc.

[0067] Optionally, Figure 3 The electronic device 310 shown in (b) is also used to perform one or more rounds of interaction with the user to clarify the task when it is necessary to clarify certain issues about the task or supplement certain details about the task, and after completing the task clarification, update or supplement the user instructions according to the clarification results and continue to perform task identification, task planning, task execution, etc., or after completing the task clarification, update or supplement the subtask according to the clarification results and continue to execute the subtask, and so on, until the execution of the subtask is successfully completed.

[0068] There is no specific limitation on the specific architecture of the task execution method provided in the embodiments of the present application, and it may depend on the specific device type, device function, device computing capability, application scenario, etc.

[0069] As an example, see Figure 4, Figure 4 Respectively Figure 3 (a) and Figure 3 Taking the architecture shown in (b) as an example, a structural schematic diagram of two system architectures provided in an embodiment of the present application is shown.

[0070] in, Figure 4 (a) in FIG. 1 shows a schematic diagram of the structure of a terminal-cloud collaborative architecture. Figure 4 As shown in (a), the electronic device 310 may include a human-computer interaction module 310-1 and a communication module 310-2; the server 320 may include a communication module 320-1, a system agent module 320-2 and a plurality of application control agent modules 320-3 ( Figure 4 Take the example of including three application manipulation agent modules 320 - 3 ).

[0071] Figure 4 The human-computer interaction module 310-1 of the electronic device 310 shown in (a) is used to interact with the user, such as receiving user input, receiving user touch operations, receiving user voice instructions and other user operations, and identifying user instructions based on user operations. The specific method for the user to interact with the electronic device 310 is not limited in the embodiment of the present application.

[0072] Figure 4 The communication module 310-2 of the electronic device 310 shown in (a) is used to send the recognized user instruction to the server 320. In some embodiments, the communication module 310-2 of the electronic device 310 is also used to perform one or more rounds of interaction with the server 320 to clarify the task and send the clarification result to the server 320 to clarify certain issues about the task and / or supplement certain details about the task.

[0073] In some embodiments, the communication module 310-2 of the electronic device 310 is also used to send the system context obtained by the intelligent assistant to the system agent. The system context of the intelligent assistant may include but is not limited to information of the electronic device (such as device identification, device name, login user name, etc.), running A applications (such as foreground applications, background applications, etc.), location information (such as location coordinates collected by the global positioning system (GPS), etc.). The intelligent assistant sends the system context for the system agent to refer to when determining the task corresponding to the user instruction.

[0074] Figure 4The communication module 320-1 of the server 320 shown in (a) is used to receive user instructions from the electronic device 310. In some embodiments, the communication module 320-1 of the server 320 is also used to perform one or more rounds of interaction with the electronic device 310 to clarify the task, clarify certain issues about the task and / or supplement certain details about the task.

[0075] Figure 4 The system agent module 320-2 of the server 320 shown in (a) is responsible for analyzing user instructions to perform task identification, task planning, and task distribution. In some embodiments, when the system agent module 320-2 or the application control agent module 320-3 of the server 320 needs to clarify certain questions about the task or supplement certain details about the task, the system agent module 320-2 of the server 320 is also used to trigger one or more rounds of interaction with the user through the electronic device 310 to clarify the task.

[0076] Figure 4 The multiple application control agent modules 320-3 of the server 320 shown in (a) are used to perform respective subtasks according to the instructions of the system agent module 320-2 of the server 320. In some embodiments, when it is necessary to clarify certain issues about the task or supplement certain details about the task, the multiple application control agent modules 320-3 of the server 320 are also used to request the system agent module 320-2 of the server 320 to clarify the task.

[0077] Figure 4 (b) in FIG. 1 shows a schematic diagram of a single-device architecture. Figure 4 As shown in (b), the electronic device 310 may include a human-computer interaction module 310-1, a system agent module 310-3 and an application control agent module 310-4.

[0078] in, Figure 4 The human-computer interaction module 310-1 of the electronic device 310 shown in (b) is used to interact with the user, such as receiving user input, receiving user touch operations, receiving user voice instructions and other user operations, and is used to identify user instructions based on user operations. The specific way in which the user interacts with the electronic device 310 is not limited in the embodiment of the present application.

[0079] Figure 4 The system agent module 310 - 3 of the electronic device 310 shown in (b) is responsible for user instruction analysis to perform task identification, task planning and task distribution.

[0080] Figure 4The multiple application control agent modules 310 - 4 of the electronic device 310 shown in (b) are used to execute respective subtasks according to the instructions of the system agent module 310 - 3 of the electronic device 310 .

[0081] In some embodiments, when it is necessary to clarify certain issues about the task or supplement certain details about the task, the application control agent module 310-4 of the electronic device 310 is also used to request the system agent module 310-3 of the electronic device 310 to perform task clarification to clarify certain issues about the task and / or supplement certain details about the task. Correspondingly, the system agent module 310-3 of the electronic device 310 is also used to trigger one or more rounds of interaction with the user to perform task clarification.

[0082] Need to explain, Figure 4 (a) and Figure 4 (b) is only used as two structural examples of system architectures for executing tasks according to user instructions. The embodiments of the present application do not limit the specific architecture and the specific functional module settings in the electronic device or server. For example Figure 4 (a) or Figure 4 The multiple modules in (b) may be integrated together or the function of a module may be completed by multiple sub-modules, depending on the specific situation.

[0083] Among them, the electronic device described in the embodiment of the present application may include but is not limited to any electronic device with image display, such as the electronic device may include but is not limited to smart phones, netbooks, tablet computers, smart drawing boards, handwriting boards, smart watches, smart bracelets, phone watches, smart glasses, smart cameras, PDAs, car computers, personal computers (personal computers, PCs), personal digital assistants (personal digital assistants, PDAs), portable multimedia players (portable multimedia players, PMPs), augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) devices, smart TVs, projection devices, or somatosensory game consoles in human-computer interaction scenarios, etc. Alternatively, the electronic device may also be other types or structures of electronic devices with image display, which are not limited in the present application.

[0084] As an example, see Figure 5 , Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0085] like Figure 5As shown, the electronic device may include a processor 510, a memory (including an external memory interface 520 and an internal memory 521), a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna 1, an antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, an earphone interface 570D, a sensor module 580, a button 590, a motor 591, an indicator light 592, a camera 593, a display screen 594, etc.

[0086] Among them, the sensor module 580 may include but is not limited to one or more of the following: touch sensor, pressure sensor, temperature sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, ambient light sensor, bone conduction sensor, etc.

[0087] The processor 510 may include one or more processing units. For example, the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a flight controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0088] The processor 510 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. The memory may store instructions or data that the processor 510 has just used or cyclically used. If the processor 510 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.

[0089] In some embodiments of the present application, the processor 510 can send the application interface, memory image, etc. to the display screen 594 for display through an interface, such as a DSI interface.

[0090] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, modem processor and baseband processor.

[0091] In some embodiments of the present application, the electronic device can communicate with other devices through antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, modem processor and baseband processor, such as sending user instructions, information for clarifying issues, clarification instructions, clarification results, etc. to the server; receiving execution engine, tool or interface call instructions from the server, etc.

[0092] The electronic device implements the display function through a GPU, a display screen 594, and an AP. The GPU is a microprocessor for image processing, connected to the display screen 594 and the AP. The GPU is used to perform mathematical and geometric calculations for drawing, rendering, or compositing graphics. The processor 510 may include one or more GPUs that execute program instructions to generate or change display information.

[0093] In some embodiments of the present application, the electronic device can implement related processing of the application interface and / or the memory image through AP, GPU, etc., and display the application interface and / or the memory image through the display screen 594.

[0094] The electronic device can realize the shooting function through ISP, camera 593, video codec, GPU, display screen 594 and AP.

[0095] The external memory interface 520 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 510 through the external memory interface 520 to implement a data storage function.

[0096] The internal memory 521 can be used to store computer executable program codes. Exemplarily, the computer program may include an operating system program and an application program. Among them, the executable program code includes instructions. The processor 510 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 521. The internal memory 521 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application program required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. In addition, the internal memory 521 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 510 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 521, and / or the instructions stored in the memory provided in the processor.

[0097] The electronic device can implement audio functions such as music playing and recording through the audio module 570, the speaker 570A, the receiver 570B, the microphone 570C and the AP.

[0098] about Figure 5 For the introduction of the charging management module 540, power management module 541, battery 542, motor 591, indicator light 592, etc., reference can be made to conventional technologies and will not be elaborated here.

[0099] It is understandable that this application Figure 5 The illustrated structure does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0100] The following is a detailed introduction to the task execution method provided in the embodiment of the present application.

[0101] The multi-layer proxy architecture provided in the embodiments of the present application can promptly, accurately and reliably respond to user instructions in various scenarios, including user instructions in various complex scenarios. Taking the proxy architecture including the system proxy and the application control proxy as an example, the multi-layer proxy architecture can perform the tasks in the following scenarios 1-3:

[0102] Scenario 1: Single-task scenario

[0103] Single-task scenarios are scenarios where, after the system agent determines the task corresponding to the user instruction based on the user instruction and the historical records of the execution of related historical tasks recorded by the system agent (i.e., the first memory information), there is no need to split the tasks, and the response to the user instruction can be completed after one task execution. In this case, after obtaining the user instruction corresponding to the user operation and determining the task corresponding to the user instruction based on the user instruction and the first memory information, the system agent can determine the application control agent for executing the task and instruct the application control agent to execute the corresponding task in response to the user instruction.

[0104] For example, a single-task scenario is a task execution (Action) instruction scenario.

[0105] For example, if the user instruction is "set an alarm at 8 o'clock", the system agent determines that the task corresponding to the user instruction is "set an alarm at 8 o'clock tomorrow in the clock application" according to the user instruction and the first memory information. For example, the system agent can instruct the clock control agent to execute the task "set an alarm at 8 o'clock tomorrow in the clock application" through the following instruction 1:

[0106] Instruction 1: CreateAlarm(time = "tomorrow at 8 o'clock"), LastAction: True.

[0107] Among them, in instruction 1, "CreateAlarm(time="tomorrow at 8 o'clock")" is used to instruct to set an alarm at 8 o'clock tomorrow, and "LastAction: True" indicates that the response to the user instruction can be ended if the task is successfully executed.

[0108] Or, for example, if the user instruction is "buy a serving of celery dumplings and send it to address C", the system agent determines the task corresponding to the user instruction as "buy a serving of celery dumplings at store B in application A and send it to address C" based on the user instruction and the first memory information (such as the record that the user often places orders at store B in application A). The execution of this task can end the response to the user instruction.

[0109] Scenario 2: Multitasking scenario

[0110] As an example, the multi-task scenario includes a multi-task scenario planned once and a multi-task scenario planned multiple times.

[0111] Among them, in a multi-task scenario planned at one time, after determining the task corresponding to the user instruction according to the user instruction and the first memory information, the system agent splits the task into multiple subtasks and distributes them to the corresponding application control agent. In this case, the system agent can obtain the user instruction corresponding to the user operation, determine the task corresponding to the user instruction according to the user instruction and the first memory information, split the task corresponding to the user instruction into multiple subtasks, and then distribute the subtask to the determined application control agent for subtask execution in response to the user instruction. Among them, when the application control agent executes its own subtask, the application control agent can automatically switch tasks according to the instructions of the system agent.

[0112] For example, a multi-task scenario planned at one time is such as a ReAct instruction scenario.

[0113] For example, taking the user instruction of "check Zhang San's phone number and then send it to Li Si through instant messaging application X", the system agent determines, based on the user instruction and the first memory information, that the task corresponding to the user instruction is "find Zhang San's phone number in the address book application and then send it to contact Li Si through instant messaging application X"; then, the system agent splits the task into a first subtask and a second subtask through task splitting, the first subtask being "find Zhang San's phone number in the address book application", and the second subtask being "send Zhang San's phone number to contact Li Si through instant messaging application X".

[0114] For example, the process of task planning by the system agent might be as follows:

[0115] plugins[search for Zhang San’s phone number in the address book app, and send Zhang San’s phone number to contact Li Si via instant messaging app X].

[0116] For example, the system agent can instruct the address book manipulation agent to perform the first subtask "search for Zhang San's phone number in the address book application" through the following instruction 2, and instruct the address book manipulation agent to perform the second subtask "send Zhang San's phone number to contact Li Si through instant messaging application X" through the following instruction 3:

[0117] Instruction 2: CheckPhone(contract = "Zhang San"), LastAction: False;

[0118] Instruction 3: SendMail(App = "Instant Messaging Application X", Receiver = "Li Si"), LastAction: True.

[0119] Among them, in instruction 2, "CheckPhone(contract="Zhang San")" is used to instruct to find Zhang San's phone number in the address book application, "SendMail(App="Instant Messaging Application X", Receiver="Li Si")" is used to instruct to send Zhang San's phone number to the contact Li Si through the instant messaging application X, "LastAction: False" indicates that other tasks need to be performed after the successful execution of this task, and "LastAction: True" indicates that the response to the user instruction can be ended after the successful execution of this task.

[0120] Or, for example, the user instruction is "buy another serving of dumplings", the system agent determines, based on the user instruction and the first memory information, that the task corresponding to the user instruction is "buy a serving of celery-stuffed dumplings at store B in application A and have it delivered to address C"; then, the system agent splits the task into a first subtask, a second subtask, and a third subtask through task splitting, the first subtask being "open application A", the second subtask being "search for store B in application A", and the third subtask being "place an order for a serving of celery-stuffed dumplings at store B and have it delivered to address C".

[0121] It should be noted that the above-mentioned scenarios and task examples are only for illustration. In actual application, user instructions, task planning and specific execution details of each subtask are not limited and can be determined according to specific circumstances.

[0122] Among them, in a multi-task scenario that has been planned multiple times, if after determining the task corresponding to the user instruction based on the user instruction and the first memory information, the system agent finds that the execution of the task cannot successfully respond to the user instruction or the execution of the task cannot accurately respond to the user instruction, the system agent will re-plan the task based on the above task and respond to the user instruction by executing the first subtask and the second subtask.

[0123] For example, taking the user instruction of "send Zhang San's phone number to Li Si via communication application X", the system agent determines that the task corresponding to the user instruction is "send Zhang San's phone number to Li Si via communication application X" according to the user instruction and the first memory information; however, since the system agent does not store Zhang San's phone number, the system agent will further perform task planning based on the task through analysis and reasoning to obtain the first subtask and the second subtask. The first subtask is "obtain Zhang San's phone number" and the second subtask is "send Zhang San's phone number to Li Si via communication application X".

[0124] For example, the process of task planning by the system agent might be as follows:

[0125] plugins[Send Zhang San's phone number to Li Si through communication application X]->plan(subtask1: Get Zhang San's phone number; subtask2: Send the obtained phone number to Li Si through communication application X).

[0126] Among them, in the above example, "subtask1: obtain Zhang San's phone number" is the first subtask mentioned above, and "subtask2: send the obtained phone number to Li Si through communication application X" is the second subtask mentioned above.

[0127] For example, the system agent can instruct the address book manipulation agent to perform the first subtask "obtain Zhang San's phone number" through the following instruction 4, and instruct the address book manipulation agent to perform the second subtask "send Zhang San's phone number to contact Li Si via instant messaging application X" through the following instruction 5:

[0128] Instruction 4: CheckPhone(contract = "Zhang San"), LastAction: False;

[0129] Instruction 5: SendMail(App = "Instant Messaging Application X", Receiver = "Li Si"), LastAction: True.

[0130] Among them, in instruction 4, "CheckPhone(contract="Zhang San")" is used to instruct to obtain Zhang San's phone number, and "LastAction: False" indicates that other tasks need to be performed after the successful execution of this task; in instruction 5, "SendMail(App="Instant Messaging Application X", Receiver="Li Si")" indicates that Zhang San's phone number is sent to the contact Li Si via the instant messaging application X, and "LastAction: True" indicates that the response to the user's instruction can be ended after the successful execution of this task.

[0131] Or, for example, assuming that the user instruction is "buy another serving of dumplings", the system agent determines that the task corresponding to the user instruction is "buy a serving of dumplings at store B in application A" based on the user instruction and the first memory information; however, since the system agent does not store the user's favorite dumplings with a certain filling, nor does it store the user's delivery address, the system agent will further perform task planning based on the task through analysis and reasoning to obtain the first subtask and the second subtask. The first subtask is "obtaining the user's order information from store B in application A", and the second subtask is "purchasing a serving of dumplings at store B in application A based on the obtained order information".

[0132] Scenario 3: Task clarification scenario.

[0133] As an example, a task clarification scenario is a scenario where the system agent needs to clarify certain issues about the task or supplement certain details about the task when determining the task corresponding to the user instruction. In this scenario, the system agent can trigger task clarification to clarify certain issues about the task or supplement certain details about the task through one or more rounds of interaction with the user.

[0134] For example, taking the user instruction "I want to buy a ticket to Beijing", when the system agent determines the task corresponding to the user instruction, since it is not sure whether the user wants to buy an airplane ticket, a train ticket or another type of ticket, the system agent can trigger task clarification to determine which type of ticket the user wants to buy through one or more rounds of interaction with the user.

[0135] Or, for example, taking the user instruction as "I want to buy a plane ticket to Beijing", when the system agent is determining the task corresponding to the user instruction, since it is not sure which day the user wants to buy the ticket, the system agent can trigger task clarification to determine which day the user wants to buy the ticket through one or more rounds of interaction with the user.

[0136] As an example, a task clarification scenario is a scenario where the application control agent needs to clarify certain issues about the task or supplement certain details about the task during the execution of the task. In this scenario, the application control agent can request the system agent to trigger task clarification to clarify certain issues about the task or supplement certain details about the task through one or more rounds of interaction with the user. Exemplarily, the system agent can have one or more rounds of interaction with the user through an intelligent assistant to clarify certain issues about the task or supplement certain details about the task.

[0137] For example, if the user instruction is "buy another serving of dumplings", the system agent determines that the task corresponding to the user instruction is "buy a serving of dumplings at store B in application A" based on the user instruction and the first memory information, and the first subtask, the second subtask and the third subtask obtained by splitting are "open application A", "search for store B in application A", and "place an order for a serving of dumplings at store B and have it delivered to address C" respectively. After the application A controls the agent to open application A and search for store B in application A, during the execution of the third subtask "place an order for a serving of dumplings at store B and have it delivered to address C", since the application A controls the agent not sure what kind of dumplings the user wants to buy, the application A controls the agent requesting the system agent to trigger task clarification to determine what kind of dumplings the user wants to buy through one or more rounds of interaction with the user.

[0138] Or, for example, taking the user instruction as "set an alarm at 8 o'clock", and the system agent determines, based on the user instruction and the first memory information, that the task corresponding to the user instruction is "set an alarm at 8 o'clock in the clock application", when the application control agent of the clock application opens the clock application to set the alarm, since the application control agent of the clock application is not sure whether the user wants to set the alarm at 8 am or 8 pm, the application control agent of the clock application requests the system agent to trigger task clarification, so as to determine, through one or more rounds of interaction with the user, whether the user sets the alarm at 8 am or 8 pm.

[0139] As a possible implementation manner, during one or more rounds of interaction with the user, the system agent may analyze, through an intelligent assistant, information input by the user in the one or more rounds of interaction for clarifying a question to obtain a clarification result.

[0140] For example, in the process of executing the third subtask "order a serving of dumplings in store B and deliver it to address C" by using application A control agent, since application A control agent is not sure what filling of dumplings the user wants to buy, application A control agent requests system agent to trigger task clarification to determine what filling of dumplings the user wants to buy through one or more rounds of interaction with the user. For example, the system agent can ask the user through the intelligent assistant "What filling of dumplings do you want?" In response to the user's answer "celery filling", the system agent can determine that the user wants to buy dumplings with celery filling and send the clarification result to the application A control agent. After receiving the clarification result, the application A control agent updates or supplements the third subtask to "order a serving of dumplings with celery filling in store B and deliver it to address C" and continues to execute the third subtask.

[0141] As a possible implementation method, during one or more rounds of interaction with the user, the system agent can obtain clarification results through an intelligent assistant according to the user's clarification instructions in one or more rounds of interaction, through clarification task planning and clarification task distribution corresponding to the clarification instructions.

[0142] For example, in the process of executing the third subtask "order a dumpling in store B and deliver it to address C" by application A control agent, since application A control agent is not sure what kind of dumplings the user wants to buy, application A control agent requests system agent to trigger task clarification to determine what kind of dumplings the user wants to buy through one or more rounds of interaction with the user. For example, the system agent can ask the user "What kind of dumplings do you want?" through the intelligent assistant. Assuming that the user indicates "call dad to ask what kind of dumplings are needed", the system agent can determine the clarification task as "call dad to ask what kind of dumplings are needed" according to the clarification instruction, and split the clarification task into the first clarification subtask "get dad's phone number" and the second clarification subtask "call dad to ask what kind of dumplings are needed" and distribute them to the corresponding application control agent. Assuming that the system agent finally determines that the user wants to buy dumplings with celery filling by executing the first clarification subtask and the second clarification subtask, the system agent can determine that the user wants to buy dumplings with celery filling and send the clarification result to the application A control agent. After receiving the clarification result, application A controls the agent to update or supplement the third subtask to "place an order for a celery-stuffed dumpling at store B and deliver it to address C" and continue to execute the third subtask.

[0143] In some embodiments, the above scenario 3 may overlap with scenario 1 or scenario 2. For example, in a single-task scenario (i.e., scenario 1), a task clarification scenario (i.e., scenario 3) may also be involved; in a multi-task scenario (i.e., scenario 2), a task clarification scenario (i.e., scenario 3) may also be involved.

[0144] As an example, see Fig. 6A , Fig. 6A Taking a single task scenario as an example, a task execution process interaction diagram provided by an embodiment of the present application is shown. Fig. 6A As shown, the intelligent assistant can rely on a multi-layer agent architecture including a system agent and at least one application control agent to execute tasks based on user instructions through S601A-S605A, wherein the relevant functions of the intelligent assistant may be completed by the electronic device, or by the server, or may be completed by the electronic device and the server in a division of labor and collaboration, without specific limitation.

[0145] S601A: The system agent obtains a user instruction corresponding to the user operation.

[0146] As an example, the user operation is an operation of the user on an intelligent assistant in the electronic device. For example, the intelligent assistant may include but is not limited to a voice assistant.

[0147] Among them, the user operation may include but is not limited to at least one of the following: voice instruction operation, gesture operation, body language operation, touch operation, information input operation (such as text input operation, etc.), etc. For example, taking the intelligent assistant as a voice assistant as an example, the user operation may be the user's voice instruction in the voice assistant, etc. The specific operation form of the user operation is not limited in the embodiments of the present application.

[0148] As an example, the intelligent assistant is deployed on an electronic device.

[0149] As a possible implementation, after an electronic device receives a user operation, such as an operation of a user on an intelligent assistant in an electronic device, the electronic device may first determine the input event corresponding to the user operation, and then analyze and infer based on the input event to identify the corresponding user instruction. For example, after an electronic device receives a user operation, the electronic device may first convert the user operation into a specific input event, and then distribute the input event to the corresponding intelligent assistant window, and the intelligent assistant may analyze and infer based on the input event to identify the corresponding user instruction.

[0150] As an example, taking the case where the user operation is to enter the voice command "order me another serving of dumplings" in the smart assistant, the electronic device can first convert the voice command into a specific input event, and then distribute the input event to its corresponding smart assistant window. The voice assistant can analyze and infer based on the input event to identify its corresponding user command as "buy another serving of dumplings."

[0151] As a possible implementation, after completing the determination of the user instruction, the intelligent assistant can call the system agent to execute the subsequent S602A.

[0152] As an example, the system agent is deployed on an electronic device, and the intelligent assistant can call the relevant interface to send the determined user instructions to the system agent.

[0153] As an example, the system agent is deployed on a server, and the intelligent assistant can send the determined user instructions to the system agent through a communication connection with the server.

[0154] In some embodiments, in addition to sending user instructions to the system agent, the intelligent assistant can also send the system context obtained by the intelligent assistant to the system agent. The system context of the intelligent assistant may include but is not limited to information about the electronic device (such as device identification, device name, login user name, etc.), running A applications (such as foreground applications, background applications, etc.), and location information (such as location coordinates collected by GPS, etc.). The intelligent assistant sends the system context for the system agent to refer to when determining the task corresponding to the user instruction.

[0155] In some embodiments, the system agent can also rewrite the user instruction in combination with the language model and the knowledge base, and determine the corresponding task based on the rewritten user instruction. For example, the system agent can rewrite the user instruction through processes such as system context loading, first memory information retrieval or loading, named entity recognition, and hot word retrieval.

[0156] S602A: The system agent determines the task corresponding to the user instruction according to the user instruction and the first memory information.

[0157] As a possible implementation, the system agent can use the user instruction and the first memory information as inputs of the language model, and perform fusion analysis and reasoning on the user instruction and the first memory information through the language model to determine the task corresponding to the user instruction. Exemplarily, the language model may include but is not limited to at least one of the following: LLM, MLLM, etc.

[0158] As an example, the first memory information includes a historical record of historical tasks related to the task performed by the system agent. For example, the first memory information includes a historical record of historical tasks during the historical interaction between the system agent and the application control agent, and the historical record includes a historical record of historical tasks related to the task corresponding to the above user instruction during the historical interaction between the system agent and the application control agent. Exemplarily, the historical record may include but is not limited to at least one of the following: the response of the application control agent to the historical task, the judgment and rewriting of the system agent in the process of executing the historical task, the actions and results performed by the system agent in the process of executing the historical task, etc.

[0159] As an example, the system agent and the application control agent may interact with each other in accordance with a prescribed communication specification.

[0160] For example, the communication specification specifies the specifications of the session contexts of the following historical conversations shown in Table 1 when the system agent and the application control agent interact:

[0161] Table 1

[0162]

[0163]

[0164] In some embodiments, as shown in Table 2 below, the communication specification followed by the system agent and the application control agent not only specifies the specifications of the session context of each historical conversation shown in Table 1, but also specifies the specifications for the system agent and the application control agent to generate their own memory information:

[0165] Table 2

[0166]

[0167] The multi-layer proxy architecture provided in the embodiment of the present application interacts with each other by following the prescribed communication specifications, which can not only reduce the analysis and reasoning costs of each layer, but also ensure the execution efficiency, reliability and accuracy of tasks when the multi-layer proxy architecture is applied to various complex scenarios.

[0168] As an example, take the user instruction "buy another serving of dumplings" and the first memory information including the session context of the system agent and the application control agent interactively executing the historical task "buy a serving of dumplings with celery filling in store B in application A and deliver it to address C". The system agent can perform fusion analysis and reasoning based on the user instruction and the first memory information to determine that the task corresponding to the user instruction is "buy a serving of dumplings with celery filling in store B in application A and deliver it to address C". It can be seen that in this example, the user instruction "buy another serving of dumplings" does not indicate which application or store to buy dumplings, nor does it indicate what kind of dumplings to buy, and does not indicate the delivery address. Through the fusion analysis and reasoning of the user instruction and the first memory information, the system agent can intelligently determine the target application, target store, target dish and delivery address, and can timely, accurately and reliably identify the task corresponding to the user instruction without the user inputting overly complex instructions.

[0169] In some embodiments, if the system agent receives the system context, the system agent may also refer to the system context when determining the task corresponding to the user instruction. For example, the system agent may use the prompt information determined according to the user instruction, the first memory information and the system context as the input of the language model, and perform fusion analysis and reasoning on the first prompt information through the language model to determine the task corresponding to the user instruction.

[0170] S603A: The system agent determines an application control agent for executing the task corresponding to the user instruction.

[0171] As a possible implementation manner, the system agent may determine, based on the application involved in the task corresponding to the determined user instruction, that the task is to be executed by an application control agent corresponding to the application.

[0172] For example, taking the task corresponding to the user instruction as "setting an alarm at 8 o'clock" as an example, the system agent determines that the application control agent of the clock application will execute the task based on application A involved in the task.

[0173] S604A: The system agent instructs the application control agent to perform the above tasks.

[0174] As an example, the system agent may instruct the application control agent to perform the above tasks through an interface between the system agent and the application control agent.

[0175] S605A: The application control agent executes the task according to the instruction of the system agent.

[0176] As a possible example, after receiving the instruction from the system agent, the application control agent calls the relevant execution engine, tool or interface to perform the corresponding task. For example, taking the task as an operation in the application, the application control agent can call the execution engine, tool or interface based on the UI understanding to perform relevant control of the application.

[0177] For example, taking the task corresponding to the user instruction as "set an alarm for 8 o'clock tomorrow", after receiving instructions from the system agent, the application control agent of the clock application can call the execution engine (such as an intelligent assistant), tool or interface based on UI understanding to start the clock and set an alarm for 8 o'clock tomorrow in the clock application.

[0178] In some embodiments, the application control agent can perform the above task according to the task corresponding to the user instruction and the second memory information of the application control agent, and the second memory information includes the historical record of the application control agent performing historical tasks, such as the historical record of the application control agent performing historical tasks related to the task corresponding to the user instruction. For example, in the process of the application control agent performing tasks according to the instructions of the system agent, the application control agent can also assist in task supplementation based on the second memory information and perform the task according to the supplemented task.

[0179] In some embodiments, when the application control agent performs a task according to the instruction of the system agent, the application control agent can also assist in task supplementation based on the system context. Based on this, the application control agent performs the task according to the supplemented subtask.

[0180] As an example, the application control agent is deployed on an electronic device, and the application control agent can call a related execution engine, tool or interface of the electronic device to perform a corresponding task.

[0181] As an example, the application control agent is deployed on a server, and the application control agent can call the communication connection between the server and the electronic device, and send the calling instruction of the execution engine, tool or interface to the electronic device; correspondingly, the electronic device calls the relevant execution engine, tool or interface to perform the corresponding task according to the calling instruction of the application control agent.

[0182] As an example, see Figure 6B , Figure 6B A task execution process interaction diagram provided by an embodiment of the present application is shown in a multi-task scenario. Figure 6B As shown, the intelligent assistant can rely on a system agent and at least one application control agent ( Figure 6BA multi-layer agent architecture (taking the first application control agent, the second application control agent and the third application control agent as an example) executes tasks based on user instructions through S601B-S605B, wherein the relevant functions of the intelligent assistant may be completed by the electronic device, or by the server, or may be completed by the electronic device and the server through division of labor and cooperation, without specific limitation.

[0183] S601B: The system agent obtains the user instruction corresponding to the user operation.

[0184] For a detailed introduction to S601B, please refer to the above description of S601A, which will not be repeated here.

[0185] S602B: The system agent determines the task corresponding to the user instruction according to the user instruction and the first memory information.

[0186] For a detailed description of S602B, please refer to the description of S602A above, which will not be repeated here.

[0187] S603B: The system agent splits the task corresponding to the user instruction into multiple subtasks, and determines one or more application control agents for executing the multiple subtasks respectively.

[0188] The system agent may split the task corresponding to the user instruction into at least two subtasks. For example, the system agent may split the task corresponding to the user instruction into a first subtask and a second subtask. For another example, the system agent may split the task corresponding to the user instruction into a first subtask, a second subtask, and a third subtask.

[0189] As a possible implementation method, the system agent can select one or more application control agents capable of executing each subtask from multiple application control agents according to the subtasks obtained by splitting the task corresponding to the user instruction. Different subtasks may be executed by different application control agents or by the same application control agent, depending on the specific subtask content and without limitation.

[0190] As a possible example, the system agent may split the task corresponding to the user instruction into multiple subtasks that need to be executed in sequence according to a certain execution order.

[0191] For example, taking the example of the system agent splitting the task corresponding to the user instruction into the first subtask, the second subtask and the third subtask, the execution order of the first subtask, the second subtask and the third subtask may be: first subtask → second subtask → third subtask, that is, when executing multiple subtasks, the subtasks need to be executed in the order of first executing the first subtask, then executing the second subtask, and finally executing the third subtask.

[0192] For example, if the task corresponding to the user instruction is "buy a set of celery dumplings at store B in application A and deliver them to address C", the system agent can analyze and split the task into the following three subtasks:

[0193] First subtask: open application A;

[0194] Second subtask: Search for store B in application A;

[0195] The third subtask: place an order for a set of celery dumplings at store B and have it delivered to address C.

[0196] For example, if the task corresponding to the user instruction is "buy a celery dumpling at store B in application A and deliver it to address C", the system agent can analyze and split the task into the following three subtasks:

[0197] First subtask: open application A;

[0198] Second subtask: Get the historical orders of "buy a set of celery dumplings at store B in application A and deliver to address C";

[0199] The third subtask: Go to the celery-stuffed dumplings product page through the historical orders and place an order for a celery-stuffed dumpling to be delivered to address C.

[0200] Among them, in the example where the tasks corresponding to the above two user instructions are "buy a portion of celery dumplings at store B in application A and deliver them to address C", the first subtask, the second subtask and the third subtask need to be performed in the order of first executing the first subtask, then executing the second subtask, and finally executing the third subtask.

[0201] It should be noted that the above examples of splitting tasks corresponding to user instructions are only two examples. In actual applications, how to split tasks depends on the specific scenarios and specific tasks corresponding to user instructions, and there is no specific limitation.

[0202] In addition, in actual applications, the multiple subtasks obtained by splitting the task corresponding to the user instruction by the system agent may not need to be executed in a certain execution order, depending on the specific situation.

[0203] For example, taking the example of the system agent splitting the task corresponding to the user instruction into the first subtask, the second subtask and the third subtask, there is no regulation on the execution order of the first subtask, the second subtask and the third subtask, that is, when executing multiple subtasks, the multiple subtasks can be performed synchronously and independently.

[0204] S604B: The system agent distributes the multiple subtasks to the determined one or more application control agents.

[0205] As a possible implementation manner, the system agent may distribute multiple subtasks to different application control agents respectively.

[0206] For example, taking the case where the multiple subtasks include a first subtask, a second subtask and a third subtask, the system agent may distribute the first subtask to the first application control agent, the second subtask to the second application control agent, and the third subtask to the third application control agent.

[0207] As a possible implementation manner, the system agent may distribute at least two of the multiple subtasks to the same application control agent.

[0208] Exemplarily, taking the case where the multiple subtasks include a first subtask, a second subtask, and a third subtask, the system agent may distribute the first subtask and the second subtask to the first application control agent, and distribute the third subtask to the third application control agent.

[0209] As a possible implementation manner, the system agent may synchronously distribute multiple subtasks to one or more determined application control agents.

[0210] For example, in the case where multiple subtasks do not need to be executed in a certain execution order, the system agent may synchronously distribute the multiple subtasks to one or more determined application control agents.

[0211] For another example, when multiple subtasks need to be executed in a certain order, the system agent can instruct the one or more application control agents to execute the specific order of the multiple subtasks when distributing the multiple subtasks to the one or more application control agents.

[0212] As another possible implementation manner, the system agent may synchronously distribute multiple subtasks to one or more determined application control agents in a certain order.

[0213] For example, in the case where multiple subtasks need to be executed in sequence according to a certain execution order, the system agent may distribute the multiple subtasks to one or more determined application control agents in a certain order.

[0214] Exemplarily, taking multiple subtasks including a first subtask, a second subtask and a third subtask, and the execution order of the first subtask, the second subtask and the third subtask is: first subtask → second subtask → third subtask as an example, the system agent can first distribute the first subtask to the first application control agent, send the second subtask to the second application control agent after the first application control agent completes the execution of the first subtask, and send the third subtask to the third application control agent after the second application control agent completes the execution of the second subtask.

[0215] There is no limitation on the specific way and method of the system agent distributing multiple subtasks, which may be determined according to the specific scenario and specific tasks.

[0216] S605B: The one or more application control agents execute corresponding subtasks.

[0217] As a possible example, after receiving the subtask distributed by the system agent, the one or more application control agents call the relevant execution engine, tool or interface to execute the corresponding subtask. For example, taking the subtask as an operation in the application as an example, the application control agent can call the execution engine, tool or interface based on the UI understanding to perform relevant control of the application.

[0218] For example, taking the first application control agent used to execute the first subtask, the second application control agent used to execute the second subtask, and the third application control agent used to execute the third subtask as an example, the first application control agent calls the first execution engine to execute the first subtask, the second application control agent calls the second execution engine to execute the second subtask, and the third application control agent calls the third execution engine to execute the third subtask.

[0219] As a possible example, when the system agent distributes multiple subtasks to one or more determined application control agents, it instructs the one or more application control agents to execute the multiple subtasks in a specific order. In this case, the one or more application control agents execute the corresponding subtasks in sequence as specified by the system agent.

[0220] For example, taking the first application control agent being used to execute the first subtask, the second application control agent being used to execute the second subtask, and the third application control agent being used to execute the third subtask, and the execution order of the first subtask, the second subtask and the third subtask being: first subtask → second subtask → third subtask, S605B includes: the first application control agent calling the first execution engine to execute the first subtask; after the first subtask is completed, the second application control agent calling the second execution engine to execute the second subtask; after the second subtask is completed, the third application control agent calling the third execution engine to execute the third subtask.

[0221] For another example, taking the first application control agent being used to execute the first subtask and the second subtask, and the third application control agent being used to execute the third subtask, and the execution order of the first subtask, the second subtask and the third subtask being: first subtask → second subtask → third subtask, S605B includes: the first application control agent calling the first execution engine to execute the first subtask, and after the first subtask is completed, calling the second execution engine to execute the second subtask; after the second subtask is completed, the third application control agent calling the third execution engine to execute the third subtask.

[0222] In the above examples, any two of the first execution engine, the second execution engine, and the third execution engine may be the same execution engine or different engines, without specific limitation.

[0223] In some embodiments, in the process of each application control agent executing its own subtask according to the instructions of the system agent, each application control agent can also assist in supplementing the subtask based on its own historical records of related historical tasks (i.e., second memory information), and each application control agent executes the task according to its own supplemented subtask.

[0224] As an example, the one or more application control agents are deployed on an electronic device, and the one or more application control agents can call relevant execution engines, tools or interfaces of the electronic device to execute corresponding subtasks.

[0225] As an example, the above-mentioned one or more application control agents are deployed on a server, and the above-mentioned one or more application control agents can call the communication connection between the server and the electronic device, and send respective execution engine call instructions to the electronic device; correspondingly, the electronic device calls the relevant execution engine to execute multiple subtasks according to the execution engine call instructions of the above-mentioned one or more application control agents.

[0226] based on Fig. 6A and Figure 6B In the task execution scheme shown, the system agent can combine the historical records of executing related historical tasks (such as session context) to perform fusion analysis and reasoning when determining the task corresponding to the user instruction, and finally determine the task corresponding to the user instruction. On the one hand, there is no need for the user to input overly complex instructions, and on the other hand, it can accurately identify the task corresponding to the user instruction and respond to the user instruction in a timely, accurate and reliably manner.

[0227] And, based on Fig. 6A and Figure 6B The task execution scheme shown can avoid the impact of inefficiency, low accuracy and low reliability of analysis, reasoning or task execution process caused by excessively long session context by following the prescribed communication specifications for interaction between the system agent and the application control agent, as well as hierarchical management of the session context at the two levels. It can assist each level in conducting relevant analysis, reasoning or task execution, further improve the timeliness, accuracy and reliability of task execution, and improve the isolation between each level in the multi-layer agent architecture to ensure the safe and smooth operation of the system.

[0228] Also, based on Fig. 6A and Figure 6BThe task execution scheme shown uses a two-layer proxy architecture to implement user command understanding, multi-round dialogue fusion, multi-modal analysis and reasoning, multi-layer task management and isolation, task switching, etc. from easy to difficult based on language models such as LLM and MLLM. It can not only save reasoning costs, but also improve reasoning efficiency and the timeliness of user command response.

[0229] In addition, based on Figure 6B The task execution scheme in the multi-task scenario shown in the figure, the flexible, timely and accurate switching between multiple tasks can ensure smooth human-computer interaction and timely, accurate and reliable task execution in complex scenarios.

[0230] In some embodiments, during the process of the application control agent performing a task, if it is necessary to clarify certain issues about the task or supplement certain details about the task, the application control agent may request the system agent to interact with the user to clarify the task, and after completing the task clarification, update or supplement the task according to the clarification result and continue to perform the task. For example, during the process of the application control agent performing a task, the system agent may interact with the user according to the task clarification request of the application control agent, obtain the clarification result, and instruct the application control agent to continue to perform the task according to the clarification result, and so on, until the task is successfully completed.

[0231] As an example, the above task clarification can be Fig. 6A The task is performed during execution, or Figure 6B The task is performed during execution.

[0232] exist Fig. 6A During the task execution process, when the application control agent executes S605A and executes the task according to the instructions of the system agent, if it is necessary to clarify certain issues about the task or supplement certain details about the task, the application control agent sends a task clarification request to the system agent, and the request carries relevant information about the issues that need to be clarified or the details that need to be supplemented; after receiving the task clarification request from the application control agent, the system agent conducts one or more rounds of interaction with the user through the human-computer interaction module of the electronic device, such as through an intelligent assistant, to clarify relevant issues or obtain relevant details. After obtaining the clarification result, the system agent sends the clarification result to the application control agent, and the application control agent updates or supplements the task according to the clarification result combined with the second memory information and continues to execute the task; and so on, until the application control agent successfully completes the execution of the task.

[0233] exist Figure 6BDuring the task execution process, when one or more application control agents execute S605B and execute their respective subtasks, if a certain application control agent needs to clarify certain issues about the subtask or supplement certain details about the subtask, the application control agent sends a task clarification request to the system agent, and the request carries relevant information about the issues that need to be clarified or the details that need to be supplemented; after receiving the task clarification request of the application control agent, the system agent conducts one or more rounds of interaction with the user through the human-computer interaction module of the electronic device, such as through the intelligent assistant in the electronic device, to clarify the relevant issues or obtain the relevant details. After obtaining the clarification result, the system agent sends the clarification result to the application control agent, and the application control agent updates or supplements the subtask according to the clarification result combined with the second memory information and continues to execute the subtask; and so on, until the application control agent successfully completes the execution of the subtask.

[0234] As an example, see Figure 7 , Figure 7 Taking the case where the execution of a task requires the participation of the first application control agent and the second application agent, and the execution of the first application control agent task involves the task clarification process as an example, a schematic diagram of a task execution interaction process provided by an embodiment of the present application is shown. Figure 7 As shown, the task execution for the user instruction can be implemented through S701-S715:

[0235] S701: The system agent obtains user instructions.

[0236] S702: The system agent obtains first memory information.

[0237] As an example, the system agent can be Figure 7 The memory module obtains the first memory information. The memory module is used to store the first memory information, which includes the historical records of the system agent executing historical tasks, for example, the first memory information includes the historical records of the system agent executing historical tasks related to the tasks corresponding to the user instructions.

[0238] As an example, the system agent may also obtain the first memory information from the intelligent assistant, which may be Figure 7 The memory module shown acquires the first memory information.

[0239] For example, Figure 7 The memory modules shown may include a first memory module. The first memory module may be located in the system agent or outside the system agent, without specific limitation.

[0240] S703: The system agent determines the task corresponding to the user instruction according to the first memory information.

[0241] S704: The system agent splits the task corresponding to the user instruction into a first subtask and a second subtask, and determines a first application control agent and a second application control agent for executing the first subtask and the second subtask respectively.

[0242] Exemplarily, the execution order of the first subtask and the second subtask is: first subtask→second subtask.

[0243] Of course, in some embodiments, the first subtask and the second subtask may not need to be executed in a certain order, depending on the specific situation. Figure 7 Just as an example.

[0244] S705: The system agent distributes the first subtask to the first application control agent.

[0245] S706: The first application control agent obtains the second memory information of the first application control agent.

[0246] As an example, the first application manipulation agent can be from Figure 7 The memory module shown obtains the second memory information of the first application control agent. The memory module is used to store the second memory information of the first application control agent, and the second memory information of the first application control agent includes the historical records of the first application control agent executing historical tasks, for example, the second memory information of the first application control agent includes the historical records of the first application control agent executing historical tasks related to the first subtask.

[0247] For example, Figure 7 The memory module shown may include a second memory module. The second memory module may be located in the first application control agent or outside the first application control agent, without specific limitation.

[0248] S707: The first application control agent calls the execution engine to start executing the first subtask according to the second memory information of the first application control agent.

[0249] S708: The first application manipulation agent sends a task clarification request to the system agent during execution of the first subtask.

[0250] The task clarification request is used to request clarification of the first subtask.

[0251] As an example, the task clarification request carries relevant information about issues that need to be clarified or details that need to be supplemented regarding the first subtask.

[0252] S709: The system agent interacts with the user according to the task clarification request to obtain a clarification result about the first subtask.

[0253] As an example, the system agent can conduct one or more rounds of interactions with the user through a human-computer interaction module of the electronic device, such as through an intelligent assistant in the electronic device, to clarify issues related to the first subtask or obtain relevant details, and finally obtain a clarified result about the first subtask.

[0254] S710: The system agent sends a clarification result of clarifying the first subtask to the first application manipulation agent.

[0255] S711: The first application manipulation agent continues to execute the first subtask according to the clarification result.

[0256] As an example, the first application manipulation agent may update or supplement the first subtask according to the clarification result and continue to execute the first subtask.

[0257] S712: After completing execution of the first subtask, the first application control agent sends a first response to the system agent.

[0258] The first response is used to indicate that the first subtask has been completed.

[0259] S713: The system agent distributes the second subtask to the second application control agent.

[0260] S714: The second application control agent obtains second memory information of the second application control agent.

[0261] As an example, the second application manipulation agent can be Figure 7 The memory module shown obtains the second memory information of the second application control agent. The memory module is also used to store the second memory information of the second application control agent, and the second memory information of the second application control agent includes the historical records of the second application control agent executing historical tasks, for example, the second memory information of the second application control agent includes the historical records of the second application control agent executing historical tasks related to the second subtask.

[0262] For example, Figure 7 The memory module shown may include a third memory module. The third memory module may be located in the second application control agent or outside the second application control agent, without specific limitation.

[0263] S715: The second application manipulation agent calls the execution engine to start executing the second subtask according to the second memory information of the second application manipulation agent.

[0264] about Figure 7 For the introduction of S701, please refer to the above description of S601A or S601B. Figure 7For the related introduction of S702-S703 shown in the figure, please refer to the above description of S602A or S602B; Figure 7 For the introduction of S704 shown, please refer to the above description of S603B; Figure 7 For the description of S705 and S713, please refer to the description of S604B above. Figure 7 For the related introduction of S706-S707, S714-S715 shown, please refer to the above description of S605B, which will not be repeated here.

[0265] based on Figure 7 In the task execution scheme shown, the application control agent can request the system agent to trigger task clarification when it needs to clarify certain issues about the subtask or supplement certain details about the subtask during the execution of the subtask, so as to clarify certain issues about the task or supplement certain details about the task through one or more rounds of interaction with the user, thereby further ensuring the accurate and reliable execution of the task based on the user's instructions.

[0266] The following will take the user instruction "order another serving of dumplings" as an example, combined with a specific scenario, to introduce the specific implementation process of the task execution method provided by the embodiment of the present application. Figure 8 , Figure 8 A schematic diagram of a task execution process for a user instruction in a specific application scenario provided by an embodiment of the present application is shown.

[0267] like Figure 8 As shown in S801 in FIG. 1 , after receiving the user instruction (query) "help me order another serving of dumplings", the intelligent assistant obtains the system context, such as, but not limited to, the information of the electronic device (such as device identification, device name, login user name, etc.), the running A application (such as foreground application, background application, etc.), and the location information (such as the location coordinates collected by GPS, etc.).

[0268] like Figure 8 As shown in S802, the intelligent assistant obtains the first memory information of the system agent, such as the first memory information represents that the user likes dumplings from store B, and represents that the user's home address is address C.

[0269] like Figure 8 As shown in S803, the system agent determines the first prompt information (such as the first Prompt) as the input of the language model according to the user instruction, the first memory information and the system context, and performs fusion analysis and reasoning on the first prompt information through the language model to determine that the task corresponding to the user instruction is "A application control agent controls the A application interface to buy a portion of dumplings and deliver it to address C". After determining the task, the system agent distributes the task to A application control agent.

[0270] like Figure 8 As shown in S804 and S805, the A application control agent determines the second prompt information (such as the second Prompt) as the input of the language model according to the task indicated by the system agent and the second memory information of the A application control agent, and performs fusion analysis and reasoning on the second prompt information through the language model to determine and call the execution engine (such as an intelligent assistant) to execute the task instruction, such as: search (dumplings), LastAction: False, AgentState: continue, and the execution engine (such as an intelligent assistant) executes the task instruction.

[0271] like Figure 8 As shown in S806-S807 in FIG. 1 , since the application A control agent is not sure what kind of dumplings the user wants to buy, the application A control agent requests the system agent to trigger the task clarification to ask the user "What kind of dumplings do you want?".

[0272] like Figure 8 As shown in S808, after receiving the request "Please call my dad for me", the intelligent assistant obtains the current system context, including but not limited to the information of the electronic device (such as device identification, device name, login user name, etc.), the running A application (such as foreground application, background application, etc.), location information (such as location coordinates collected by GPS, etc.), and the interface information of application A; and sends the information to the system agent.

[0273] like Figure 8 As shown in S809-S810, the system agent determines the task as "call (Dad)" based on "Please call Dad for me", the first memory information and the system context, and calls the execution engine (such as an intelligent assistant) to execute the task.

[0274] like Figure 8 As shown in S811 and S812 in , after completing the call, the intelligent assistant determines that the user wants dumplings with celery filling and sends this information, the current system context, and the first memory information to the system agent.

[0275] like Figure 8 As shown in S813 and S814, the system agent determines the clarification result "I want dumplings with celery filling" based on the above information from the intelligent assistant, and instructs the A application control agent to continue to execute "A application control agent controls the A application interface to purchase a portion of dumplings with celery filling and deliver it to address C".

[0276] like Figure 8 As shown in S815, the A application control agent continues to execute the task according to the instruction of the system agent and completes the task of "A application control agent controls the A application interface to purchase a portion of celery dumplings and deliver them to address C".

[0277] based on Figure 8 In the task execution process for user instructions in the specific application scenario shown, the system agent can combine the first memory information of the system agent, such as the running A application, the user's favorite B store, the user's home address, etc. to perform fusion analysis and reasoning when determining the task corresponding to the user instruction "Please help me order another serving of dumplings". Finally, the task corresponding to the user instruction "A application controls the agent to control the A application interface to buy a serving of dumplings and deliver it to address C" is determined, and one or more rounds of interactions with the user are used to clarify what kind of dumplings the user wants. On the one hand, the user does not need to enter overly complex instructions. On the other hand, the task corresponding to the user instruction can be accurately identified, and the user instruction can be responded to in a timely, accurate and reliable manner.

[0278] And, based on Figure 8 The task execution scheme shown can avoid the low efficiency, low accuracy and low reliability of the analysis, reasoning or task execution process caused by too long historical records (such as session context) by following the prescribed communication specifications in the interaction between the system agent and the application control agent, as well as the hierarchical management of the memory information of the two layers (such as the first memory information and the second memory information), assist each layer in conducting relevant analysis, reasoning or task execution, further improve the timeliness, accuracy and reliability of task execution, and improve the isolation between each layer in the multi-layer agent architecture to ensure the safe and smooth operation of the system. In addition, based on Figure 8 The task execution scheme shown in the figure uses a two-layer proxy architecture to implement user command understanding, multi-round dialogue fusion, multi-modal analysis and reasoning, multi-layer task management and isolation, task switching, etc. from easy to difficult based on language models such as LLM and MLLM. This can not only save reasoning costs, but also improve reasoning efficiency and the timeliness of user command response. In addition, based on Figure 8 The task execution scheme in the multi-task scenario shown in the figure, the flexible, timely and accurate switching between multiple tasks can ensure smooth human-computer interaction and timely, accurate and reliable task execution in complex scenarios.

[0279] It should be understood that the various schemes of the embodiments of the present application can be used in reasonable combination, and the explanations or descriptions of the various terms appearing in the embodiments can be mutually referenced or explained in the various embodiments, without limitation.

[0280] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0281] It is understandable that, in order to implement the functions of any of the above-mentioned embodiments, the electronic device or server includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0282] The embodiments of the present application can divide the electronic device or server into functional modules. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0283] It should also be understood that each module in the electronic device or server can be implemented in software and / or hardware form, without specific limitation. In other words, the electronic device or server is presented in the form of functional modules. The "module" here can refer to a specific application integrated circuit ASIC, a circuit, a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions.

[0284] In an optional manner, when data transmission is implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is implemented in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disk (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0285] The steps of the method or algorithm described in conjunction with the embodiments of the present application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in an electronic device or a server. Of course, the processor and the storage medium can also exist as discrete components.

[0286] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

Claims

1. A task execution method, characterized in that: Applied to a system including a system agent and at least one application manipulation agent, the method comprises: The system agent obtains user instructions; The system agent determines the task corresponding to the user instruction according to the user instruction and first memory information, wherein the first memory information includes a historical record of executing historical tasks related to the task; The system agent instructs the application control agent to perform the task.

2. The method according to claim 1, characterized in that The method further comprises: During the process of the application control agent executing the task, the system agent interacts with the user according to the task clarification request of the application control agent to obtain a clarification result; The system agent instructs the application manipulation agent to continue executing the task according to the clarification result.

3. The method according to claim 2, characterized in that The system agent interacts with the user according to the task clarification request of the application control agent to obtain a clarification result, including: The system agent interacts with the user according to the task clarification request of the application control agent to obtain information input by the user; The system agent analyzes the information input by the user to obtain the clarification result; or instructs the application control agent to execute a clarification task according to the information input by the user, and obtains the clarification result according to the execution result of the clarification task.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The application control agent performs the task according to the task and second memory information of the application control agent, wherein the second memory information includes a history record of the application control agent performing historical tasks related to the task.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The system agent splits the task into a plurality of subtasks, the plurality of subtasks including a first subtask and a second subtask; the system agent determines a first application control agent for executing the first subtask, and determines a second application control agent for executing the second subtask; The system agent instructs the application control agent to perform the task, including: The system agent distributes the first subtask to the first application control agent, and distributes the second subtask to the second application control agent.

6. The method according to any one of claims 1 to 5, characterized in that The system agent determines the task corresponding to the user instruction according to the user instruction and the first memory information, including: The system agent determines the task corresponding to the user instruction based on a language model according to the user instruction and the first memory information.

7. The method according to any one of claims 1 to 6, characterized in that The system is deployed on a server, and the system agent obtains user instructions, including: The system agent receives the user instruction from the electronic device; The method further comprises: The application manipulation agent calls an execution engine, a tool or an interface on the electronic device to execute the task.

8. The method according to any one of claims 1 to 6, characterized in that The system is deployed on an electronic device, and the method further includes: The application manipulation agent calls an execution engine, a tool or an interface on the electronic device to execute the task.

9. The method according to any one of claims 1 to 8, characterized in that The system agent is a system agent, and the application control agent is a domain agent.

10. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer program instructions; A processor is used to execute the computer program instructions to support the electronic device to implement the method as described in any one of claims 1-6, 8-9.

11. A server, characterized in that: The server comprises: a memory for storing computer program instructions; A processor, configured to execute the computer program instructions to support the server in implementing the method as described in any one of claims 1-7 and 9.

12. A task execution system, characterized in that: The task execution system comprises: An electronic device and / or server, used to support the task execution system to implement the method as described in any one of claims 1-9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processing circuit, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

Images