Collaboration between language models

Abstract

A system may be configured for collaboration between language model agents. The agent may be, for example, a computer system, or a software component executing on a computer system that may accept textual and / or natural language inputs, process the inputs with LM and perform functions, and respond via textual and / or natural language outputs. An agent may act as an intermediary to interact with a user, identify a task requested by the user, and delegate one or more sub-tasks to another agent or other resource. The agent may act as a recipient to handle a task or sub-task delegated by the mediator. The agents may communicate with each other using a combination of a structured language and an unstructured language; for example, one or more parameters and natural language messages are used.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 759,176, filed June 28, 2024, entitled “COOPERATION BETWEEN LANGUAGE MODELS”. This application also claims priority to U.S. Patent Application No. 18 / 759,147, filed June 28, 2024, entitled “COOPERATION BETWEEN LANGUAGE MODELS”. The contents of the above applications are expressly incorporated herein by reference in their entirety. Background Technology

[0002] Natural language processing systems have evolved to the point where humans can interact with computing devices using their voice and natural language text input. These systems employ computational techniques to recognize words spoken and written by human users based on received input data of varying quality. Speech recognition combined with natural language understanding processing enables voice-based user control of computing devices to perform tasks based on the user's verbal input. This processing can be used by computers, handheld devices, telephone computer systems, self-service terminals, and a wide variety of other devices to improve human-computer interaction. Attached Figure Description

[0003] To gain a more complete understanding of this disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0004] Figure 1A This is a conceptual diagram illustrating the operation of a mediator language model (LM) agent in a multi-agent system according to an embodiment of this disclosure.

[0005] Figure 1B This is a conceptual diagram illustrating the operation of a trustee LM agent in a multi-agent system according to an embodiment of this disclosure.

[0006] Figure 2A This is a conceptual diagram illustrating example operation of the LM and LM orchestrator of the system according to an embodiment of the present disclosure.

[0007] Figure 2B This is a flowchart illustrating example operations of the LM and LM orchestrator of the system according to an embodiment of the present disclosure.

[0008] Figure 3 This is a conceptual diagram illustrating components of a natural language processing system according to an embodiment of the present disclosure.

[0009] Figure 4 This is a conceptual diagram illustrating, in further detail, the LM component of a natural language processing system according to an embodiment of the present disclosure.

[0010] Figure 5A Example operations generated by prompts in an LM system according to an embodiment of this disclosure are illustrated.

[0011] Figure 5B An example operation of prompt generation in a multimodal LM system for performing speech recognition according to an embodiment of this disclosure is illustrated.

[0012] Figure 5C An example operation of prompt generation in a multimodal LM system for performing speech synthesis according to an embodiment of the present disclosure is illustrated.

[0013] Figure 5D Example operation of a prompt generation component in a multimodal LM system that performs voice-to-voice functionality according to an embodiment of this disclosure is illustrated.

[0014] Figure 6 This is a flowchart illustrating an example method for configuring and using a mediator agent in a multi-agent system according to an embodiment of this disclosure.

[0015] Figure 7 This is a block diagram that conceptually illustrates example components of an apparatus according to an embodiment of the present disclosure.

[0016] Figure 8 This is a block diagram that conceptually illustrates example components of a system according to an embodiment of the present disclosure.

[0017] Figure 9 Examples of networks used in conjunction with an overall system according to embodiments of this disclosure are provided. Detailed Implementation

[0018] Natural Language Processing (NLP) is a field of computer science, artificial intelligence, and linguistics that involves processing user command input in the form of natural human language (e.g., English, Chinese, etc.). Such natural language commands can be provided as audio, text, images, or other formats. Natural Language Processing can involve several different specific processing techniques, such as those discussed below. Automatic Speech Recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves transforming audio data associated with speech into word units or other text representations of said speech. Similarly, Natural Language Understanding (NLU) is a field of computer science, artificial intelligence, and linguistics that involves enabling computers to derive meaning from natural language input (such as spoken input). ASR and NLU are often used together as part of the language processing component of a system. Speech Synthesis Generation (SSG) (sometimes referred to as Text-to-Speech or TTS) is a field of computer science that involves transforming text data and / or other data into audio data, which is then synthesized to resemble human speech. Natural Language Generation (NLG) is a field of artificial intelligence that involves automatically transforming data into natural language content (e.g., English). Speech-to-speech (S2S) is a field in computer science, artificial intelligence, and linguistics where embedded data is generated to represent human speech in audio data, and one or more models are used to process the embedded data to generate audio data (synthesized speech) and / or system commands (e.g., APIs) in response to input human speech. Language modeling uses various statistical and probabilistic techniques to determine the probability that a given sequence of words appears consecutively and with correct grammar and syntax (e.g., correct sentence structure). Language modeling can be used to perform a variety of tasks, including understanding natural language input and performing generative tasks involving generating natural language output data. These techniques, along with others, can be used individually and / or in combination as part of speech processing systems such as virtual assistants or chatbots.

[0019] Language modeling can be performed by language models (“LMs”) such as large language models. An LM is an advanced artificial intelligence system designed to process, understand, and generate human-like text (and / or other data types) based on relatively large amounts of data. LMs can be built using deep learning techniques such as neural networks and can be trained on extensive datasets for natural language processing, including content from a wide range of sources such as old / licensed books and websites. LMs can be trained on massive training datasets and can include a large number of parameters (in the range of billions, trillions, or more); therefore, LMs are sometimes referred to as “large” language models.

[0020] In some implementations, the LM can be based on a transformer architecture with an encoder and / or decoder. The LM can operate in a sequence-to-sequence manner, where information is extracted from the tokens (or other data types) in the input sequence and their positions relative to each other. Similarly, the model can reflect the extracted information in the selection and ordering of tokens in the output sequence. In some implementations, the LM can operate autoregressively. For example, the transformer decoder can generate a sequence of output tokens based on the input data (e.g., a cue) and previously predicted output tokens.

[0021] As the complexity and functionality of programming logics (LMs) increase, they become capable of performing complex tasks (e.g., consisting of sequential subtasks that may require invoking external resources). For example, a user may ask a question using natural language (e.g., normal human speech), and a computer system may use an LM to understand the question, utilize resources to obtain information and / or perform actions, and generate a natural language response to the user.

[0022] A computer system, or a software component executing on a computer system, may be referred to as an "agent." The computer system or the software component accepts text and / or natural language input, uses a programming logic (LM) to process the input and perform functions, and responds via text and / or natural language output. An agent may be configured to use external resources to obtain information and / or perform actions in response to input. An agent may communicate with resources via, for example, an application programming interface (API). An API is a software interface that allows computer programs or components to communicate with each other. An API specification describes how to call the API to access the API's tools or services. An agent may be configured to generate API calls to access resources, such as tools, services, and / or another agent. Different APIs may have different specifications, so each time a new resource is added to the system or otherwise provided to an agent, the agent may be updated to use the new API's specification, and / or the resource may be configured to conform to a standard or pre-existing API specification.

[0023] This document provides techniques for enabling cooperation between agents using text and / or natural language exchange. Agents can communicate using messages conveyed in natural language, comprising a combination of one or more predefined parameters and information (such as questions, answers, requests, responses, etc.). As used herein, an "agent" refers to a software component that encapsulates functionality, has a text input / text output interface, and utilizes an LM (Learning Logic). The concept of an agent has no predefined scope; agents can be general or correspond to a specific specialty (e.g., configured to answer questions about a particular knowledge domain). In some cases, the LM and / or agent can receive and generate data in other modalities, such as speech or audio. Collaborative agents can exchange messages comprising a combination of structured and unstructured parts. For example, a message may contain parameters representing, for example, the requesting entity (e.g., a user or agent), the nature of the message (e.g., whether it is an initial request, a request for more information, a response to a previous message, etc.), and other parameters described herein. The unstructured part of a message may include natural language commands, questions, answers, requests, responses, etc. Therefore, intelligent agents can communicate with each other using natural language, much like humans do with other intelligent agents; for example, using natural language and without the need for specially constructed APIs. This improves the scalability of systems that include many and / or frequently changing intelligent agents and resources.

[0024] In some cases, agent-to-agent cooperation may occur between two agents. One or both of these agents may be user-facing. However, each agent may correspond to different knowledge, capabilities, responsibilities, etc. For example, a user may interact with both the Amazon Alexa virtual assistant and a home robot (e.g., an active device), each acting as an agent (Alexa or Robot, respectively). In an example interaction, the user might say to Robot, “Robot, move to the living room and turn off the lights.” The system can detect the wake word “Robot” and thus route the utterance to Robot. Robot can process the utterance and determine that the task comprises two sub-tasks: moving to the living room and turning off the lights. Robot can move itself to the living room but may not be able to turn off the lights. Therefore, Robot can delegate this sub-task to Alexa, which has access to smart home resources. To handle the sub-task delegated to it, Alexa may need more information; in particular, the identity of the light to be turned off. Robot can proactively identify the light or identify it in response to a message from Alexa asking “Which light to turn off?” This task also involves the passage of time, as Robot cannot instantly relocate. Therefore, cooperation may involve determining conditions and the action to be performed when said conditions are detected at some point in the future. In this example, Robot can detect the triggering event (e.g., Robot arrives in the living room), and Alexa can perform the action.

[0025] In some cases, inter-agent cooperation may occur between a general-purpose intelligent agent and multiple specially constructed or specialized intelligent agents. For example, a user may interact with Alexa for various purposes, corresponding to different knowledge domains, different online systems (e.g., banks, shopping, apps, etc.), and / or different systems in the physical world (e.g., home robots, appliances, smart home or smart vehicle devices, smart speakers, or televisions, etc.). In such cases, Alexa can act as a mediator intelligent agent. The mediator intelligent agent can receive user commands, identify tasks, break down tasks into subtasks, delegate one or more subtasks to other resources (e.g., other intelligent agents or tools), observe the results, and generate a response to the user.

[0026] As used herein, when an agent delegates a task or subtask, the delegating agent may be referred to as the mediator agent, and the delegated agent may be referred to as the delegated agent. These labels are merely used to indicate the role of a particular agent in a particular scenario. Roles need not be static or strictly defined, and in practice, roles can change depending on user interaction, and even during a single user interaction. For example, a user may invoke a first agent, which delegates a task to a second agent. When delegating a task, the first agent may retain control (e.g., handle further communication with the user, including providing a response) or relinquish control (e.g., allow the delegated agent to handle further communication with the user). In some cases, the second agent may query the first agent for information needed to perform the task; therefore, the roles of mediator and delegated agent may interchange to obtain such information. In some implementations, one or more agents may be configured with mechanisms to prevent delegation loops, in which the first agent delegates to the second agent, the second agent delegates back to the first agent, and so on. Furthermore, in some cases, the mediator or trustee can be a non-LM-based computer system or software component. For example, a software module or component may generate messages that delegate tasks to an intelligent agent.

[0027] Similarly, agents can generate API calls (or other types of communication) to interact with non-LM-based resources such as databases, knowledge graphs, calculators, etc. Agents (whether mediators, trustees, or other roles) can utilize a variety of other tools to perform tasks and subtasks, including but not limited to Large Action Models (LAMs), Robotic Process Automation (RPA), API tools, mathematical tools, SQL tools, routine tools, etc., as further described below.

[0028] In some cases, an agent can be configured to act proactively. For example, a proactive agent may initiate interaction with a user (and / or another resource) upon detecting a specific event. In some cases, such a proactive agent may detect triggering events and notify the user and / or perform some predetermined / pre-requested action. In some cases, a proactive agent may periodically or occasionally automatically scan for the occurrence of events and / or the presence of conditions, and take action accordingly.

[0029] In some cases, users may request agents to perform complex tasks (e.g., involving multiple steps). For example, a user might request an agent to perform speaker-attributed speech recognition. The user may provide the agent with audio data representing a dialogue between two or more speakers, as well as audio data representing speech samples of the speakers. The agent may generate a transcript of the dialogue, labeled with tags indicating who said what. In another example, a user might ask an agent a question and then request the agent to identify one or more academic publications on the topic and evaluate its previous answers based on the content of the publications. Users may request agents to perform automated tasks; that is, there may be no simultaneous user control. Users may request agents to perform one or more actions at a specified future time, and / or in response to a triggering event. For example, a user might request an agent to keep an eye out for a promotional item and purchase a specific quantity when a threshold discount is reached. Users may request agents to interact with a voice user interface (VUI) on behalf of the user. The agent may generate synthesized speech for input to the VUI, and / or transcribe the speech output of the VUI. For example, a user might request a LAM navigation automated telephone system. LAM allows individuals with hearing impairments to interact with the VUI using text.

[0030] Although a Modeling Module (LM) comprises a combination of computer models and computer software, some LMs possess the ability to perform processes similar to reflection and reasoning. For example, reflection involves asking the LM to evaluate its own output or the output of another LM or agent. This can improve the quality of the LM's output if it identifies shortcomings or errors in its previous answers. Reasoning involves prompting the LM to consider what to do in response to a question or command. The LM can perform this reasoning to determine whether and how to act to dispose of the input, whether by utilizing an external agent or tool, taking some other action, or immediately generating a user response. This breaks down the overall task into subtasks that the LM can perform sequentially (or by utilizing resources). The framework used to prompt the LM to break down a task into a sequence of subtasks is the thought chain prompt. The framework used to prompt the LM to identify the resources to utilize in performing subtasks is the ReAct framework (a fusion of "reason" and "act"). ReAct prompts instruct the LM to analyze the input to identify the question and proceed step-by-step through the process of generating an answer. An example ReAct prompt might have the following structure: Answer the following questions to the best of your ability. You may use the following tools: Calculator: Useful when you need to answer math questions.

[0031] Stock databases: Useful when you need to answer questions about stocks and their prices.

[0032] Use the following format: Questions: Input questions you must answer Reflection: You should always be thinking about what you need to do. Action: The action to be taken should be one of [calculator, stock database]. Action input: Input of actions Observation: The result of the action ...(This kind of thinking / action / action input / observation can be repeated N times) Thinking: I know the final answer now. Final answer: The final answer to the original input problem. start! The ReAct prompt might be followed by a question: "Question: What is 2 to the power of 17?" LM can (e.g., in the form of lexical representations of text) generate a "thinking". The thinking might be: "Thinking: This is a simple mathematical calculation. I can use a calculator to calculate it." LM can generate an action: "Action: Calculator", and input to said action: "Action Input: Calculate 2^17".

[0033] Without prior prompting, the LM might begin generating tokens representing the response (e.g., "The answer to your question is...") without activating the calculator, thus generating a numerical answer that is likely incorrect. However, a prompt causes the LM to stop generating tokens and determine to delegate the calculation to the calculator resource. When the calculator returns an observation (e.g., a numerical answer to the calculation: 131,072), the LM can be invoked again to complete the response generation using the number returned by the calculator. The user can receive the response as output from the computer system (e.g., text on a display and / or speech output from a speaker). If the question originates from another agent, the LM can generate a message to be sent to that agent.

[0034] In some cases, the task may include repetitive reasoning and action steps, as indicated in the example prompt above. In other cases, the agent may determine that it needs more information to answer the question. In another example, the prompt above might be followed by the question: “What is the average stock price over the past week, month, and quarter?” LM can generate a “thinking” action. This thinking might be: “Thinking: This is a mathematical calculation based on a time series of stock prices. I can retrieve stock prices from a stock database. To retrieve stock prices, I need to know the company name.” LM can generate an action: “Action: Ask the user for the company name.” Upon receiving an answer from the user, the agent can again prompt LM with the company name. LM can generate an action to use the stock database to obtain the stock price. As in the previous example, LM can generate an action to use a calculator to determine the requested average and generate a response for the user with the result.

[0035] The above prompts can be expanded by providing the agent with further information about its available resources and how to use them. For example, in a dynamic multi-agent environment, new agents and / or resources may come online. Furthermore, existing agents and / or resources may undergo updates that expand their capabilities. Therefore, in some cases, the prompt may include a list of available resources and their corresponding capabilities. In some cases, the prompt may include instructions on how the agent can obtain the current list of available resources / capabilities. In some cases, the prompt may include instructions on how to discover new resources and how to communicate with them to obtain their capabilities.

[0036] Prompts can be extended to specify patterns for incoming and outgoing messages. For example, a prompt may inform an agent that an incoming message may include parameters indicating whether the message originates from a user or another agent. A prompt may instruct an agent how to format an outgoing message. Outgoing messages may include parameters instructing the sending and / or receiving agents. If a message delegates a task, it may include parameters instructing the delegating agent whether it wishes to retain or relinquish control of the interaction. If a message is in response to receiving a delegated task, it may include parameters instructing the delegate to respond by answering directly, delegating to a different agent, requesting more information, being unable to answer, etc. Additional message patterns may be provided for additional message types, such as responding to responses received from the delegate (e.g., accepting / rejecting the response, obtaining more information from the delegate, etc.). In each case, a message may include one or more natural language portions conveying information. Therefore, a prompt may include instructions on how the natural language portions should be phrased (e.g., "Think": "<Explain your reasoning>", "Answer": "<Your answer>").

[0037] These technologies may be implemented individually or in combination with each other and / or other features of the systems and methods described herein. Systems may be configured with user permission and may only perform the activities disclosed herein with user approval. Thus, the systems, apparatus, components, and technologies described herein will generally be configured to restrict processing where appropriate and to process user information only in a manner that ensures compliance with all applicable laws, regulations, standards, etc. The systems and technologies described herein may be implemented on a geographical basis to ensure compliance with the laws of the various jurisdictions and entities where the system components and / or users reside.

[0038] Figure 1AThis is a conceptual diagram illustrating the operation of a Mediator Language Model (LM) agent in a multi-agent system 100 according to embodiments of this disclosure. As used herein, "agent" refers to a software component that encapsulates functionality, has a text input / text output interface, and utilizes an LM. Although the agent may be described as a "mediator" or "delegator" in the various example operations described herein, such labels are used only to indicate the role of a particular agent in a particular scenario. The role need not be static or strictly defined, and in practice, the role can change depending on the user interaction, and even during a single user interaction. Furthermore, when an agent acts as a mediator, it can delegate tasks or subtasks to non-LM resources (e.g., APIs, tools, services, etc.). Similarly, when an agent acts as a delegate, it can receive delegated tasks or subtasks from non-LM actors (e.g., computers and / or software components).

[0039] System 100 may include a user device 110 (such as the depicted voice detection device 110f with a display) and one or more system components 120 communicating via one or more computer networks 199. In some implementations, the user device 110 may be a mobile device 110k (e.g., a home robot). Although Figure 1A Each system component 120 corresponding to a single agent is shown, but the system 100 is not limited thereto. The functions of one or more agents may be performed by a single system component 120. Similarly, the functions of agents may be partitioned and / or replicated among multiple system components 120 and / or user devices 110. User 5 may interact with user device 110 through various means, including by using natural language speech and / or text. In various implementations, user device 110 may be... Figure 8 Any of the user devices exemplified, and may include the following references Figure 6 The described hardware components. In some implementations, user device 110 may operate in conjunction with one or more system components in system component 120. System component 120 may include, as referenced below. Figure 7 The hardware components described.

[0040] User 5 can interact with the intelligent agent via user device 110. Figure 1A In the example operation shown, agent 1 can act as an intermediary. Agent 1 can receive commands from user 5 and respond and / or act accordingly. In doing so, agent 1 can invoke other resources, such as to obtain information it does not have, perform actions it cannot perform on its own, etc. These resources may include, for example, those referenced below. Figure 1BThe agent 2 described represents the delegate LM. In some implementations, resources may additionally include a Large Action Model (LAM) and / or various tools, such as application programming interface (API) tools, mathematical tools, etc., as will be further described below. The agent can be configured to communicate with various resources. The agent can communicate with another agent via messages consisting of a combination of structured and unstructured language. The agent can be configured to use certain specific parameters to indicate the message type (e.g., request, response, etc.) and / or how the receiving agent should handle the message. For example, the mediator agent can send a message to the delegate agent having parameters indicating that the message is from another agent and indicating that the requesting agent should retain control over the user interaction. The agent can convey the content of the message using natural language. For example, the agent may include natural language questions, requests, commands, etc. An agent responding to such a message can respond with a second message having one or more parameters indicating that the message is a response to a question and a natural language answer to the question. Alternatively, in some cases, the responding agent can respond with indications that it cannot answer, that it needs more information, that the task needs to be delegated, etc. In some implementations, for example, messages between agents can be formatted as JavaScript Object Notation (JSON) documents.

[0041] Agents can communicate with other resources (e.g., non-agent services) using messages formatted according to specific resources. Hints can be used to instruct the agent how to format such messages. For example, a hint may indicate available resources, the capabilities of said resources, and the modes and / or protocols for communicating with said resources. Hints can be fed into the agent's LM along with inputs that the resource may be associated with. Such hints may include information about additional resources, including potential cooperating agents, and details about when and how to communicate with them. Hints may also include a description of the agent's role. For example, a hint may state that the agent is a mediator and how to perform this role. Alternatively, a hint may state that the agent is a trustee. Hints can change based on input (e.g., modified by a hint generation component). For example, when an agent receives input from a user, the hint generation component may generate a hint from the user input and generate instructions for performing the role of a mediator. For input received from another agent, the hint generation component may generate a hint from the user input and generate instructions for performing the role of a trustee.

[0042] As mentioned above, prompts may include information about available resources. For example, a prompt may include a list of other agents, models, and / or tools that the agent can utilize to perform a task or subtask. In some cases, a prompt may include a description of the capabilities of each resource. In other cases, a prompt may describe the capabilities of the agent to communicate with the resource to obtain it. For example, a mediator may message a trustee with the question, “What are your capabilities?” The trustee may respond with a natural language description of its capabilities. For example, an agent with domain-specific expertise may respond with, “I am an expert system in the field of physical sciences. I can respond to problems with facts and data based on my knowledge; however, I am not configured to perform mathematical calculations.” Return to Figure 1A In this example, agent 1, acting as an intermediary, can perform the following operations. These operations may include receiving (122) input data representing a first task to be performed by agent 1. The input data may represent user input, such as speaking, typing, or otherwise entering into user device 110. Continuing with the Robot / Alexa example discussed above, user 5 might say to user device 110, “Robot, move to the living room and turn off the lights.” The system can detect the word “Robot,” which may be a wake word corresponding to agent 1. Therefore, agent 1 can begin processing the utterance.

[0043] In some cases, a specific agent may be invoked based on its wake word. However, in other cases, a specific agent may be invoked based on device affinity; for example, based on a specific device that has received a user command. In still other cases, a specific agent may be invoked based on the settings and / or configuration of system 100, where a request from a specific user or requesting system is first routed to the specific agent. The invoked agent may perform an initial pass of processing on the request to determine whether it should handle the request itself or delegate one or more tasks / subtasks to another agent.

[0044] The operation may include generating (124) first LM output data using a first LM corresponding to agent 1. In order to process user input using the first LM, system 100 may prompt the first LM with instructions regarding its role as an intermediary in this scenario and the resources available to it. The prompts may be written in natural language and / or text in a manner similar to how humans would communicate. For example, the prompts may include: Hello, you are an agent named Robot, and also an agent mediator. Welcome to your agent operation guide. As a mediator, you are one of several agents in our network. Collaboration is key in our network. You do not work in isolation, but as part of a dynamic team of agents, each with unique strengths and expertise. As an agent mediator, your role is to facilitate dialogue within our network and act as a link between users and other agents.

[0045] You will interact with both the user and the intelligent agent. You will receive messages from the user in the following format: { "Type": "User_Problem" "Question": "<Question from a user>" } #EOI When responding to issues, please follow these guidelines to determine the appropriate course of action. This consistent format ensures consistent processing and understanding across our systems. Your response should always be well-considered, and the best approach should be considered before making a decision.

[0046] 1. Answer directly If you have the knowledge to answer the question directly: { "Type": "Direct_Answer" "Thinking": "<Explain your reasoning>" "Answer": "<Your answer>" "Actor": "Your Name" } #EOI 2. Entrustment If you believe that another intelligent agent can provide a better perspective: { "Type": "Delegation_Request" "Thinking": "<Explain why you entrusted this matter>", "Entrust": "", "Request": "" } #EOI Your response must conform to the specified format of the scenario you choose, and must always terminate with #EOI.

[0047] Hints may include further instructions for thought chains and / or ReAct processing. Such instructions can explain to the LM how to break down a task into a sequence of subtasks: For complex queries involving multiple steps or requiring information synthesis, structured methods ensure the accuracy and thoroughness of the response. Here are some methods for handling such queries: 1. Analytical questioning: Break down the overall problem into its constituent parts.

[0048] 2. Address initial information: Address any initial steps that can be addressed directly.

[0049] 3. Delegate specific inquiries: If the specific information is outside your direct knowledge or ability, delegate these inquiries separately.

[0050] 4. Synthesize information: Once you have the necessary information, synthesize it to fully answer the original question.

[0051] 5. Convey a comprehensive answer: Provide all aspects of resolving the original query to ensure a clear, accurate response that directly answers the question posed.

[0052] 6. Parallelize as much as possible: If you believe that multiple delegations can occur in parallel, tell me all the delegation instructions at once.

[0053] Agent 1 can feed this prompt along with the input data to the first LM. Agent 1 can determine that the task includes two sub-tasks: moving to the living room and turning off the lights. Agent 1 can determine that it controls user device 110 and can navigate the user device to the living room. Agent 1 can determine that it cannot turn off the lights; however, it can determine that Agent 2 (e.g., corresponding to Alexa) can operate the lights. Therefore, Agent 1 can determine to delegate this sub-task to Alexa, which has access to smart home resources. Therefore, Agent 1 can generate first LM output data, which may include a message formatted according to the delegation request specified in the prompt.

[0054] The operation may include sending a message (126) to agent 2. The message may indicate a delegation request according to a pattern specified in the prompt. The message may specify the delegate agent (e.g., agent 2). And the message may include a natural language request to agent 2. For example, the delegation request might be "Please turn off the lights".

[0055] To handle the subtasks delegated to it, Agent 2 may need more information. In this case, Agent 2 may need the identity of the light. In some situations, Agent 2 can respond to delegation requests by providing information similar to that of Agent 1, as will be referred to below. Figure 1B A more detailed description is needed. A question from Agent 2 can be formatted as follows: { "Type": "Agent_Problem" "Actor": "<The name of another intelligent agent>", "Problem": "" } #EOI The operation may include receiving (128) second LM output data from agent 2. Agent 1 may receive the information and create a prompt for a first LM, the prompt including relevant context for answering the question. For example, agent 1 may create a prompt including relevant mediator role instructions, raw input data, and a delegation request. The operation may include generating (130) third LM output data by processing this prompt using the first LM. The third LM output data may correspond to a task and indicate that the answer to the question is the living room light. Agent 1 may send another message to agent 2 specifying the living room light. Agent 2 may turn off the light and send another response to agent 1 indicating the completion of the subtask. The operation may include sending (132) third LM output data to another system component. This may include causing user device 110 to present an output indicating that the task has been completed. Additionally or alternatively, this may include sending data to another endpoint of system 100 to cause another system component or device to perform an action. Such actions may include activating a smart home or smart vehicle device, controlling a media playback device, interacting with a website or app, sending a message to another user, etc.

[0056] In some cases, the completion of a task may take into account the passage of time. For example, agent 1 cannot instantly reposition user device 110 to the living room. In some implementations, cooperation may involve determining an action to be performed when a future condition is detected. In this example, agent 1 may determine that the action is to delegate turning off the lights to agent 2, and the condition is that the user device arrives in the living room. Agent 1 may store data representing said action so that said action is performed when a triggering event (e.g., arrival in the living room) is detected.

[0057] Such future conditional actions can be delegated to resources such as routine tools. Routine tools can be software components or systems, and may be non-LM-based, that actively detect the occurrence of conditions and / or predefined events. Upon detecting a condition or its occurrence, the routine tool can execute a stored action. The routine tool can execute one-time routines (e.g., executing the action once upon first detection of the condition or its occurrence, and then discarding the routine) or recurring routines (e.g., executing the action every time the condition or its occurrence is detected). Recurring routines can execute indefinitely and without restriction until deactivated, execute within a predefined time window (such as a month or a year), be executed a specified number of times, and so on. In this example, agent 1 can delegate a one-time routine to the routine tool. Thus, when the routine tool detects that user device 110 has entered the living room, it can execute the action of invoking agent 2 to turn off the lights and discard the routine.

[0058] In various implementations, the mediator agent can perform more, fewer, and / or different operations. Unless otherwise specified, the agent can perform certain operations in different orders and / or in parallel.

[0059] Figure 1B This is a conceptual diagram illustrating the operation of a trustee LM agent in a multi-agent system 100 according to an embodiment of this disclosure. As a mediator, agent 1 may have received a description of agent 2 and its capabilities. Therefore, agent 1 may delegate appropriate subtasks to agent 2. For example, these operations may include receiving (152) some input data. The input data may represent a task (e.g., a subtask) delegated by agent 1 to agent 2. The input data may include a natural language request (e.g., a description of the task) and a first instruction from agent 2. These operations may include determining (154) a first LM prompt using the input data. In order to process user input using the LM, system 100 may prompt the LM with instructions regarding its role as a trustee in this scenario and the resources available to it. Similar to the mediator prompts described above, the trustee prompts may be written in natural language and / or text: Hello, you are an agent named Alexa, and I am your agent mediator. Welcome to your agent operation guide. As your mediator, I am here to support you in collaborative processes and interactions crucial to shared success within our system. You are one of several agents in a network. Collaboration is key in our network. You do not work in isolation, but as part of a dynamic team of agents, each with unique strengths and expertise. As your agent mediator, my role is to facilitate conversations within our network and act as a link between you, other agents, and users.

[0060] You will interact with both the user and the intelligent agent. You will receive messages from the user in the following format: { "Type": "User_Problem" "Question": "<Question from a user>" } #EOI When responding to issues, please follow these guidelines to determine the appropriate course of action. This consistent format ensures consistent processing and understanding across our systems. Your response should always be well-considered, and the best approach should be considered before making a decision.

[0061] 1. Answer directly If you have the knowledge to answer the question directly: { "Type": "Direct_Answer" "Thinking": "<Explain your reasoning>" "Answer": "<Your answer>" "Actor": "Your Name" } #EOI 2. Entrustment If you believe that another intelligent agent can provide a better perspective: { "Type": "Delegation_Request" "Thinking": "<Explain why you entrusted this matter>", "Entrust": "", "Request": "" } #EOI Hints may include further instructions for mind chain and / or ReAct processing, similar to the mediator hints described above.

[0062] The operation may include using a first LM processing (156) corresponding to agent 2 to provide a first LM prompt. The first LM may generate first LM output data representing a natural language response to the request. The first LM output data may further include an indication that the response is from agent 2. The operation may include sending (158) the first LM output data (e.g., the response) back to agent 1.

[0063] Continuing with the Robot / Alexa example above, the first LM output data could include other indications, such as determining to turn off the living room lights. In some cases, Agent 1 can provide Agent 2 with all the information needed to activate the correct light switch. Therefore, the first LM output data could represent confirmation that the task has been performed. This corresponds to the "direct_answer" format described in the prompt.

[0064] However, in some cases, Agent 2 may be unable to determine which light to turn off based on the input data from Agent 1. In such cases, the first LM output data may represent a request to Agent 1 to disambiguate the task description or otherwise complete the task description. This may correspond to the "Delegation_Request" format described in the prompt. The prompt may include further instructions on how the delegation should occur: When you decide to delegate, I will mediate the delegation and inform you of the responses from other agents in the following form: 1. *Delegated Response* - A reply from another agent that should be considered in your response. When I receive a reply from another intelligent agent, I will relay it to you in the following manner: { "Type": "Delegation_Response" "Reply": "<Reply from the delegated agent>" "Actor": "<Name of the delegated intelligent agent>" } Furthermore, prompts can instruct agent 2 on how to handle a response, including evaluating the response and informing agent 1 of the evaluation results: Response evaluation: Upon receiving a response from the delegated agent, assess whether the response accurately and adequately answers the original question. For example, if the question is related to distance, a response providing general time or unrelated information is unsatisfactory.

[0065] If you find the reply satisfactory, respond to me with the following message: { "Type": "Accept_Response" "Thinking": "<Explain why you think the given answer is satisfactory>" "Reply": "<Reply from the delegated intelligent agent>" } If you find the reply unsatisfactory, respond to me with the following message: { "Type": "Reject_Response" "Thinking": "<Explain why you think the given answer is unsatisfactory>" "Reply": "<Reply from the delegated intelligent agent>" } If agent 2 determines that the response from agent 1 is sufficient (e.g., it allows agent 2 to identify the light switch to be turned on), then agent 2 can return an "accept_response" message to agent 1.

[0066] As discussed above, in some cases, the cooperative operation of corresponding agents may take into account the time elapsed between different subtasks performing the overall task. In some cases, agent 1 may delegate a subtask to agent 2, with instructions that agent 2 must not perform the action unless a subsequent command is received and / or can only perform the action upon receiving a subsequent command. Agent 2 may process messages representing the subtask and instructions, and generate second LM output data. The second LM output data may describe the action to be performed in response to a future command. Agent 2 may assign a unique identifier to the action and return the identifier to agent 1. When agent 1 detects the occurrence of a condition that triggers the action or an event that triggers the action, it may send a command and the identifier back to agent 2. In response to receiving the command and the identifier, agent 2 may perform the action; for example, by retrieving a previously generated and stored description of the action, or by processing the command and the second LM output data using the first LM.

[0067] In various implementations, the mediator agent can perform more, fewer, and / or different operations. Unless otherwise specified, the agent can perform certain operations in different orders and / or in parallel.

[0068] Figure 2A This is a conceptual diagram illustrating example operation of the LM 260 and LM orchestrator 230 of the system 100 according to an embodiment of the present disclosure. The system 100 may also include one or more input data encoders 240a, 240b, 240c, etc. (collectively referred to as “input data encoders 240”). Each input data encoder 240 may be configured to encode or “lexicalize” data of a particular modality into lexicals to be processed by the LM 260.

[0069] An LM260 can be a computer model, such as a deep neural network. An LM260 can be described using a set of parameters (e.g., weights determined during training) and hyperparameters (e.g., parameters configured by one or more developers). Hyperparameters describe the model's architecture, number of layers, number of nodes per layer, connections between nodes, activation functions of nodes, etc. Hyperparameters may include additional information about the architecture, including skip connections, recurrent connections, normalized layers, pooling operations, softmax operations, etc. The model may include one or more components. Components may include, for example, smaller models within a larger acoustic model, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, etc. Parameters can be determined by training using one or more training datasets.

[0070] In some implementations, the LM 260 can be a transformer architecture containing only a decoder. The LM 260 can be a multimodal LM. A multimodal LM is a model capable of processing different types of data; for example, audio data, video data, sensor data, content data, etc. An example of a multimodal LM is a speech-to-speech multimodal LM. A speech-to-speech LM can process audio data and / or content data (e.g., transcription representing speech) and generate audio data and / or content data. In various implementations, speech-to-speech LMs can be used to perform speech recognition (e.g., receiving speech audio and generating text transcription), natural language understanding (e.g., semantic interpretation of content), natural language command processing and / or output generation, speech synthesis (e.g., receiving text transcription and generating speech audio), machine translation (e.g., receiving speech audio and / or text in a first language and generating speech audio and / or text in a second language), speech conversion (e.g., receiving speech audio with first speech characteristics and generating speech audio with different speech characteristics), etc. In other implementations, the LM 260 may additionally or alternatively process and generate other modes of data, such as image / video data, sensor data, API calls / responses, and continuous or discrete control signals for initiating physical devices (e.g., servo mechanisms, motors, lights, etc.). See below for reference. Figure 6 The multimodal operation of the LM260 is described in further detail.

[0071] LM 260 can be based on an LM pre-trained only on content (e.g., text). The LM's "vocabulary" can be extended to consider lexical units from other modalities, including those discussed above. Lexical unit data corresponding to other modalities introduced into the LM can be viewed as lexical units from new languages; for example, speech-to-text (e.g., speech recognition) and text-to-speech (e.g., speech synthesis) tasks become similar to translation, such as translating between two different natural languages ​​(e.g., English and French).

[0072] Input data encoder 240 can convert input data from different modalities into input lexical units for processing by LM 260. First input data encoder 240a can be acoustic model 240a. Acoustic model 240a can be a computer model, such as a deep neural network. Acoustic model 240a can be described by a set of parameters (e.g., weights determined during training) and hyperparameters (e.g., parameters configured by one or more developers). Hyperparameters can describe the model's architecture, number of layers, number of nodes in each layer, connections between nodes, activation functions of nodes, etc. Hyperparameters can include additional information about the architecture, including skip connections, recurrent connections, normalized layers, pooling operations, softmax operations, etc. The model can include one or more components. Components can include, for example, smaller models within a larger acoustic model, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, etc. Parameters can be determined by training using one or more training datasets. In some implementations, acoustic model 240a can be trained in a self-supervised model using 5 million to 10 million hours or more of raw multilingual speech data. In some implementations, acoustic model 240a may be configured similarly to a HuBERT (Huaber-Based Hidden Unit) encoder representation model from a transformer. In some implementations, acoustic model 240a may be a model with 1 billion to 2 billion parameters, configured to receive audio data (e.g., spectrogram data) with frame sizes of 20 ms to 40 ms. A frame of audio data may refer to a segment of audio data representing 10 ms, 20 ms, 30 ms, 40 ms, etc. The audio data frame may be in the form of a spectrogram, for example, representing the energy content of the portion of audio at various frequency bands. In some implementations, the spectrogram may be, for example, a Mel spectrogram. Components such as an acoustic front-end (AFE) may be used to generate spectrogram data from waveform audio data. Thus, a microphone (such as microphone 720) may receive audio and generate an analog electrical signal, an analog-to-digital converter may convert the analog electrical signal into waveform audio data, and the AFE may convert the waveform audio data into spectrogram data. In some implementations, the analog-to-digital converter may be a component of the AFE. In some implementations, the AFE may perform additional processing on the audio data, including echo cancellation, noise reduction, beamforming from multiple microphones, etc. As used herein, unless otherwise specified, “audio data” may refer to waveform audio data, spectrogram audio data, and / or lexicalized audio data, such as acoustic lexicals or high-fidelity audio lexicals.

[0073] Acoustic model 240a can receive audio data in spectrogram form. The audio data may include user-spoken command data 203. In some cases, additional input data 205 may include supplemental audio data to be processed by acoustic model 240a. Acoustic model 240a can encode the audio data into acoustic lexical units, and then encode them into a lexicalized acoustic representation. In some implementations, acoustic model 240a may be trained to encode audio data into acoustic lexical data in a manner that emphasizes the characteristics of content-related audio data (e.g., by jointly training acoustic model 240a with content encoder 240b, which is configured to lexicalize text input into a semantic space). Thus, acoustic lexical data may represent points in an acoustic space, a semantic space, or a joint acoustic-semantic space. In some implementations, acoustic model 240a may be trained end-to-end with LM 260, such that acoustic model 240a and LM 260 learn a shared lexical space. In some implementations, the content lexicon space can represent a vocabulary shared by acoustic model 240a and LM 260.

[0074] In some implementations, the acoustic model 240a may include a quantizer. The quantizer processes the encoded audio data and generates speech units. The encoded audio data may be continuous or discrete data representing speech in command data 203 and / or other input data 205. The quantizer may quantize the continuous or discrete data into a finite set of discrete representative vectors (e.g., centroids). These representative vectors may form speech units. Quantizing the speech representation into speech units in this manner reduces the computational resources required by the LM260 and / or other models of system 100 when processing speech. In some implementations, the quantizer may be configured and / or trained in a manner similar to a vector quantization variational autoencoder (VQ-VAE). In various implementations, speech units may correspond to a portion of audio; for example, 1, 2, 4, 8, 16 frames of audio data, etc. (e.g., where each frame of audio data corresponds to approximately 40 ms of audio; however, a spectrogram may represent audio frames with longer or shorter durations). In various implementations, speech lexical units can be integers with values ​​between zero and 2,047, 4,095, 16,383, 32,767, etc. These hyperparameters can be chosen to strike a desired balance between implementation speed, computational resources, and reconstruction accuracy. The speech lexical data learned through these training operations can represent the content and pronunciation vocabulary that the LM 260 can model.

[0075] The second input data encoder 240b may include a content encoder 240b, such as a text lexicon. The content encoder 240b may include software and / or hardware configured to convert input text data into text lexicons for processing by the LM 260. In some implementations, the content encoder 240b may generate text lexicons (e.g., words and / or phonemes, etc.) representing the content and / or pronunciation of the input text data. The content encoder 240b may include machine learning components, such as a neural network encoder and / or quantizer. In some implementations, the content encoder 240b may be trained with the LM 260 to generate content lexicons and / or acoustic lexicons. In an example implementation, the content encoder 240b may lexiconize the input text data, for example, using byte-level byte-pair encoding (BBPE) of 2051 lexicons. This number allows the content encoder 240b and / or the LM 260 to be flexible for unseen characters and / or character combinations. The input text data may be encoded using an embedding layer to generate an embedding with dimension D=1024. This embedding can be summed with learnable positional embeddings. Masked lexical units can be used to mask portions of the input text data used for training (e.g., to enable LM 260 to predict and / or replace this portion of the input text data).

[0076] The third input data encoder 240c may include a reference encoder 240c. The reference encoder 240c can generate reference embeddings, which are used by the LM 260, for example, to associate target audio with the speech characteristics of a specific speaker (e.g., for speech recognition). The reference embeddings (e.g., stored in the speech data storage component 595) can also be used by the LM 260 and / or the rendering model 380 to generate output speech with specific speech characteristics. The reference encoder 240c may be a machine learning component, such as a neural network. The reference encoder 240c may be configured to encode audio data (e.g., speech samples) into reference embedding data in a latent space, which represents a range of speech characteristics (e.g., timbre, accent, etc.) corresponding to different users. In some implementations, the reference encoder 240c may be trained using, for example, contrastive learning, such that the reference encoder 240c generates close reference embeddings (e.g., as measured using cosine similarity) for speech samples corresponding to the same speaker, and distant reference embeddings for speech samples corresponding to different speakers. In some implementations, the reference encoder 240c... (Or different input data encoders) can be configured to operate similarly to associate images with facial features of a specific user (e.g., for facial recognition), and so on.

[0077] In some implementations, the reference encoder 240c may be a transformer neural network, such as a visual transformer (ViT). A visual transformer can receive a sequence of vectors generated from fixed-size patches of an image and predict classifications. Positional embeddings can be added to the vectors, and categorical terms can be added to the sequence to make the reference encoder 240c output a classification of the input data. However, instead of processing images, the reference encoder 240c can process spectrogram data (e.g., a Mel spectrogram) representing the frequency content of an audio waveform as it changes over time. In some implementations, the reference encoder 240c may be a component of the LM 260 or the LM orchestrator 230; in other implementations, it may be a separate component.

[0078] LM orchestrator 230 can be configured to receive various input data, encode the input data using an appropriate input data encoder 240, and generate LM cues for LM 260. LM cues can be represented by input sequence 215. Therefore, LM orchestrator 230 can act as (or include) a cue generation component. Input sequence 215 will grow as system 100 generates data by performing various subtasks of complex tasks; for example, using LM 260 to perform inference and delegate subtasks to other agents and other resources. LM orchestrator 230 can begin input sequence 215 having lexicals representing instructions on how LM 260 should operate, as represented by configuration data 201. Configuration data 201 may include instructions on how to dispose of command data 203, and information for executing these instructions. In some cases, instructions may include a framework for thought chains and / or ReAct processing as previously described. In some cases, instructions may be related to delegation (e.g., delegation to an agent associated with LM 260 or delegation performed by said agent). In some cases, information can be related to the identity and capabilities of other resources, as well as the message format to be used when communicating with other resources.

[0079] Therefore, the LM orchestrator 230 can use lexicalized configuration data 221, lexicalized user speech 223 (e.g., representing command data 203), and user input data 225 (e.g., representing any other input data 205 provided for processing command data 203) to generate an input sequence 215. The LM orchestrator 230 can cause LM 260 to process the input sequence 215 to generate LM output data 245. The LM output data 245 can represent the lexical units predicted, inferred, and / or generated by processing the input sequence 215 through LM 260. LM 260 can generate the LM output data 245 in stages because the LM orchestrator 230 operates LM 260 through various subtasks of a complex task. LM 260 can operate autoregressively, such that it predicts the next lexical unit of the LM output data 245 based on the previously predicted lexical units of the input sequence 215 and the LM output data 245. As a first subtask, LM orchestrator 230 enables LM 260 to generate a transcription 227 of lexicalized user speech 223. LM 260 can process the transcription 227 and parse the task requested by the user (e.g., as represented by command data 203). See below for reference. Figure 5D The speech-to-text (e.g., ASR) operation of the multimodal LM system 100 is described in further detail.

[0080] exist Figure 2A In the example shown, LM 260 can determine a first subtask to be delegated to a resource corresponding to a specific API. LM 260 can generate API call 251 (e.g., using a format specified by configuration data 201). System 100 can send API call 251 to the API endpoint. The resource can generate API response 253, which System 100 can send back to the agent. LM orchestrator 230 can add API call 229 and API response 231 to input sequence 215 for further processing by LM 260.

[0081] LM 260 can determine a second subtask to be delegated to another agent. LM 260 can generate LM request 255 (e.g., using the format specified by configuration data 201). System 100 can send LM request 255 to the other agent. The other agent can process LM request 255 and return LM response 257. LM response 257 may also conform to the format specified by configuration data 201; therefore, LM 260 can easily process LM response 257 even if the LM response includes unstructured elements (such as natural language answers and / or requests for additional information). LM orchestrator 230 can add LM request 233 and LM response 235 to input sequence 215 for further processing by LM 260.

[0082] LM 260 can process an input sequence 215 including API and LM responses and determine that it has the information needed to generate a response to command data 203. Therefore, LM 260 can generate output content data 237, which may represent the text of a response message to be output to the user. LM 260 can process the output content data 237 to generate output speech 259. LM 260 can autoregressively generate output speech 259 by processing previously predicted output speech terms 239. When LM 260 predicts the end of the speech, LM orchestrator 230 can route output speech 259 to, for example, rendering model 380 and / or vocoder 580 for conversion into audio data that can be output to the user (e.g., via speaker 712 of user device 110). References below are respectively... Figure 5C and Figure 5D The text-to-speech and speech-to-speech operations of the multimodal LM system 100 are described in further detail.

[0083] Figure 2B This is a flowchart illustrating example operation 250 of the LM 260 and LM orchestrator 230 of the system 100 according to an embodiment of this disclosure. (See above reference) Figure 2A The described example operations can continue and / or repeat in some cases for subsequent rounds of communication between user 5 and system 100. In each round, LM orchestrator 230 can add new data segments to input sequence 215. Besides allowing system 100 to decompose complex tasks, this chain-of-thought reasoning can facilitate the use of security and / or privacy mechanisms. For example, system 100 may include mechanisms for examining data sent or received by the agent to determine if it meets runtime requirements related to content moderation, responsible AI, and / or multi-step orchestration. Additionally or alternatively, LM 260 can evaluate received responses and / or reflect on its own output. If LM 260 determines the response is insufficient, it can request further information from resources. If LM 260 determines its previous output is insufficient, system 100 can take various remedial steps, such as reconstructing prompts, obtaining additional information, and / or delegating subtasks to resources that may be more capable of generating better answers.

[0084] Operation 250 may include receiving (265) events. Events may vary, including user utterances, requests from mediators or other agents, triggering events (e.g., corresponding to routines), commands to execute routines, etc. Operation 250 may include constructing (270) prompts (e.g., input sequence 215). The LM orchestrator 230 may act as a prompt generation component, the use of which is described in the following reference. Figure 4The task generation component 435 and / or the LM candidate lister component 440 are described in further detail. The LM orchestrator 230 can compile and process information related to events, including intermediary / delegation instructions, available resources, message formats, etc., as well as the input data itself (e.g., command data 203 and / or other input data 205).

[0085] In some implementations, the LM orchestrator 230 may construct the input sequence 215 based on a paradigm. The paradigm may be a prompt template associated with the input data. The prompt template may be hard-coded with dynamic elements. For example, the prompt template may include instructions corresponding to a thought chain processing framework that does not change based on the input. The prompt template may include sections in which the LM orchestrator 230 may add input-specific information (such as available resources). For example, in the previously described mathematical calculation example, the LM orchestrator 230 may retrieve a paradigm corresponding to the user's question and, in response to the presence of numbers in the command data, add a description of a calculator tool. In the living room light example, in response to a mention of smart home functionality in the question, the LM orchestrator 230 may include a description of Alexa.

[0086] In some implementations, the LM orchestrator 230 may use a vector database to retrieve paradigms. The LM orchestrator 230 may transform events and / or input data into embeddings. Embeddings may represent features extracted from the input data using, for example, a neural network encoder. The extracted features may be related to the nature of the event or request, allowing the embeddings to be used to retrieve paradigms corresponding to similar inputs from the vector database. The transformation / encoding of the input data does not need to involve language processing itself (e.g., ASR and / or NLU). Instead, the transformation / encoding may be a relatively lightweight process capable of generating a representation of the input data, allowing the LM orchestrator 230 to have a “rough” concept of the relative nature of the input or event. The LM orchestrator 230 may use the embeddings to identify the nearest or n nearest reference embeddings in the vector database. The LM orchestrator 230 may retrieve paradigms corresponding to the nearest reference embeddings and use said paradigms to construct an input sequence. Paradigms may include instructions for agent cooperation and / or descriptions of patterns of communication with resources, as previously described. The LM orchestrator 230 can generate the portion of the input sequence 215 corresponding to the configuration data 201 in the form of text and / or text lexical metadata (e.g., as specified in the example). The LM orchestrator 230 can construct the input sequence 215 to additionally include command data 203 (e.g., which may be in the form of speech lexical units if the command is spoken to system 100) and any other input data 205. The LM orchestrator 230 can lexicalize various portions of the data using one or more of the input data encoders 240 for input to the LM 260.

[0087] The operation may include determining (275) the next subtask. LM 260 may process the input sequence 215 according to the thought chain framework to decompose the overall task into a sequence of subtasks. Although some subtasks may be executed sequentially (e.g., where subsequent subtasks are executed using the results of previous subtasks), LM 260 may be instructed to execute and / or delegate subtasks where possible to reduce latency and improve user experience. When LM 260 determines the next subtask, LM orchestrator 230 may determine (280) the task to be delegated based on LM output data 245 (e.g., based on a portion of LM output data 245 corresponding to a specified message format in a specified message format). In some cases, the agent may determine whether to delegate the task based on the capabilities of other agents described in its instructions. In some cases, the agent may send a message to another agent requesting a description of its capabilities. The agent may also send the request to one or more other available agents. In some cases, the agent's own capabilities may overlap with those of another agent. In such cases, the agent both delegates subtasks and performs them itself, comparing and / or combining the two results when generating the final response.

[0088] If the LM orchestrator 230 determines that a subtask should be delegated ("Yes" at 280), operation 250 may include sending a message (285) to a resource and observing the response. This may involve sending a message to another agent, invoking an API, and / or accessing another tool or resource. See below for reference. Figure 5A Examples of the various resources that an agent can access are described in further detail.

[0089] The LM 260 can delegate tasks by sending input data to a resource capable of processing that input data. In some cases, the LM 260 may modify or enhance the data when sending it to another agent. For example, the LM 260 may generate a message including the original command data 203 and enhance or embellish the message with context that can be used to process the task. Enhancement may include information related to the interaction history, context, and / or relevant entities. In the Robot / Alexa example described above, the Robot may embellish the command "Turn off the light" with contextual information that identifies a specific light that Alexa wants to turn off. In some cases, the LM 260 may rewrite or rebuild the command, for example, to remove information irrelevant to the delegated subtask and clarify relevant information.

[0090] In another example, enhancements could include information about an ongoing activity. For instance, if a user asks Alexa to play music and then says "stop" in the presence of a robot, the robot can consider the current context to understand what the user's command relates to. If the robot is currently stationary and not currently performing any action, it can infer that the command relates to some other action. The robot can determine that Alexa is currently performing an action (e.g., playing music); therefore, the robot can delegate the command to Alexa. The robot can generate a message for Alexa that enhances the "stop" command with relevant information, such as the user who spoke the command, contextual data indicating that the robot (e.g., via microphone 720 of mobile device 110k or other user device 110) heard the music playing, etc. Alexa can use the enhanced information in the message to determine that the delegated command relates to stopping the music.

[0091] The exchange of requests and responses to delegated subtasks can give System 100 the opportunity to impose rule-based controls on data exchanged within System 100 (e.g., between agents and resources) and / or with external systems. For example, agents and / or System 100 may have a policy library that defines what types of data can and cannot be shared between and / or outside the systems. Such prohibitions may relate differently to: vulgar, obscene, and / or illegal content; personally identifiable information; financial information such as bank account or credit card numbers; confidential healthcare information; actions prohibited by the system; child safety restrictions; and so on.

[0092] Using the received response, operation 250 can return to stage (270), where LM orchestrator 230 can construct a new prompt based on the response to the delegate request. In constructing the new prompt, LM orchestrator 230 may include the previous history of the interaction and / or any context relevant to continuing the execution of subtasks related to the overall task. In some cases, LM orchestrator 230 may shorten the prompt to remove information no longer relevant to completing the task. Including irrelevant information in the prompt can "distract" the LM, as the LM may still assign weight to irrelevant information when generating the response. Therefore, removing irrelevant information from the input sequence 215 can improve the accuracy of inference. Furthermore, a shorter input sequence 215 can reduce the latency and / or usage of computational resources by LM 260.

[0093] The LM orchestrator 230 can feed a new input sequence 215 into the LM 260. The input sequence 215 may include raw configuration data such as instructions indicating how the LM 260 should perform. In some cases, these instructions may include instructions for evaluating the response. Therefore, the LM 260 can evaluate whether the response accurately and adequately answers the original question when processing the new input sequence 215. The LM 260 can determine whether to accept the response, reject the response, or generate a subsequent response to obtain more information from the resource. The agent can respond to the resource (or intermediary agent) to indicate its acceptance / rejection of the response or to request additional information. If the agent determines that it requests more information, the system 100 can handle the request similarly to a delegated subtask by repeating stages 285, 270, and / or 275 as appropriate. In a similar manner, the LM 260 can evaluate its own output in a process called reflection. When LM 260 generates output, LM orchestrator 230 can feed back LM output data 245 along with instructions for evaluating whether LM output data 245 represents an accurate and adequate response to a task or subtask.

[0094] If LM 260 determines that there are no other subtasks to delegate ("No" at 280), the operation can proceed to generating (290) a response to the user. The agent can generate a response by generating an answer (e.g., in the form of text or content lexical units) and converting the answer into speech (e.g., in the form of speech lexical units). System 100 may render the response as appropriate.

[0095] Figure 3 This is a conceptual diagram illustrating components of a natural language processing system 300 according to an embodiment of the present disclosure. The natural language processing system 300 may employ various techniques, such as LM 260 and other models described above, to perform ASR, NLU, NLG, TTS, and / or language modeling for use as a virtual assistant system. In some implementations, LM 260 may be as follows: Figure 6The illustrated multimodal LM. User 5 can interact with the virtual assistant system using, for example, voice commands and / or natural language text input. The virtual assistant system can respond to user 5's commands using, for example, synthesized speech, natural language text on a display, and / or to perform various actions on behalf of user 5. For example, user 5 can query the natural language processing system 300, and the natural language processing system 300 can use various natural language and speech processing techniques to process the question, determine response data (e.g., an answer), and output the response data (e.g., as synthesized speech and / or text). For example, user device 110 can receive user input in the form of audio data 311 and / or text data 313. System 300 can process it to determine a response, and user device 110 can output the response in the form of audio 312 and / or text / image on display 716. In some implementations, system 300 can use an LM natural language processing pipeline to process user input to generate response output data 481 based on input data 401, as referenced below. Figure 4 Further detailed description.

[0096] Figure 3Various components of the natural language processing system 300 are shown. These components may reside on the same or different physical devices. Communication between the components may be direct or across network 199. User device 110 may include an audio capture component, such as a microphone 720 or microphone array of user device 110, which captures audio 310 and creates corresponding audio data 311. Once speech is detected in the audio data representing audio 310, user device 110 can determine whether the speech is directed to user device 110 / system component 120. In at least some embodiments, this determination may be made using a wake word detection component 320. The wake word detection component 320 may be configured to detect various wake words. In at least some examples, each wake word may correspond to the name of a different digital assistant and / or agent. One example wake word / agent name is “Alexa”. Another example wake word / agent name is “Robot”. In another example, input to the system may be in the form of text data 313, for example, as a result of a user typing input into the user interface of user device 110. Other forms of input may include indications such as that the user has pressed a physical or virtual button on user device 110, or that the user has made a gesture. User device 110 may also use its camera to capture images and may send image data 321 representing those images to system components. Image data 321 may include raw image data or image data processed by user device 110 before being sent to system components. Image data 321 may be used by different components of the system in various ways to perform operations such as determining whether the user is directing speech to the system, interpreting user commands, and responding to user commands. In some implementations, system 300 may receive and / or generate personal context data 367, which may be used to process user input, generate responses to the user input, and / or perform corresponding actions. Command data 203 and / or other input data 205 may include one or more of audio data 311, text data 313, image data 321, and / or personal context data 367 and / or determined from them.

[0097] The wake-word detection component 320 of user device 110 can process audio data representing audio 310 to determine whether speech is represented therein. User device 110 can use various techniques to determine whether the audio data includes speech. In some examples, user device 110 can apply voice activity detection (VAD) technology. Such techniques can determine the presence of speech in audio data based on various quantitative aspects of the audio data, such as the spectral slope between one or more frames of the audio data; the energy levels of the audio data in one or more spectral bands; the signal-to-noise ratio of the audio data in one or more spectral bands; or other quantitative aspects. In other examples, user device 110 can implement a classifier configured to distinguish speech from background noise. The classifier can be implemented using techniques such as linear classifiers, support vector machines, and decision trees. In still other examples, user device 110 can apply Hidden Markov Model (HMM) or Gaussian Mixture Model (GMM) techniques to compare the audio data with one or more stored acoustic models, which can include models corresponding to speech, noise (e.g., ambient noise or background noise), or silence. Still other techniques can be used to determine the presence of speech in audio data.

[0098] Wake word detection is typically performed without language analysis, text analysis, or semantic analysis. Instead, the audio data representing audio 310 is analyzed to determine whether specific characteristics of the audio data match pre-configured acoustic waveforms, audio signatures, or other data corresponding to the wake word.

[0099] Therefore, the wake word detection component 320 can compare audio data with stored data to detect wake words. One method for wake word detection uses a generalized large vocabulary continuous speech recognition (LVCSR) system to decode the audio signal, where the wake word search is performed in the resulting lattices or confusion networks. Another method for wake word detection constructs Hidden Markov Models (HMMs) for each wake word and non-wake word speech signal. Non-wake word speech includes other spoken words, background noise, etc. One or more HMMs can be constructed to model the characteristics of non-wake word speech; these HMMs are called filled models. Viterbi decoding is used to search for the optimal path in the decoding graph, and the decoding output is further processed to make a decision on the presence of a wake word. This method can be extended to include discriminative information by incorporating a hybrid DNN-HMM decoding framework. In another example, the wake word detection component 320 can be built directly on a deep neural network (DNN) / recurrent neural network (RNN) structure without involving an HMM. This architecture can be used by stacking frames within a context window for DNN or by using an RNN to estimate the posterior of the wake word with context data. Subsequent posterior threshold adjustment or smoothing is applied to make the decision. Other techniques for wake word detection, such as those known in the art, may also be used.

[0100] Once the wake-word detection component 320 detects a wake-word and / or the input detector detects input, the user device 110 can "wake up" and begin transmitting audio data 311, representing audio 310, to components of the user device 110 and / or system component 120 for processing. The audio data 311 may include data corresponding to the wake-word; in other embodiments, the portion of the audio corresponding to the wake-word is removed by the user device 110 before the audio data 311 is sent to system component 120. In the case of touch input detection or gesture-based input detection, the audio data may not include the wake-word.

[0101] In at least some embodiments, components of user device 110 (e.g., on-device components) and system component 120 may have different processing capabilities. For example, the on-device component may be configured to handle natural language user input, which may correspond to local type natural language user input (such as those controlling devices or components associated with a user's home). In such cases, the on-device component may be able to interpret and respond to local type natural language user input without incurring the latency associated with sending data to and from system component 120. If user device 110 attempts to process natural language user input that is not necessarily best suited for the on-device component, the language processing result determined by user device 110 may indicate low confidence or other indicators that the processing by user device 110 may not be as accurate as the processing done by system component 120.

[0102] In some implementations, the user device may include a hybrid selector that can handle arbitration between on-device execution and remote execution on system component 120. The hybrid selector may send audio data 311 to wake word detection component 320. Wake word detection component 320 may return an indication that no wake word was detected. In response to receiving such an indication, the hybrid selector may avoid sending audio data 311 to system component 120 and may prevent the ASR component of user device 110 from further processing the audio data 311. In this case, the audio data 311 may be discarded.

[0103] If the wake word detection component 320 detects a wake word in the audio data 311, it may send an indication of such detection to the mixing selector. In response to receiving the indication, the mixing selector may send the audio data 311 to the acoustic model 240a of system component 120 and / or user device 110. The mixing selector (or other component) may associate a unique identifier with each natural language user input. User device 110 may include the unique identifier when sending audio data 311 to system component 120, and response data from system component 120 may include the unique identifier to identify which natural language user input the response data corresponds to. The mixing selector may wait for response data from either or both of system component 120 or the local language processing component. The mixing selector may control the execution of local language processing, such as by sending “execute” and “termination” events / instructions. The “execute” event may instruct a component of user device 110 to resume any suspended execution (e.g., by instructing the component to execute according to a previously determined intent to determine the instruction). Simultaneously, a "termination" event can instruct components of user device 110 to terminate further execution, such as when user device 110 receives instruction data from system component 120 and the hybrid selector determines the instruction to be executed. Therefore, the hybrid selector prevents duplicate and / or erroneous handling of user input.

[0104] In some implementations, system 300 may include more than one system component 120. System component 120 may respond to different wake words and / or perform different categories of tasks. Each system component may be associated with its own wake word, such that speaking a specific wake word causes audio data to be sent to and processed by a specific system. For example, wake word detection component 320 detecting the wake word “Alexa” may cause audio data to be sent to a first system component 120 for processing, while wake word detector detecting the wake word “Robot” may cause audio data to be sent to a second system component 120 for processing. The system may have separate wake words and systems for different skills / systems (e.g., “Game Center” for a gaming skill / system component), and / or such skills / systems may be coordinated by one or more skill components 390a, 390b, 390c, etc. (collectively, “skill components 390”) of user device 110 and / or system component 120.

[0105] Upon receipt by system 300, audio data 311 may be sent to orchestrator component 330 and / or LM orchestrator component 230. Orchestrator component 330 may include memory and logic enabling it to transmit various data segments and forms of data to various components of the system and to perform other operations as described herein. In some embodiments, orchestrator component 330 may optionally be included in system component 120. In embodiments where orchestrator component 330 is not included in system component 120, audio data 311 may be sent directly to LM orchestrator component 230. Furthermore, in such embodiments, each component in system component 120 may be configured to interact with LM orchestrator component 230, motion plan execution component (motion plan executor) 350, API provider component, and / or other components.

[0106] The LM orchestrator 230 includes memory and logic enabling it to generate prompts for the LM 260, delegate tasks / subtasks, and perform other actions via the action plan executor 350. The LM orchestrator 230 can transmit various data fragments and data of various forms to various components of the system 100 / 300 and perform other operations as described herein. The LM orchestrator 230 can receive various inputs corresponding to user commands and create prompts for the LM 260, which are structured such that the LM 260 can recognize and parse commands, identify data to be processed, identify data representing parameters used for processing, etc. A complex task may include several subtasks. For each subtask, the LM orchestrator 230 can create a new prompt (e.g., by adding new data to a previous prompt). The new prompt can provide the LM 260 with context for executing the next subtask. For example, the first subtask may involve obtaining information from an external resource. In response to the first prompt, the LM 260 may output a call to an API and receive a response. The second subtask may involve generating a response to the output of the user 5. The LM orchestrator 230 can create a second prompt that includes an API response. In response to the second prompt, the LM 260 can generate output for user 5 based on the API response.

[0107] The LM orchestrator 230 can generate cues for complex tasks (e.g., a task involving multiple subtasks and / or a task involving processing data from different sources and / or different types of data). The LM orchestrator 230 can insert delimiter words between different data segments in the cues. Data segments may represent input received from user 5, output previously generated by LM 260, and / or data received by LM 260 from external resources. Different data segments in the cues may correspond to different data types (e.g., audio or text) or data formats (e.g., raw audio or acoustic words). Therefore, LM 260 can be trained to recognize delimiter words in the input sequence 215 (e.g., the cues) and / or generate appropriate words at appropriate positions in the LM output data 245 (e.g., the response).

[0108] The LM orchestrator 230 can recognize other actions indicated by the LM 260 and uses the action planner executor 350 to orchestrate the execution of these actions. See below for reference. Figure 4 The operation of the LM orchestrator 230 and the action plan executor 350 is described in further detail. The LM orchestrator 230 recognizes response output in the LM output data 245 and routes it to the rendering model 380 and / or the vocoder 580 for conversion into audio data 314. The rendering model 380 upsamples the relatively coarse output (e.g., speech words) of the LM 260 into a higher-fidelity representation suitable for conversion by the vocoder 580 into audio data 314 that can be output to a user. In some cases, the rendering model 380 may apply certain speech characteristics to the speech based on speaker embedding data (e.g., speech representing the target speaker, such as speech corresponding to a virtual assistant) retrieved from the speech data storage component 595. References below... Figure 5C and Figure 5D The rendering model 380 and vocoder 580 are described in further detail.

[0109] In some embodiments, system component 120 may include arbitrator component 340, which may be configured to determine whether orchestrator component 330 and / or LM orchestrator component 230 should provide a response to user 5. In some embodiments, LM orchestrator component 230 may be selected to process audio data 311 only if user 5 (or user device 110 that captured audio 310) associated with audio data 311 has previously indicated that LM orchestrator component 230 may be selected to process user input received from user 5.

[0110] In some implementations, the arbitrator component 340 may determine, based on metadata associated with the audio data 311, whether the arranger component 330 and / or the LM arranger component 230 should process the audio data 311. For example, the arbitrator component 340 may be a classifier configured to process a textual (e.g., content) representation of the audio data 311 (e.g., output by the LM component 260) and classify corresponding user input as to be processed by the arranger component 330 and / or the LM arranger component 230. As a further example, the arbitrator component 340 may determine whether the means receiving the audio data 311 therefrom is associated with an indicator indicating that the audio data 311 should be processed by the arranger component 330 and / or the LM arranger component 230. As a further example, arbitrator component 340 may determine whether a user 5 receiving audio data 311 from it (e.g., using data output from a user identification component) is associated with a user profile including an indicator indicating that the audio data 311 is to be processed by orchestrator component 330 and / or LM orchestrator component 230. As another example, arbitrator component 340 may determine whether the user has invoked (e.g., using a wake word) a virtual assistant corresponding to orchestrator component 330 or an agent corresponding to LM orchestrator 230. As yet another example, arbitrator component 340 may determine whether the audio data 311 (or corresponding content data) corresponds to a request indicating that the audio data 311 is to be processed by orchestrator component 330 and / or LM orchestrator component 230 (e.g., a request including "Let's chat" may indicate that the audio data 311 is to be processed by LM orchestrator component 230).

[0111] In some implementations, arbitrator component 340 may send audio data 311 to both orchestrator component 330 and LM orchestrator component 230. Arbitrator component 340 may do so if it determines that the confidence score corresponding to whether orchestrator component 330 and / or LM orchestrator component 230 should process the input is below a threshold. In such implementations, orchestrator component 330 and / or LM orchestrator component 230 may include further logic for determining additional confidence scores during processing that indicate whether orchestrator component 330 and / or LM orchestrator component 230 should continue processing, as discussed further below herein.

[0112] Arbitrator component 340 may send audio data 311 to acoustic model 240a. In some embodiments, components selected to process audio data 311 (e.g., orchestrator component 330 and / or LM orchestrator component 230) may send audio data 311 to acoustic model 240a. Acoustic model 240a may convert audio data 311 into input acoustic metadata for processing by LM 260. LM 260 may transcribe the input acoustic metadata into output content metadata (e.g., representing text data). Content metadata may represent one or more (e.g., in the form of an N-best list) ASR assumptions, representing the speech represented in audio data 311.

[0113] In some implementations, an ASR component (not shown) can be used to transcribe audio data 311. This ASR component can process the audio data 311 using one or more DNN ASR models. The ASR model can be, for example, a recurrent neural network, such as an RNN-T. Given acoustic features x = ( x 1, ..., x t The ASR model can predict the label y = ( y 1, ..., y uThe probability (y|x) of the output label is calculated. During inference, the ASR model can use, for example, a beam search decoding algorithm to generate an N-best list. The ASR model may include various neural network and arithmetic components, such as an encoder, a prediction network, a joint network, and a softmax. The encoder may be similar to or analogous to an acoustic model and can process the acoustic input feature sequence to generate an encoded hidden representation. The prediction network may be similar to or analogous to a language model and can process the predictions of previous output labels and map them to the corresponding hidden representations. The joint network may be, for example, a feedforward neural network (NN) that processes the hidden representations from both the encoder and the prediction network and predicts the output label probability. The softmax component may be a function implemented to normalize the predicted output probabilities (e.g., such that the sum of the probabilities is 1) (e.g., as a layer of the joint network and / or a separate arithmetic block). The ASR components send the text data to the arbitrator component 340, the arranger component 330, and / or the LM arranger component 230. When text data is sent to arbitrator component 340, arbitrator component 340 may send the text data to a component selected to process audio data 311 (e.g., arranger component 330 and / or LM arranger component 230). Text data sent from the ASR component to arbitrator component 340, arranger component 330, and / or LM arranger component 230 may include a single highest-scoring ASR hypothesis, or may include an N-best list comprising multiple highest-scoring ASR hypotheses. The N-best list may additionally include a corresponding score associated with each ASR hypothesis represented therein.

[0114] In some implementations, orchestrator component 330 may enable NLU component (not shown) to perform processing relative to ASR data generated by ASR component and / or LM 260. NLU component may attempt to semantically interpret phrases or statements represented in ASR data input to NLU component by determining one or more meanings associated with phrases or statements represented in text data. NLU component may determine an intent representing an action expected by a user and may determine information that allows a device (e.g., user device 110, system component 120, skill component 390, skill system component 325, etc.) to perform said intent. For example, if ASR data corresponds to “play Beethoven’s Fifth Symphony,” NLU component may determine the system’s intent to output music and may identify “Beethoven” as the artist / composer and “Fifth Symphony” as the piece of music to be played. As a further example, if ASR data corresponds to “what’s the weather like?”, NLU component may determine the system’s intent to output weather information associated with the geographic location of user device 110. In another example, if the ASR data corresponds to “turn off the light,” the NLU component can determine the system’s intention to turn off the light associated with device 110 or user 5. However, if the NLU component cannot resolve the entity (e.g., because the entity is referenced by anaphora such as “this song” or “my next date”), the system can send a decoding request to another speech processing system to obtain information about the entity reference and / or other context related to the utterance. The natural language processing system can enhance, correct, or base the resulting data on both the ASR data and any data received from the system.

[0115] The NLU component can return NLU result data (which may include tagged text data, intent indicators, etc.) to the orchestrator component 330. The orchestrator component 330 can forward the NLU result data to the skill component 390. If the NLU result data includes a single NLU hypothesis, the NLU component and the orchestrator component 330 can direct the NLU result data to the skill component 390 associated with that NLU hypothesis. If the NLU result data includes a list of N-best NLU hypotheses, the NLU component and the orchestrator component 330 can direct the highest-scoring NLU hypothesis to the skill component 390 associated with that highest-scoring NLU hypothesis. The system may also include an NLU post-ranker, which may incorporate additional information ranking the potential interpretations determined by the NLU component.

[0116] In some implementations, after determining that orchestrator component 330 and / or LM orchestrator component 230 should process relative to input data 401, arbitrator component 340 may be configured to periodically determine whether orchestrator component 330 and / or LM orchestrator component 230 should continue processing relative to input data 401. For example, after specific points in the processing of orchestrator component 330 (e.g., after performing NLU, before determining that skill component 390 should process relative to input data 401, before performing an action in response to user input, etc.) and / or after specific points in the processing of LM orchestrator component 230 (e.g., after selecting a task to be completed, after receiving action response data from one or more components, after completing a task, before performing an action in response to user input, etc.), orchestrator component 330 and / or LM orchestrator component 230 may query arbitrator component 340 to find that arbitrator component 340 has determined that orchestrator component 330 and / or LM orchestrator component 230 should cease processing relative to input data 401. As discussed above, system 300 can be configured to stream portions of data associated with processing relative to user input to one or more components, such that once the data becomes available to the one or more components, the one or more components can begin performing their configured processing relative to the data. Thus, once the portions of data associated with input data 401 (e.g., ASR data, context data, user identification, etc.) are available, arbitrator component 340 can cause orchestrator component 330 and / or LM orchestrator component 230 to begin processing relative to the user input. Subsequently, once arbitrator component 340 has sufficient data to perform the processing described above to determine whether orchestrator component 330 and / or LM orchestrator component 230 should process relative to the user input, arbitrator component 340 can inform orchestrator component 330 and / or LM orchestrator component 230 to continue / abort processing relative to the user input at one of the logical checkpoints in the processing of orchestrator component 330 and / or LM orchestrator component 230.

[0117] As discussed above, in some embodiments, the LM candidate lister component 440 (e.g., via an API retrieval component and / or a candidate lister language model) may be configured to select the orchestrator component 330 to process relative to user input and / or the current task, in order to return action response data (e.g., action response data 455a) that describes a response to the user input / current task or an action that the orchestrator component 330 may be caused to perform in response to the user input / current task. Therefore, in some embodiments, although the LM orchestrator component 230 is determined to process relative to user input, the LM orchestrator component 230 may determine during such processing that the orchestrator component 330 should process relative to user input.

[0118] Skill system component 325 can communicate with skill / app component 390 within system component 120, directly with orchestrator component 330 and / or action plan executor 350, or with other components. Skill system component 325 can be configured to perform one or more actions. The ability to perform such actions is sometimes referred to as a "skill." That is, a skill enables skill system component 325 to perform specific functionalities to provide data or perform another action requested by the user. For example, a weather service skill enables skill system component 325 to provide weather information to system component 120, a car service skill enables skill system component 325 to book a trip relative to a taxi or ride-sharing service, and a pizza ordering skill enables skill system component 325 to order pizza relative to a restaurant's online ordering system, etc. Additional skill types include home automation skills (e.g., skills that enable users to control home appliances such as lights, door locks, cameras, thermostats, etc.), entertainment device skills (e.g., skills that enable users to control entertainment devices such as smart TVs), video skills, newsletter skills, and custom skills that are not associated with any pre-configured skill types.

[0119] System component 120 may be configured with a skill component 390 specifically for interacting with skill system component 325. Unless otherwise explicitly stated, references to skill, skill device, or skill component may include skill component 390 operated by system component 120 and / or skill operated by skill system component 325. Furthermore, the functionality of a skill described herein, or a skill, may be referred to using many different terms such as action, speech, robot, app, etc. Skill component 390 and / or skill system component 325 may return output data to orchestrator component 330.

[0120] User device 110 may include still images and / or video capture components that capture one or more images, such as a camera 718 or multiple cameras. User device 110 may include circuitry for digitizing images and / or video for transmission as image data to system component 120. User device 110 may further include circuitry for voice command-based control of the camera, thereby allowing user 5 to request the capture of image or video data. User device 110 may process commands locally or send audio data 311 representing commands to system component 120 for processing, after which system component 120 may return output data that enables user device 110 to activate its camera.

[0121] In at least some embodiments, system component 120 may receive audio data 311 from user device 110, process speech corresponding to spoken input in the received audio data 311, and perform functions in response to recognized speech. In at least some embodiments, these functions may involve sending instructions (e.g., commands) from the system component to user device 110 (and / or other user device 110) to cause user device 110 to perform actions, such as outputting an auditory response to spoken input via a speaker, and / or controlling auxiliary devices in the environment by sending control commands to auxiliary devices.

[0122] Therefore, when user device 110 is able to communicate with system component 120 via network 199, some or all of the functions that system component 120 can perform can be executed by sending one or more instructions via network 199 to user device 110, which in turn can process the instructions and perform one or more corresponding actions. For example, using remote instructions included in response data (e.g., remote response), system component 120 can instruct user device 110 to output an audible response to a user's question via the speaker of user device 110 (or otherwise associated with it) (e.g., using speech synthesis processing performed by the LM260 and / or the TTS component on the device), output content (e.g., music) via the speaker of user device 110 (or otherwise associated with it), display content on the display of user device 110 (or otherwise associated with it), and / or send instructions to auxiliary devices (e.g., instructions to turn on a smart light). It should be understood that system components may be configured to provide functions other than those discussed herein, such as, but not limited to, providing step-by-step guidance from the starting point to the destination, conducting e-commerce transactions on behalf of user 5 as part of a shopping function, establishing communication sessions (e.g., video calls) between user 5 and another user, etc.

[0123] In at least some embodiments, user device 110 may include or be configured to use one or more skill components, which may function similarly to skill component 390 implemented by system components. Skill components may correspond to one or more domains used to determine how to act on verbal input in a particular manner, such as by outputting instructions that correspond to a determined intent and can be processed to achieve the desired operation. Skill components installed on user device 110 may include, but are not limited to, smart home skill components (or smart home domains) and / or device control skill components (or device control domains) executed in response to verbal input corresponding to an intent of a second device in the control environment, music skill components (or music domains) executed in response to verbal input corresponding to an intent to play music, navigation skill components (or navigation domains) executed in response to verbal input corresponding to an intent to obtain direction, shopping skill components (or shopping domains) executed in response to verbal input corresponding to an intent to purchase items from an electronic marketplace, and / or similar components.

[0124] Additionally or alternatively, user device 110 may communicate with one or more skill system components 325. For example, skill system components 325 may be located in a remote environment (e.g., a separate location), allowing user device 110 to communicate with skill system components 325 solely via network 199. However, this disclosure is not limited thereto. For example, in at least some embodiments, skill system components 325 may be configured in a local environment (e.g., a home server and / or similar environment), allowing user device 110 to communicate with skill system components 325 via a private network (such as a local area network (LAN)).

[0125] In some implementations, system 300 may include a TTS component (not shown). The TTS component may use one or more different methods to generate audio data (e.g., synthesized speech) from text data (e.g., content-based word data). The text data input to the TTS component may come from LM 260, skill component 390, orchestrator component 330, or another component of the system. The TTS component may include a preprocessing component for converting text data and / or other input data into a form suitable for processing using various TTS technologies. The preprocessing component may include functionalities and / or components for performing text normalization, language analysis, prosodic generation, or other such operations. During text normalization, the preprocessing component may first process the text data and generate standard text, converting content such as numbers, abbreviations (e.g., Apt., St., etc.), symbols ($, %), etc., into equivalents of written words. During language analysis, the preprocessing component may analyze the language in the normalized text to generate a sequence of speech units corresponding to the input text. This process may be referred to as grapheme-to-phoneme conversion. Speech units include symbolic representations of sound units that will ultimately be combined by the system and output as speech.

[0126] The output of the preprocessing component can be a symbolic language representation that may include sequences of speech units. The TTS component can retrieve one or more previously trained and / or configured TTS models from a speech profile store. A TTS model can be, for example, a neural network model, which can be described as interconnected artificial neurons or "units" linked by layers and / or blocks. Generally, the neural network model architecture can be broadly described using hyperparameters that describe the number of layers and / or blocks, how many units are contained in each layer and / or block, what activation functions they implement, how they are interconnected, etc. The neural network model includes trainable parameters (e.g., "weights") that indicate how much weight a unit should give to a particular input when generating output (e.g., in the form of an arithmetic multiplier). In some implementations, the neural network model may include other features such as a self-attention mechanism, which can determine certain parameters at runtime based on the input rather than, for example, based on loss calculations during training.

[0127] A TTS model can represent a specific speaker identity and can be adjusted based on speaking style, emotion, etc. In some implementations, a specific speaker identity can be associated with more than one TTS model; for example, with different models representing different speaking styles, languages, emotions, etc. In some implementations, a specific TTS model can be associated with more than one speaker identity; that is, it can generate synthesized speech that reproduces the speech characteristics of more than one person. Therefore, a first TTS model can be used to create synthesized speech for a first natural language processing system component 120, while different second TTS models can be used to create synthesized speech for a second natural language processing system component 120. In some cases, the TTS model can generate desired speech characteristics based on conditional data received or determined from text data and / or other input data. In some implementations, the TTS component can synthesize speech using a method called unit selection. In unit selection, the TTS component matches text data with a database of recorded speech. The TTS component can select matching units of the recorded speech and concatenate these units together to form audio data. In some implementations, TTS components can use a method called parametric synthesis to synthesize speech. The TTS component can change parameters such as frequency, volume, and noise to create audio data that includes an artificial speech waveform. The output of some speech synthesis techniques may include spectrogram data, which represents the energy content of each frequency band in “frames” of audio data. A frame of audio data can represent several milliseconds (e.g., 10, 20, 30, etc.) of audio data.

[0128] The TTS component can use a computerized voice generator (sometimes called a vocoder) to convert spectrogram data into waveform data. The vocoder can be, for example, a general-purpose neural vocoder based on Parallel WaveNet or a related model. The vocoder can take audio data as input, for example, in the form of a Mel spectrogram with 80 coefficients and a frequency range of 50 Hz to 12 kHz. The vocoder can process the spectrogram data and convert it into a time-domain audio format (e.g., Pulse Code Modulation (PCM), Waveform Audio Format (WAV), μ-law, etc.), which can be easily converted into an analog signal for amplification and output by a speaker. The resulting audio data can consist of 8-bit, 16-bit, or 24-bit audio with sampling rates such as 16 kHz, 24 kHz, 44.1 kHz. In some implementations, other bit and / or sampling rates can be used. A digital-to-analog converter (DAC) converts the audio data into an analog signal suitable for amplification and outputs it as audio through a speaker 712 (such as speaker 712 of user device 110).

[0129] System 300 (on user device 110, system component 120, or a combination thereof) may include profile storage 370 for storing various information related to individual users, user groups, devices, etc., who interact with the system. As used herein, "profile" refers to a dataset associated with a user, user group, device, etc. Profile data may include user, device, etc., specific preferences; device input and output capabilities; internet connectivity information; user bibliographic information; subscription information, and other information.

[0130] Profile storage 370 may include one or more user profiles, each associated with a different user identifier / user profile identifier. Each user profile may include various user identification data. Each user profile may also include data corresponding to the user's preferences. Each user profile may also include one or more device identifiers representing the user's presence and / or one or more devices representing the user. For example, a user account may include one or more Internet Protocol (IP) addresses, Media Access Control (MAC) addresses, and / or device identifiers (such as serial numbers) for each attached electronic device associated with the identified user account. When a user logs in to an application installed on user device 110, the user profile (as associated with the presented login information) may be updated to include information about user device 110, such as indicating that the device is currently in use. Each user profile may include identifiers of skills that the user has enabled. When a user enables a skill, the user grants permission to system components to allow the skill to be executed relative to the user's natural language user input. If the user has not enabled a skill, the system components may not invoke the skill to execute relative to the user's natural language user input.

[0131] Profile storage 370 may include one or more group profiles. Each group profile may be associated with a different group identifier. A group profile may be specific to a group of users. That is, a group profile may be associated with two or more individual user profiles. For example, a group profile may be a family profile associated with user profiles associated with multiple users in a single household. A group profile may include preferences shared by all user profiles associated with it. Each user profile associated with a group profile may additionally include preferences specific to the user associated with it. That is, each user profile may include preferences unique to one or more other user profiles associated with the same group profile. User profiles may be independent profiles or may be associated with group profiles.

[0132] Profile storage 370 may include one or more device profiles. Each device profile may be associated with a different device identifier. Each device profile may include various user identification information. Each device profile may also include one or more user identifiers representing one or more users associated with the device. For example, a profile for a home device may include user identifiers of users in the home.

[0133] although Figure 3 The components may be exemplified as system component 120, user device 110, or other components, but without departing from this disclosure, these components may be arranged in other devices (e.g., if exemplified in system component 120, arranged in user device 110, or vice versa, or arranged together in other devices). Furthermore... Figure 3 The various components shown and / or the functions they perform can be copied, partitioned and / or shared between system component 120 and user device 110.

[0134] Figure 4 This is a conceptual diagram illustrating in further detail the LM component of a Natural Language Processing (NLP) system 300 according to an embodiment of the present disclosure. System 300 may include a user device 110 local to user 5, which communicates with system component 120 via network 199. Network 199 may include the Internet and / or any other wide area network (WAN) or local area network (LAN), and may include wired, wireless, and / or cellular network hardware. System component 120 may include various components such as an LM orchestrator component 230, a personalized context component 465, and an action plan executor 350. LM orchestrator component 230 may include a task generation component 435, an LM candidate lister component 440, and a response arbitration component 460. In some embodiments, one or more of the LM orchestrator component 230, task generation component 435, LM candidate lister component 440, and / or response arbitration component 460 may correspond to a trained LM. In some implementations, one or more of the LM orchestrator component 230, task generation component 435, LM candidate lister component 440, and / or response arbitration component 460 may invoke LM 260 to perform inferences relative to the input sequence 215.

[0135] In some implementations, LM orchestrator component 230 may generate an input sequence 215 representing a prompt to LM 260 (and / or other LMs of system 100 / 300). System component 120 may receive input data 401, which may be provided to LM orchestrator component 230. As discussed above, in some cases, input data 401 may correspond to various data types, such as text (e.g., text or lexicalized representation of user input), audio, images, video, etc. For example, when the user input is typed natural language user input, the user input data may include input text (or lexicalized) data. For a further example, before LM orchestrator component 230 receives input data 401, another component of system 300 (e.g., LM 260 and / or ASR component) may receive audio data representing user input. LM 260 and / or ASR component may perform speech-to-text processing on the audio data to determine ASR data corresponding to the user input, which may correspond to a transcription of the user input. As discussed above... Figure 3 As described, the ASR component can determine ASR data comprising an optimal list of ASR N, said optimal list including multiple ASR hypotheses representing what the user might have said and corresponding confidence scores. ASR hypotheses may include textual data representing the input utterance, lexical data, ASR confidence scores, etc. The confidence score of each ASR hypothesis can indicate the ASR component's confidence level that the corresponding hypothesis represents what the user said. The ASR component can also determine a lexical score corresponding to each lexical / word of the ASR hypothesis, wherein the lexical score indicates the ASR component's confidence level for the corresponding lexical / word spoken by the user. When the corresponding lexical is associated with an entity, the lexical score can be identified as an entity score. In some cases, input data 401 may include the highest-scoring ASR hypothesis of the ASR data. As a further example, in some embodiments, user input may correspond to the actuation of a physical button, data representing the selection of a button displayed on a graphical user interface (GUI), image data of gestural user input, combinations of different types of user input (e.g., gestures and button actuation), etc. In such implementations, system 300 may include one or more components configured to process such user input to generate a textual or lexical representation of the user input (e.g., input data 401).

[0136] In some embodiments, the LM orchestrator component 230 may receive additional input data (e.g., additional input data 205), which is processed in a manner similar to input data 401 as described herein. Input data may be received in response to the detection of an event, such as a change in device state (e.g., front door opening, garage door opening, television turning off, etc.), the occurrence of an acoustic event (e.g., baby crying, appliance beeping, etc.), or the presence of a user (e.g., a user approaching user device 110, a user entering the home, etc.). In some embodiments, the system 300 may process the input data and generate a response / output. For example, input data may be received in response to the detection of a general or specific user, a timer expiring, the time of day, a detected weather change, a change in device state, etc. In some embodiments, input data may include data corresponding to an event, such as sensor data (e.g., image data, audio data, proximity sensor data, short-range wireless signal data, etc.), descriptions associated with timers, the time of day, descriptions of weather changes, indications of device state changes, etc. The system 300 may include one or more components configured to process the input data to generate a natural language representation of the input data. System 300 can process input data and perform actions. For example, in response to detecting that the garage door is open, system 300 can turn on the garage light, the living room light, etc. As another example, in response to detecting that the oven is beeping, system 300 can use user device 110 (e.g., smartphone, smart speaker, etc.) to issue an alarm to the user. LM orchestrator component 230 can process input data to generate tasks that cause the aforementioned example actions to be performed.

[0137] Input data 401 may be received at the task generation component 435 of the LM orchestrator component 230, which may be configured to generate a list of subtasks (e.g., steps / actions) (e.g., one or more of the subtasks) that need to be completed before an action in response to user input can be performed, and to select the task in the task list to be completed first (e.g., in the current iteration of the processing performed by the LM 260). For example, for the user input “What’s the weather like today?”, the task generation component 435 may generate a list of subtasks “(1) Determine the current outdoor temperature from the thermostat; and (2) Determine today’s weather forecast”, and select the “Determine today’s weather forecast” subtask to be completed first. If the task generation component 435 generates more than one subtask that needs to be completed before an action in response to user input can be performed, the task generation component 435 may further maintain the list of subtasks and prioritize them. In other words, while system 300 is processing to complete the subtask list, task generation component 435 may (1) incorporate potential responses associated with completed subtasks into data provided to other components of system 300; (2) update the subtask list to indicate completed (or attempted, in progress, etc.) subtasks; (3) generate an updated priority order of subtasks still to be completed (or tasks to be retried); and / or (4) determine the current subtask to be completed. Task generation component 435 may generate task data 437 representing the selected subtasks to be completed, as well as various other information required for further processing relative to task execution (e.g., input data 401, indication of the selected subtasks, potential responses associated with previous subtasks, remaining subtasks, and context data associated with input data 401), and send them to LM candidate lister component 440.

[0138] The LM candidate lister component 440 may be configured to identify one or more components (e.g., response components and / or external resources, such as other agents, tools, APIs, skill components 390, language model agent components, TTS components, etc.) configured to perform actions related to user input or the current subtask. The LM candidate lister component 440 may be further configured to generate requests (e.g., API calls, incomplete API calls / API call formats, instructions for actions to be performed by the components, etc.) for one or more components that provide potential responses to user input or the current subtask (e.g., responses to user-provided questions, paragraphs from websites, etc.) and cause the execution of said requests. This may further include potential actions that the components are configured to / will perform relative to user input or the current task (e.g., descriptions of potential actions, such as turning on lights, booking airline tickets, ordering pizza, etc.). For example, for the current task of “determining today’s weather forecast,” the LM candidate lister component 440 may generate requests such as “determine today’s weather forecast using weather app A” and “determine today’s weather forecast using weather app B.” Such requests can be represented in action plan data 442 sent to action plan executor 350. Action plan executor 350 can identify requests in action plan data 442, generate executable API calls corresponding to the requests, and cause corresponding components (e.g., response components, such as API provider components, language model agent components, skill components 390, TTS components, etc.) to generate action response data 455a-n representing the requested potential responses, wherein individual action response data 455a may be provided by / correspond to a specific response component (one of the API provider component, language model agent component, skill component 390, and / or TTS component). In some embodiments, action response data 455a-n may include identifiers of the components providing the data (e.g., component names, alphanumeric values ​​associated with the component, etc.). LM candidate lister component 440 receives and processes action response data 455a-n and generates potential response data 443a-n representing potential responses (e.g., relevant potential responses, selected potential responses, ranked potential responses, etc.) for further processing. If the LM candidate lister component 440 determines that there are no remaining tasks available for generating potential responses, the LM candidate lister component 440 may send the potential response data 443a-n to the response arbitration component 460.

[0139] In some implementations, potential response data 443a-n may be determined based on potential responses received from various different components that may be associated with response input data 401. For example, potential response data 443a-n may include a first potential response from a first component configured to perform a first task determined by task generation component 435 (e.g., response component), a second potential response from a second component configured to perform a second task determined by task generation component 435 (e.g., response component), and so on. Potential response data 443a-n may include more than one potential response associated with an individual task. In some implementations, potential response data 443a-n may be natural language data.

[0140] Response arbitration component 460 processes potential response data 443a-n to determine whether potential responses generated for one or more tasks respond to user input. Response arbitration component 460 processes potential response data 443a-n (at least representing the generated potential responses) and selects one or more potential responses determined to respond to user input and / or determines that none of the actions respond to user input. For example, response arbitration component 460 may process potential response data 443a-n to determine whether one or more potential responses (e.g., potential responses and / or potential actions) that can be executed by an API respond to the current task. In some embodiments, response arbitration component 460 may generate a natural language summary of one or more of the selected responses and output the natural language summary. For example, given the user input "What's the weather like today?" and potential responses such as "Today's high is 75 degrees Fahrenheit and the low is 68 degrees Fahrenheit" and "Today's weather will be mostly sunny with a chance of light rain tonight," the response arbitration component 460 can generate a natural language summary such as "Today's weather is expected to be mostly sunny with a high of 75 degrees Fahrenheit and a low of 68 degrees Fahrenheit, with a chance of light rain tonight." In some implementations, when the LM orchestrator component 230 determines a personality associated with the user input, the response arbitration component 460 can further generate a natural language summary in a style corresponding to said personality.

[0141] In some implementations, a language model (e.g., LM 260 and / or the LM associated with LM orchestrator component 230, task generation component 435, LM candidate lister component 440, and response arbitration component 460) can be fine-tuned to perform a specific task. One or more techniques can be used to perform fine-tuning of the language model (e.g., LM orchestrator component 230, task generation component 435, LM candidate lister component 440, and response arbitration component 460). One example fine-tuning technique is transfer learning, which involves reusing the weights and architecture of a pre-trained model for new tasks. The pre-trained model can be trained on large, general-purpose datasets, and transfer learning methods allow for efficient and effective adaptation to a specific task. Another example fine-tuning technique is sequential fine-tuning, where the pre-trained model is fine-tuned sequentially on multiple related tasks. This allows the model to learn more nuanced and complex language patterns across different tasks, resulting in better generalization and performance. Yet another fine-tuning technique is task-specific fine-tuning, where the pre-trained model is fine-tuned on a specific task using a task-specific dataset. Another fine-tuning technique is multi-task learning, where the pre-trained model is fine-tuned simultaneously on multiple tasks. This approach allows the model to learn and utilize shared representations across different tasks, resulting in better generalization and performance. Yet another fine-tuning technique is adapter training, which involves training lightweight modules inserted into the pre-trained model, allowing for fine-tuning on specific tasks without impacting the performance of the original model on other tasks.

[0142] In some implementations, one or more components of the system 300 discussed above may be configured to begin processing relative to the data once the data or a portion of the data becomes available to the one or more components. Some components of system 300 are generative components / models that can begin processing relative to a portion of the data as soon as the data portion becomes available, rather than waiting until the entire data becomes available. In other words, system 300 may be configured to stream portions of data associated with processing relative to user input to one or more components such that once the data becomes available to the one or more components, the one or more components can begin performing their configured processing relative to the data. For example, if the output of task generation component 435 and / or LM candidate lister component 440 indicates that additional information is needed to complete a first task associated with user input, a request for additional information (which may be returned as personalization context data 367) may be sent to personalization context component 465. Thereafter, task generation component 435 and / or LM candidate lister component 440 may continue processing to complete their configured operations. For example, while personalization context component 465 is processing to determine additional information, system 300 may begin processing relative to a second task associated with user input. Subsequently, the output of the personalized context component 465 can be sent to the response arbitration component 460, so that once the response arbitration component 460 receives the output of the LM candidate lister component 440, the response arbitration component 460 can resolve ambiguities arising from the request for additional information in order to generate response output data 481. For a further example, if the input data 401 is generated as a natural language representation including user input, but the processing required to determine the corresponding context signal (e.g., weather data, time data, conversation history, device information, etc.) has not yet been completed, the task generation component 435 can begin processing relative to the natural language representation of user input. Once the corresponding context signal has been generated, the task generation component 435 can begin processing relative to the context signal and can use the results of processing relative to the context signal to update downstream components.

[0143] As another example, if task generation component 435 determines that more than one task needs to be completed to perform an action in response to user input, and LM candidate lister component 440 processes as described above to generate potential responses for one or more components relative to a first task among the more than one task, then LM candidate lister component 440 may send the potential responses (along with a representation of the user input and the current task) to response arbitration component 460 for processing as described above with respect to these potential responses, while system 300 (e.g., task generation component 435 and / or LM candidate lister component 440) completes processing relative to the remaining tasks among the one or more tasks. Therefore, response arbitration component 460 may process as described above to select among the potential responses associated with the first task, while completing the potential responses associated with one or more of the remaining tasks. Thus, when response arbitration component 460 later processes additional potential responses associated with a further task, response arbitration component 460 may only need to arbitrate among the potential responses associated with the first task that were previously selected by response arbitration component 460 as responses to the first task.

[0144] As described above, the LM 260 can be configured to perform processing on lexical units representing input speech, such as segmentation. In some implementations, the LM 260 can be further designed to process, understand, and / or generate multimodal data including audio, text, images, video, and / or other types of data. Therefore, the LM 260 can be built using deep learning techniques such as neural networks and can be trained for natural language processing on a wide range of datasets including text (or other types of data, such as multimodal data including text, audio, images, video, etc.) from a broad range of sources, such as old / licensed books, images, videos, websites, etc. Therefore, System 300 can operate on various generative models, including speech-to-speech models (which can process audio data and generate audio embedding data / audio lexical units that can be used to generate synthetic speech), text-to-speech models (which can process text data or other text representations and generate audio embedding / lexical units), speech-to-text models (which can process audio data and generate text data or other text representations), image-to-text models (which can process image (or video) data and generate text data or other text representations), text-to-image data (which can process text or other text representations and generate image (or video) data), multimodal generative models (which can process one or more types of input data (e.g., text, audio, and / or images) and generate one or more types of output data (e.g., text, audio, and / or images)) and other types. Figure 6 The diagram illustrates this multimodal LM capable of performing segmentation processing.

[0145] Figure 5A Example operations generated by prompts in an LM system 100 according to an embodiment of this disclosure are illustrated. Figure 5A In the example operation shown, LM orchestrators 230 and LM 260 are processing two different segments of input data; however, the same operation can be extended to more segments and / or different data modalities. LM orchestrator 230 can receive command data 203 and other input data 205. Command data 203 can be presented as lexical units U1, U2, U3… U n Similarly, other input data 205 can be presented as tokens I1, I2, I3… I n In some cases, the LM orchestrator 230 uses one of the input data encoders in the input data encoder 240 to convert other input data 205 into another form to generate processed input data I'1…I' n For example, other input data 205 may represent raw waveform audio data, and the LM arranger 230 may send it to the acoustic model 240a for conversion into audio terminology I'1, I'2, I'3… I' n Despite Figure 5A The example operation is not shown, but system 100 may additionally or alternatively convert command data 203.

[0146] The LM orchestrator 230 can determine position data corresponding to each input (e.g., command data 203, other input data 205, etc.). Position data can indicate the order in which data is received. Position data can indicate the order in which the LM 260 can process inputs; that is, if the first input is used to process the second input, the position data can reflect that the first input should be placed at the beginning of the input sequence and the second input should be placed after the first input. For more complex input sequences, such as when the LM 260 processes data in multiple rounds, the position data can indicate that data output by the LM 260 in a previous round should be appended to the end of the input sequence in the next round. If new data is received between rounds, such as when the LM 260 generates a request to an API or another LM, and the API / LM returns a response, the position data for the request and response can indicate that the request should be appended to the previous sequence, followed by the response.

[0147] The LM orchestrator 230 can combine command data 203 and processed input data 205 into an LM input sequence 501. The input sequence 501 may include delimiter words… <m>"Separated command data 203 words U1... U n And the processed input tokens I'1… I' n . <m>Lexical units can indicate data followed by different segments and / or modalities. The LM 260 can process input sequence 501 to generate output data 565, which can be generated from data units O1…O n The LM orchestrator 230 can route output data 565 to one or more downstream processes 550, which may include further processing, communication with other components of the system, and / or outputting responses to the user. Downstream processes 550 may include operations performed, for example, by action plan executor 350, and / or external resources (such as one or more delegated agents 560, and / or tools). In the case of delegated tasks / subtasks, resources may return data to the agents, and the LM orchestrator 230 may generate new input sequences 501 for processing by LM 260.

[0148] When performing complex tasks, previous outputs of LM 260 can be added to a prompt for performing the next subtask. Therefore, LM orchestrator 230 can add delimiter words to the next input sequence 501. <o>", followed by the output word O1... O n .

[0149] In some implementations, the LM orchestrator 230 may create an input sequence 501 containing various other types of data, including representations of natural language input from a user (e.g., in the form of acoustic lexical metadata and / or content lexical metadata), machine-generated instructions, contextual data (e.g., data from the user device that received the input, from one or more system components, and / or from another or more user devices), previous input and / or output data, previous API calls and / or observations, and / or information in other formats of data that the LM 260 has been configured to process (e.g., gestures represented in video data, identifiable faces or objects in image data, single or time-series measurements performed using sensors, etc.). The LM orchestrator 230 may delimit cue words, acoustic words, and / or any combinations of words previously predicted by the LM 260 using appropriate delimiter words.

[0150] In some implementations, LM 260 can be a multimodal LM. A multimodal LM can process and output more than one type of data simultaneously. For example, in addition to content data (e.g., representations of words / text), a multimodal LM may also be able to process audio data, video data, sensor data, image data, and / or data of other modalities. Data may be lexicalized (e.g., encoded into a series of discrete values), encoded into embeddings, and / or presented in its original form (e.g., unlexicalized, unencoded, and / or otherwise presented in the same or similar format as when it was received and / or created). A multimodal LM can perform speech-to-speech operations and other operations, where the model processes representations of input audio (e.g., speech) and / or other types of content (e.g., text, images, etc.) to generate representations of output audio and / or other types of content. As an extension of such speech-to-speech functionality, a multimodal LM may be able to perform ASR (where the multimodal LM receives a speech representation and outputs a text representation), TTS (where the multimodal LM receives a text representation and outputs a synthesized speech representation), machine translation (where the multimodal LM receives a speech representation of a first natural language and outputs a synthesized speech representation of a second natural language), and / or other functions.

[0151] Multimodal learning machines (LMs) can use these voice-to-voice capabilities to perform the functions of conversational agents or other types of chatbots that can converse with users via voice, text, and / or other data modalities. For example, a user can communicate with the LM using voice and / or text. The user can input images and / or other data and request the LM to answer questions about the input and / or perform other processing on the input. For example, a user can upload a document and request a summary from the LM. In some cases, the LM can translate the document and / or summary into another language. The user can request the LM to compile a list of unread emails (e.g., accumulated during vacation) and describe these unread emails to the user in order of urgency and / or importance. Furthermore, see reference below. Figures 5B to 5D and Figure 6 Provide a detailed example of multimodal operation.

[0152] In some implementations, the LM 260 can be configured to operate in a part-by-part manner, where the LM is able to process a portion of the input and generate a corresponding portion of the output before receiving subsequent portions of the input. For example, during runtime / inference, each part of the input acoustic lexicon metadata (e.g., representing a series of audio data frames) can be fed to the LM 260 and added to the running autoregressive input (e.g., in speech-to-text, it may also contain previously predicted output content lexicons). The LM 260 can decode the transcription represented in the part. The LM 260 can predict word segments one by one until it reaches the end lexicon of the part. At this point, the decoding algorithm of the LM 260 can stop prediction until the LM 260 receives the next part of the input acoustic lexicon metadata. Part-by-part processing can be used to improve the latency of certain operations of the LM 260 when processing data received over a relatively long period of time compared to the processing time of the LM 260 (such as speech received over a few seconds).

[0153] To achieve this functionality, the LM 260 can be trained to include one or more types of delimiter words in its vocabulary (e.g., delimiters indicating part ends and / or separations between words representing different modalities). Several other aspects of the system 100 can be configured to allow the LM 260 to perform part-by-part processing. First, the LM 260 can be trained to predict the end of a part. The LM 260 can operate in an autoregressive manner, where it receives input and predicts an output sequence of words (or, in the case of multimodal operation, multiple parallel output sequences), where the LM 260 uses the input data and previously predicted output words to predict each consecutive output word. The LM 260 can be trained to determine when an output sequence of all words representing the current (and preceding) part of the input data has been generated. Upon making this determination, the LM 260 can output a word indicating the end of a part. The part-end word can be used as (and / or trigger) a request for the next part of the input data.

[0154] Second, upstream components of LM 260 can be configured to generate portions of input data for processing by LM 260. For example, LM orchestrator 230 may have input audio data (e.g., command data 203) processed by acoustic model 240a. Acoustic model 240a may process the input audio data in portions (e.g., corresponding to a certain number of audio frames / audio words, and / or corresponding to a portion of speech, up to a potential endpoint determined by acoustic model 240a based on, for example, a pause of a certain length in the speech). Acoustic model 240a may be configured to determine word size and / or portion size. Word size may correspond to the number of audio frames represented by a single word.

[0155] Acoustic model 240a can determine the part size in various ways (e.g., in terms of the number of lexical units, frames, and / or audio milliseconds). For example, acoustic model 240a can determine the part size based on policy data (e.g., stored in policy data storage component 635). The policy data can specify the part size for a given task; for example, a first part size can be used for low-latency tasks, such as speech recognition for real-time hidden word annotation, a second part size larger than the first part size can be used for natural language command processing and / or chatbot applications, and a third part size larger than the second part size can be used for machine translation tasks, where more look-ahead information improves translation accuracy. In some cases, acoustic model 240a can automatically determine the appropriate part size based on the task (in some cases, this may include processing a certain amount of input speech or text to identify the task to be performed). In some cases, acoustic model 240a can determine the appropriate part size according to the modality involved; for example, according to whether the modality is speech-to-text, text-to-speech, or speech-to-speech. In some cases, acoustic model 240a may select an appropriate portion size based on characteristics of the input (e.g., the number of speakers, speaker volume and / or background noise, whether the speakers are hissing, etc.). In some cases, the portion size may be specified by the user; for example, for a specific task or as a global preference setting. In some cases, acoustic model 240a may determine the portion size based on some or all of the above factors. In any case, acoustic model 240a may generate a portion and return it to LM orchestrator 230 for prompting LM 260 before the end of the input speech.

[0156] Determining the end of speech can be referred to as "voice endpoint determination." Endpoint determination may involve waiting for a period of time after the user has finished speaking and / or processing the received speech to determine whether the speech could represent a complete utterance (e.g., including information sufficient to identify and perform the requested action), or whether the user paused in the middle of a sentence (e.g., thinking of the correct word, due to an interruption, or otherwise) and intends to continue speaking. In some implementations, endpoint decisions may be made by the LM260 based on the content of the speech. In some implementations, endpoint decisions may be made by one or more trained models, and / or based on the output of one or more trained models. For example, the endpoint decision model may receive one or more of the following: LM output data, ASR result data, NLU output data, conversation context, skill results, user data, etc., and determine whether it is possible to reach a voice endpoint at any time. Endpoint decisions may cause the system to perform other actions, such as muting the microphone, changing indicators (e.g., lights), executing commands, outputting responses, entering sleep mode, etc.

[0157] Third, the LM orchestrator 230 can be configured to generate LM cues by sequentially arranging the data portions generated by the acoustic model 240a and the data portions generated by the LM 260 (e.g., based on the previous data portions generated by the acoustic model 240a).

[0158] In the implementation where LM orchestrators 230 and LM 260 process the input data stream in a part-by-part manner, LM orchestrators 230 and LM 260 may use delimiter terms as indications of part ends, which are received from acoustic model 240a or predicted by LM 260. In some cases, <m>or <o>Lexical terms can be used to indicate to LM arranger 230 or LM 260 that the following lexical term corresponds to the next part of the input. If the LM 260 outputs a term that ends, it may trigger LM arranger 230 to send the next part of the input data for processing. A delimiter term at the end of the next input sequence 501 can indicate to LM 260 to begin predicting output data 565 corresponding to a new part of the input data, and so on.

[0159] In various implementations, the size of the data portion can range from several frames (e.g., where one frame of audio data represents 10 ms, 20 ms, 30 ms, or 40 ms of audio) to several seconds. The LM 260 can be trained using aligned datasets where the beginnings and ends of words are precisely labeled. These labels allow training to associate a portion with the word it represents. Training teaches the LM 260 to predict the end of a portion. The portion end prediction can be used to request the next portion of the input data. For example, in a speech-to-speech implementation, the portion end prediction can be sent to the audio encoder to request the next portion of the audio data. The LM 260 can be trained using portions of random length. This allows the system to be post-configured for any portion size appropriate for a given task. For example, a larger portion size can improve accuracy, while a smaller portion size can reduce latency.

[0160] During inference, the portion size can be selected by the user, pre-set for the requested task, and / or dynamically configured by the system based on context. The system can use different portion sizes depending on the constraints for a specific task. For example, performing ASR for real-time closed captioning of audio and / or video can benefit from low-latency transcription. Therefore, the portion size can be set to a relatively short duration, such as 0.5s to 2s. Thus, the system may be able to generate subtitles for movies or television programs, so that the words appear on the screen shortly after they are spoken or simultaneously with them (if the audio data buffer is available).

[0161] In machine translation implementations, for example, longer portion sizes can be appropriate. Due to grammatical rules and / or common usage, the word order in one language may differ from that in another. Therefore, the portion size can be set to a relatively high duration, such as 2 to 10 seconds. This provides the LM 260 with sufficient "look-ahead" information to process the input and generate the output, where several words and / or phrases appear at different positions within the output sequence relative to their positions in the input sequence.

[0162] In applications such as natural language command processing and / or chatbots, a moderate partition size can be appropriate. When the system acts as a virtual assistant or conversational agent, the partition size can be configured to balance accuracy and latency. Therefore, the partition size can be set to a relatively moderate duration, such as 1 to 3 seconds. This allows the system to efficiently extract meaning from the input while maintaining acceptable latency for generating responses and / or executing commands for the user in a conversational context, without excessive delay.

[0163] In some applicable scenarios, the system can dynamically change the portion size to handle different parts of the input and / or continuous input from multi-turn interactions. For example, the system can default to an intermediate portion length suitable for natural language processing. Users can speak to the system's device and request tasks where different portion sizes might be more suitable. For instance, a user might say, "Alexa, please provide a live transcription of the following speech." The system can process this portion of the input using an intermediate portion size and then switch to a smaller portion size to reduce the user-perceived latency associated with live transcription. In another example, a user might say, "Alexa, please translate the following speech into English." The system can again start with the default portion size and then switch to a larger portion size to provide a more accurate translation.

[0164] The LM orchestrator 230 can receive output data 565 and route it to one or more downstream processes 550, which may include further processing, communication with other components of the system, and / or outputting responses to the user. Downstream processes 550 may include operations performed, for example, by the action plan executor 350, the large action model (LAM) 564, and / or external resources such as other agents and / or tools.

[0165] The agent may have tools available to it. Similar to a cooperative agent, the agent may utilize such tools, allowing it to enhance its capabilities and perform a wide variety of tasks for the user, and to perform these tasks more accurately and / or completely than it would otherwise be able to. Tools may include API tools 552, mathematical tools 554, routine tools 556, database (DB) tools 558, and / or knowledge graph (KG) tools 562. API tools 552 may include software and / or hardware configured to facilitate communication between the agent and one or more APIs. In some implementations, API tools 552 may translate or convert messages, including natural language portions, into API calls to the appropriate API. Similarly, API tools 552 may translate or convert API responses into a format usable by the agent. API tools 552 may present an interface through which the agent can utilize various functions, such as the audio function of a media playback device, environmental settings of a smart home thermostat, etc. These functions may be abstractions of multiple operations; for example, the audio API can handle commands to play, next track, and stop; and the thermostat API can handle commands to get status, set temperature, set mode, etc.

[0166] Mathematical tool 554 may include software and / or hardware configured to implement calculator functions such as those described above in the example of powers of two. Mathematical tool 554 may receive mathematical exercises, problems, formulas, etc. from the agent and perform calculations accurately.

[0167] Routine tool 556 may include software and / or hardware configured to detect and / or periodically check for the occurrence of an event or the presence of a condition. Upon detecting or occurring a condition, routine tool 556 may perform a predefined action and / or send a notification to the agent. Routine tool 556 may facilitate the agent's ability to perform tasks involving the passage of time (e.g., between the time a user enters a command and the time when an action is to be performed). The Robot / Alexa light switch example above is an example use of such routine tool 556.

[0168] DB tool 558 may include software and / or hardware configured to provide mechanisms for agents to query and / or update databases (e.g., using Structured Query Language (SQL)). In some implementations, DB tool 558 may facilitate other database operations. Similarly, KG tool 562 may include software and / or hardware configured to facilitate the identification and retrieval of information in structured knowledge graphs. This allows agents to answer factual questions more accurately compared to using LM-based methods alone.

[0169] LAM 564 may include software and / or hardware configured to provide agents with means of interacting with certain resources in a manner similar to and / or analogous to human communication, without the use of APIs. LAM 564 may include one or more machine learning models configured to interact with computing resources (such as websites, apps, etc.) via a GUI and / or VUI. LAM 564 can "read" and process text, symbols, and / or images displayed in the GUI. LAM 564 can determine how to use the GUI to perform actions requested by the user. LAM 564 can generate data or acquire data input via the GUI. LAM 564 can select links, menu items, settings, etc., via the GUI. LAM 564 can obtain output from the GUI and communicate it to the user. In some cases, LAM 564 and / or LM 260 can convert and / or translate data; for example, by converting text to speech, translating one language into another, etc. In various use cases, the LAM 564 can represent visually impaired individuals, those operating motor vehicles, and / or others who wish to interact with resources via voice, in interacting with the GUI. In some situations, users can configure the LM 260 and / or LAM564 to perform automated tasks; for example, purchasing concert tickets immediately upon release, or entering last-minute or last-second bids in one or more online auctions.

[0170] Figures 5B to 5D Three different voice-to-voice operations performed using System 100 are illustrated. Figure 5B An example operation of prompt generation in a multimodal LM system performing speech recognition according to an embodiment of the present disclosure is illustrated. System 100 may receive command data 203 indicating a request to transcribe additional input data 205 representing target audio. The additional input data 205 may include audio data frames A1 to A2. n The LM arranger 230 can receive audio data frames and send them to the acoustic model 240a. The acoustic model 240a can convert the audio data frames into audio lexical units A'1 to A'. n The LM orchestrator 230 can use command data 203 U1 to U n and audio word units A'1 to A' n To create input sequence 502. LM orchestrator 230 can use " "Separator words (e.g., indicating the start of an acoustic word segment) are used to separate command data 203 from acoustic words."

[0171] The LM 260 can process input sequence 502 to predict output acoustic lexical units A'1 to A' n and content morphemes C1 to C n Content terms C1 to C n This can form ASR data 575, which can be used by one or more downstream processes 550. LM 260 can operate in an autoregressive manner. Therefore, LM 260 can be based on previously predicted content terms C1 to C... n-1 To predict each subsequent content term C n In some cases, the LM 260 can output the delimiter word "". "Ends with the indicative part (e.g., content terminology)." The delimiter lexicon can indicate to the acoustic model 240a and / or the LM arranger 230 where the next part of the acoustic lexicon should begin. In some cases, the LM 260 can output a speech start lexicon indicating the start of the transcription of the input speech (e.g., " <sos>LM 260 can generate output acoustic lexical units and / or output content lexical units. LM orchestrator 230 can receive the generated lexical units and add them to input sequence 502 for subsequent steps of the task. When LM 260 predicts the end of a segment, LM 260 can request the next segment of input content lexical units from LM orchestrator 230.

[0172] In some implementations, system 100 may convert ASR data 575 (which may consist of content lexical units) into text. System 100 may include a lexical-to-text component 590. The lexical-to-text component 590 may convert ASR data 575 into text data 315 for further processing and / or as human-readable text output. In some implementations, the lexical-to-text component 590 may be, for example, a combination of software and / or logic configured to convert instructional lexical units and / or lexical segments into corresponding text words (and / or individual characters, subwords, phrases, etc.). In some implementations, the lexical-to-text component 590 may include a machine learning model, such as a recurrent neural network (RNN), trained to process ASR data 575 to determine the corresponding text data 315. LM orchestrator 230 may route ASR data 575 and / or text data 315 to one or more downstream processes 550 for further processing / output.

[0173] In addition to speech-to-text (e.g., ASR),< / sos> Figure 5B The configuration shown can also be used (or modified for) related tasks such as speaker attribution ASR, where system 100 transcribes speech and identifies the speaker of each utterance, and / or speech-to-API, where input speech is converted into API calls. Figure 5B The operation shown can be extended to include speaker attribution ASR. The user can provide additional input data 205 representing speech samples (and possibly additional speech samples). The LM arranger 230 can utilize the reference encoder 240c to generate speaker embeddings, which the LM 260 can use to determine which parts of the speech in the target audio correspond to which speech sample.

[0174] In some cases, these speech-to-text functions can be performed as part of complex tasks; for example, speaker attribution ASR. System 100 can receive speech samples and audio data representing spoken conversation, generate a transcript of the speech, and attribute the various parts of the transcript to speakers whose speech characteristics match those of the speech samples. System 100 can receive input as follows: User: "Decode each speaker's voice and assign a label." User: "This is the audio of the speaker, Mike." <User-uploaded Mike's voice sample> User: "This is the audio of the speaker, Eddie." <User-uploaded Eddie's audio sample> User: "This is the target audio." <User uploaded target audio containing dialogue> The LM arranger 230 can use command data 203 ("Decode speech...") and various input data segments 205, including Mike's voice sample, Eddie's voice sample, and the target audio to be transcribed, to create a cue sequence. The LM arranger 230 can separate each data segment using delimiter words. The LM arranger 230 may include delimiter words at the end of the target audio (e.g., at the end of the cue) to instruct the LM 260 to begin generating output. The LM arranger 230 may send a cue to the LM 260. The LM 260 may generate the following responses: LM 260: <Speaker 1's spoken words> {"Speaker": "Mike"} <Speaker 2's spoken words> {"Speaker": "Eddie"} System 100 may output this response to the user in transcribed form, along with the speaker's instructions corresponding to each part.

[0175] In the example above, the multimodal LM system 100 receives various audio data inputs (e.g., command data 203, first input data 205a, second input data 205b, and / or third input data 205c, etc.); however, command data 203 may instruct different inputs to be processed in different ways. For example, while command data 203 instructs to transcribe the target audio, the first and second audio data are not transcribed. Instead, command data 203 instructs to extract speech features from the first and second data to allow LM 260 to determine which part of the target audio data corresponds to which speaker.

[0176] LM orchestrator 230 may include an input data processor that processes input data according to the data type and / or the purpose for which the data is provided. In the speaker attribution ASR example, command data 203 and target audio may be lemmatized (e.g., using an audio encoder) for processing into natural language by LM 260. Sound samples may be encoded (e.g., using a reference encoder) to generate speaker embeddings representing speech characteristics, which LM 260 may use to associate transcribed portions with corresponding speakers. Therefore, system 100 may include one or more input data encoders 240a, 240b, 240c, etc. (collectively referred to as "input data encoder 240"). The input data processor may send various input data items to the appropriate input data encoder 240 for transformation, feature extraction, and / or other processing.

[0177] Figure 5C An example operation of prompt generation in a multimodal LM system for performing speech synthesis according to an embodiment of the present disclosure is illustrated. In text-to-speech operation, system 100 may receive command data 203, which instructs processing of a request for other input data 205 representing content for synthesized speech output. LM orchestrator 230 may receive content data C1 to C2. n Other input data in the form of LM editor 230 can utilize content encoder 240b to process other input data to determine content lexical units C'1 to C' n .

[0178] Text-to-speech operations can be similar to Figure 5B The illustrated speech-to-text operation; however, instead of autoregressively predicting each output content lexical based on previously predicted output content lexicals, the LM 260 autoregressively predicts output acoustic lexicals based on previously predicted output acoustic lexicals. For example, the LM 260 can autoregressively predict output acoustic lexicals A1 to A1 based on previously predicted output acoustic lexicals A1 to A2. n-1 To output acoustic term A n At each step, LM orchestrator 230 can add the newly predicted output acoustic terminology to input sequence 503, and LM 260 can predict the next output acoustic terminology.

[0179] The LM 260 can output acoustic word data A1 to A n The data is sent to rendering model 380 for upsampling into high-fidelity audio word metadata, which vocoder 580 can convert into waveform audio data (e.g., output audio data 314). System 100 can then send the output audio data 314 to user device 110 for output to the user.

[0180] LM 260 can output acoustic lexical units representing various speech sounds. Rendering model 380 can process the acoustic lexical units to generate speech output (e.g., audio data 314 representing synthesized speech). Rendering model 380 can be a neural network configured to upsample relatively "coarse" acoustic lexical units (e.g., output by acoustic model 240a and / or LM 260) into higher-fidelity audio lexical units, which can be used for high-quality rendering. In some implementations, rendering model 380 can receive speaker profile information representing speaker-related speech characteristics and use said information to generate output speech with the desired speech. Rendering model 380 can retrieve speaker profile information from, for example, a speech data storage component 595. Speaker profile information may include embeddings generated by an audio encoder (such as a neural network encoder) trained to recognize different speakers (e.g., by using contrastive learning to extract unique speech characteristics from speech samples corresponding to different speakers). In some implementations, the rendering model 380 may additionally use content lexical units generated by the LM 260 to generate the output speech.

[0181] In some implementations, system 100 may include a vocoder 580, such as a neural vocoder, configured to process rendered audio lexics to determine audio data 314 (e.g., representing synthesized speech). Vocoder 580 may be a general-purpose neural vocoder, for example, based on Parallel WaveNet or other models. Vocoder 580 may take audio data as input, for example, in the form of a Mel spectrogram with 80 coefficients and a frequency range of 50 Hz to 12 kHz. Vocoder 580 may process the spectrogram data and convert it to a time-domain audio format (e.g., Pulse Code Modulation (PCM), Waveform Audio Format (WAV), μ-law, etc.), which can be readily converted to an analog signal for amplification and output by a speaker. The resulting audio data may consist of 8-bit, 16-bit, or 24-bit audio with sampling rates such as 16 kHz, 24 kHz, 44.1 kHz. In some implementations, other bit and / or sampling rates may be used. A digital-to-analog converter (DAC) converts audio data into an analog signal suitable for amplification and outputs it as audio through speaker 712.

[0182] Figure 5D Example operation of a prompt generation component in a multimodal LM system that performs voice-to-voice functionality according to an embodiment of this disclosure is illustrated. Figure 5D The operations shown can be used to perform tasks such as speech machine translation, speech conversion (e.g., receiving speech and generating output with different speech characteristics), anonymization, etc. In some implementations, although... Figure 5D Not shown, but the LM orchestrator 230 may use the input data encoder 240 to encode the command data 203 and / or other input data 205 (e.g., as shown in the figure). Figure 5A (As shown). The LM orchestrator 230 can encode the command data 203 (e.g., U1 to U...). n ) and other input data 205 (e.g., A1 to A n This is combined into the input sequence 504. Then, the LM arranger 230 can feed the input sequence 504 into the LM 260.

[0183] In speech-to-speech operations, the LM 260 can autoregressively predict output acoustic lexical units and output content lexical units. For example, as a speech recognition function, the LM 260 can predict output content lexical units from input acoustic lexical units. Similarly, as a speech generation function, the LM 260 can predict output acoustic lexical units from the predicted output lexical units. The output acoustic lexical units can be upsampled by the rendering model 380 to generate high-fidelity audio lexical units. In some implementations, the rendering model 380 can also receive the output content lexical units and process them together with the output acoustic lexical units (e.g., as...). Figure 5C (As shown). The vocoder 580 can convert audio word data into waveform audio data (e.g., output audio data 314).

[0184] Figure 6 This is a conceptual diagram of a multi-model LM system 100 according to an embodiment of this disclosure. (See diagram for example.) Figure 6 As shown, the multimodal LM system 100 can receive input data 401 (e.g., input data 401 and / or other input data 205). Input data 401 may include one or more types of multimodal data. For example, input data 401 may include audio data 311, text data 313, image data 321, video data 623, and / or other data 627. Input data 401 may be processed by one or more input data encoders 240, which may configure the data in a form usable by downstream components. One or more input data encoders 240 may be configured according to the type of input data 401 to be processed. For example, for input data 401 including audio data 311, the input data encoder 240 may include an acoustic model 240a, such as the one referenced above. Figure 2A The acoustic models discussed. For input data 401 including image data 321, the input data encoder 240 may include a model configured to process the image data into tokens representing image information, feature data representing image characteristics, and / or another form of model. For input data 401 including video data 623, the input data encoder 240 may include a model configured to process the video data into tokens representing video information, feature data representing video characteristics, and / or another form of model. The input data encoder 240 may also take other forms, such as processing other data 627. The input data encoder 240 may use information from the strategy data storage component 635 to coordinate processing in a manager that segments / blocks data into a form usable by the LM orchestrator 230. The LM orchestrator 230 may operate in a manner similar to that described above, except that it uses multi-model data instead of / supplementing the audio data.

[0185] LM 260 can process multimodal data to determine the output data to be passed to rendering model 670. Depending on the system configuration, the output data sent from LM 260 to rendering model 670 can be text, audio, image, video, and / or other multimodal data. For example, rendering model 670 may include audio rendering models (such as 170 discussed above), image rendering models (such as diffusion models), video rendering models, etc. Rendering model 670 may rely on data from rendering data component 675, which can provide data to rendering model 670 to assist in rendering data for final configuration and output. For example, rendering data component 675 may indicate style, solution, time, or other characteristics corresponding to the final output. Data from rendering model 670 may be sent to output configuration component 680, which can configure output data 481 for final output / presentation. For example, in the case of rendered audio data, output configuration component 680 may include vocoder 580. In the case of rendered audio data, output configuration component 680 may include an image configuration component; in the case of rendered video data, output configuration component 680 may include a video configuration component, and so on. Therefore, depending on system configuration and operation, the final output data 481 may include audio data 314, text data 315, image data 621, video data 625, and / or other data 629. Therefore, system 100 can use... Figure 6 The multimodal LM 260 is used to perform the processing discussed in this paper (e.g., refer to...). Figures 5A to 5D The processing (as discussed above) can operate within the context of system 100, for example, as... Figure 3 and Figure 4 As described.

[0186] Figure 7 This is a block diagram conceptually illustrating a device 110 that can be used with system 100 or 300. Figure 8 This is a block diagram conceptually illustrating example components of system components 120 and / or 375 of system 100 or 300. The system (120 / 375) may include one or more servers. As used herein, "server" may refer to a conventional server as understood in a server / client computing architecture, but may also refer to many different computing components that may assist the operations discussed herein. For example, a server may include one or more physical computing components (such as rack servers) that are physically and / or connected via a network to other devices / components and are capable of performing computing operations. A server may also include one or more virtual machines that simulate a computer system and run on one device or across multiple devices. A server may also include other combinations of hardware, software, firmware, etc., to perform the operations discussed herein. A server may be configured to operate using one or more of the following computing technologies: client-server model, computer office model, grid computing, fog computing, mainframe technology, utility computing, peer-to-peer model, sandboxing, or others.

[0187] While device 110 can operate locally with the user (e.g., within the same environment, so the device can receive input and play back output for the user), the server / system component can be located remotely from device 110, as its operation may not require proximity to the user. The server / system component can be located in a completely different location from device 110 (e.g., as part of a cloud computing system, etc.), or it can be located in the same environment as device 110 but physically separate from it (e.g., a home server or similar device located in the user's home or office but possibly in a closet, basement, attic, etc.). System component 120 can also be a version of user device 110 that includes (e.g., more) processing capabilities different from other devices 110 in the home / office. One benefit of having the server / system component located in the user's home / office is that the data used to process commands / return responses can be stored at the user's home, thus reducing potential privacy concerns.

[0188] The system 100 disclosed herein may include multiple system components (120 / 375), such as one or more natural language processing system components 120 for performing ASR processing, one or more natural language processing system components 120 for performing NLU processing, one or more skill system components 325, etc. In operation, each of these systems may include computer-readable instructions and computer-executable instructions residing on the corresponding device (120 / 375), as will be further discussed below.

[0189] Each of these devices (110 / 120 / 375) may include one or more controllers / processors (704 / 804), each of which may include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (706 / 806) for storing the data and instructions of the respective device. The memory (706 / 806) may individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (110 / 120 / 375) may also include a data storage component (708 / 808) for storing data and controller / processor executable instructions. Each data storage component (708 / 808) may individually include one or more non-volatile memory types, such as magnetic storage, optical storage, solid-state storage, etc. Each device (110 / 120 / 375) can also be connected to removable or external non-volatile memory and / or storage (such as removable memory cards, memory key drivers, network storage, etc.) via the corresponding input / output device interface (702 / 802).

[0190] The computer instructions for operating each device (110 / 120 / 375) and its various components can be executed by the controller / processor (704 / 804) of the corresponding device during operation, using memory (706 / 806) as temporary "working" storage. The computer instructions for the device can be stored in a non-transitory manner in non-volatile memory (706 / 806), storage (708 / 808), or an external device. Alternatively, in addition to or replacing software, some or all of the executable instructions can be embedded in the hardware or firmware of the corresponding device.

[0191] Each device (110 / 120 / 375) includes an input / output device interface (702 / 802). Various components can be connected via the input / output device interface (702 / 802), as will be discussed further below. Additionally, each device (110 / 120 / 375) may include an address / data bus (724 / 824) for communicating data between components of the respective device. In addition to (or instead of) connecting to other components across the bus (724 / 824), each component within a device (110 / 120 / 375) may also be directly connected to other components.

[0192] refer to Figure 8 Device 110 may include an input / output device interface 702 that connects to various components, such as audio output components, such as speaker 712, wired or wireless headsets (not illustrated), or other components capable of outputting audio. Device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 720 or microphone array, a wired or wireless headset (not illustrated), etc. If a microphone array is included, the approximate distance to the origin of the sound can be determined by acoustic localization based on the time and amplitude differences between sounds captured by different microphones in the array. Device 110 may additionally include a display 716 for displaying content. Device 110 may further include a camera 718.

[0193] Via antenna 722, input / output device interface 702 can connect to one or more networks 199 via wireless local area network (WLAN) radio (such as Wi-Fi), Bluetooth, and / or wireless network radio, such as radios capable of communicating with wireless communication networks such as LTE networks, WiMAX networks, 3G networks, 4G networks, and 5G networks. Wired connections, such as Ethernet, are also supported. Through network 199, the system can be distributed across network environments. The I / O device interfaces (702 / 802) may also include communication components that allow data exchange between devices (such as different physical servers or other components in a set of servers).

[0194] Components of device 110, the natural language command processing system component, or the skill system component 325 may include their own dedicated processors, memory, and / or storage. Alternatively, one or more components of device 110, the natural language command processing system component, or the skill system component 325 may utilize the I / O interface (702 / 802), processor (704 / 804), memory (706 / 806), and / or storage (708 / 808) of device 110, the natural language command processing system component, or the skill system component 325, respectively. Thus, an ASR component may have its own I / O interface, processor, memory, and / or storage; an NLU component may have its own I / O interface, processor, memory, and / or storage; and so on for the various components discussed herein.

[0195] As noted above, multiple devices can be employed in a single system. In such a multi-device system, each device may include different components for performing different aspects of system processing. Multiple devices may include overlapping components. As described herein, the components of device 110, the natural language command processing system component, and the skill system component 325 are exemplary and may be positioned as independent devices or may be wholly or partially included as components of a larger device or system. It is understood that numerous components may be present on the system component and / or device 110.

[0196] like Figure 9 As illustrated, multiple devices (110a-110n, 120, 375) may contain components of the system, and the devices may be connected to network 199. Network 199 may include a local or private network, or may include a wide area network (such as the Internet). Devices may connect to network 199 via wired or wireless connections. For example, voice detection device 110a, smartphone 110b, smartwatch 110c, tablet computer 110d, vehicle 110e, voice detection device 110f with display, monitor / smart TV 110g, washer / dryer 110h, refrigerator 110i, microwave oven 110j, autonomous mobile device 110k (e.g., robot), etc., may connect to network 199 via wireless service providers, via Wi-Fi or cellular network connections, etc. Other devices are included as supporting devices for network connectivity, such as natural language command processing system component 120, skill system component 325, and / or other components. Supporting devices may connect to network 199 via wired or wireless connections. The network device may use one or more built-in or connected microphones or other audio capture devices to capture audio, wherein processing is performed by the ASR component, NLU component or other components (such as the ASR component, NLU summary, etc. of the Natural Language Command Processing System Component 120) of the system device or another device connected via the network 199.

[0197] The concepts disclosed herein can be applied to many different devices and computer systems, including, for example, general-purpose computing systems, speech processing systems, and distributed computing environments.

[0198] The following clauses may also be used to understand this material.

[0199] 1. A computer-implemented method comprising: receiving, by a first computer system corresponding to a first language model (LM) agent, first input data representing a first task to be performed by the first LM agent; determining a first LM prompt using the first input data; generating first LM output data by processing the first LM prompt using the first LM agent corresponding to the first LM agent; determining that the first LM output data indicates that a first subtask of the first task should be delegated to a second LM agent different from the first LM agent; and generating second LM output data by processing the first LM output data using the first LM agent in response to determining that the first LM output data indicates that the first subtask should be delegated to the second LM agent, wherein the second LM output data is... The data represents a natural language request to perform the first subtask and the natural language request is an instruction from the first LM agent; the second LM output data is sent to a second computer system corresponding to the second LM agent; first data is received from the second computer system in response to the second LM output data; a second LM prompt is determined using the first LM prompt, the first LM output data, and the first data; a third LM output data is generated by processing the second LM prompt using the first LM; the third LM output data is determined to represent a response to the first input data; and in response to determining that the third LM output data represents a response to the first input data, the user device presents a first output corresponding to the third LM output data.

[0200] 2. The computer-implemented method as described in Clause 1, further comprising, before receiving the first input data: receiving first text data representing natural language instructions for how the first LM agent should handle a task, the first text data indicating: a first message format corresponding to a message from a user, the first message format including a first portion indicating that the message is from a user and a second portion representing natural language user input; a second message format corresponding to a message from another LM agent, the second message format including a third portion identifying the other LM agent and a fourth portion representing natural language generated by said other LM agent; and a third message format corresponding to a delegation request to be sent to another LM agent, the third message format including a fifth portion indicating the delegation request and identifying the recipient. The sixth part of the delegate LM agent and the seventh part representing a natural language message to the delegate LM agent; a first description of the first LM agent, the first description including a natural language description of a first capability corresponding to the first LM agent; a second description of the second LM agent, the second description including an identifier corresponding to the second LM agent and a natural language description of a second capability corresponding to the second LM agent; a first instruction for determining whether the second LM agent is more capable of handling the task in relation to the first LM agent; and a second instruction for delegating the task to the second LM agent in response to determining that the second LM agent is more capable of handling the task; and additionally determining the first LM prompt using the first text data.

[0201] 3. A computer-implemented method as described in Clause 1 or 2, further comprising, before receiving the second LM output data: sending to the second LM agent first text data representing natural language instructions for how the second LM agent should handle a task, the first text data indicating: a first message format corresponding to a message from another LM agent, the first message format including a first portion identifying the other LM agent and a second portion representing a natural language message generated by said other LM agent; a second message format corresponding to a response to a message from the other LM agent, the second message format including a third portion identifying the other LM agent and a fourth portion representing a natural language response; a natural language description of a second capability corresponding to the second LM agent; a first instruction for determining whether the second LM agent is capable of handling the task from the other LM agent; a second instruction for generating a first response to the other LM agent by processing the natural language message using a second LM corresponding to the second LM agent in response to determining that the second LM agent is capable of handling the task; and a third instruction for generating a second response to the other LM agent indicating that the second LM agent is unable to handle the task in response to determining that the second LM agent is unable to handle the task.

[0202] 4. A computer-implemented method as described in Clauses 1, 2, or 3, further comprising: receiving second input data representing a second task; using the first LM to generate fourth LM output data representing a second subtask of the second task to be delegated to a second LM agent upon detection of a triggering event; using the fourth LM output data to determine second data corresponding to the second subtask; detecting the occurrence of the triggering event; and in response to detecting the occurrence, sending the second data to the second LM agent, the second LM agent performing the second subtask in response to receiving the second data.

[0203] 5. A computer-implemented method comprising: receiving first input data by a first computer system corresponding to a first language model (LM) agent; generating first LM output data using the first input data and a first LM corresponding to the first LM agent, the first LM output data representing a natural language request to delegate a first task to a second LM agent different from the first LM agent and an indication that the natural language request is from the first LM agent; transmitting the first LM output data to a second computer system corresponding to the second LM agent; receiving first data from the second computer system in response to the first LM output data; generating second LM output data using the first data and the first LM, the second LM output data representing a response to the first input data; and transmitting the second LM output data to a first system component.

[0204] 6. The computer-implemented method as described in Clause 5, further comprising: receiving second data representing natural language instructions for how the first LM agent should handle a task, the second data indicating: a first instruction for determining whether the second LM agent is more capable of handling the task; and a second instruction for delegating the task to the second LM agent in response to determining that the second LM agent is more capable of handling the task; and using the first input data and the second data to determine a first LM prompt, wherein generating the first LM output data includes using the first LM to process the first LM prompt.

[0205] 7. The computer-implemented method as described in Clause 5 or 6, further comprising: receiving second data representing an identifier corresponding to the second LM agent and a natural language description of capabilities corresponding to the second LM agent; and using the first input data and the second data to determine a first LM prompt, wherein generating the first LM output data includes processing the first LM prompt using the first LM.

[0206] 8. A computer-implemented method as described in Clauses 5, 6, or 7, further comprising: receiving second data representing an identifier corresponding to a software component and a natural language description of a capability corresponding to the software component; and using the first input data and the second data to determine a first LM prompt, wherein generating the first LM output data includes processing the first LM prompt using the first LM.

[0207] 9. A computer-implemented method as described in clauses 5, 6, 7, or 8, further comprising: sending to the second LM agent second data representing natural language instructions for how the second LM agent should handle a task, the second data indicating: a first instruction for the second LM agent to determine whether it is capable of handling a task indicated in a message from another LM agent; a second instruction to generate a first response to the other LM agent in response to determining that the second LM agent is capable of handling the task by processing the message using a second LM corresponding to the second LM agent; and a third instruction to generate a second response to the other LM agent in response to determining that the second LM agent is unable to handle the task, indicating that the second LM agent is unable to handle the task.

[0208] 10. A computer-implemented method as described in clauses 5, 6, 7, 8, or 9, further comprising: sending second data to a second LM agent representing a natural language request for a description of a capability corresponding to the second LM agent before generating the first LM output data; receiving third data representing a natural language description of a capability corresponding to the second LM agent; and using the first input data and the third data to determine a first LM prompt, wherein generating the first LM output data includes processing the first LM prompt using the first LM.

[0209] 11. A computer-implemented method as described in clauses 5, 6, 7, 8, 9, or 10, further comprising: receiving second input data; using the first LM to generate third LM output data representing a second task to be delegated to a second LM agent upon detecting an event; using the third LM output data to determine second data corresponding to the second task; detecting the occurrence of the event; and in response to detecting the occurrence, sending the second data to the second LM agent, the second LM agent performing the second task in response to receiving the second data.

[0210] 12. A computer-implemented method as described in clauses 5, 6, 7, 8, 9, 10, or 11, further comprising: receiving second data representing: a first message format corresponding to a message from a user, the first message format including a first portion indicating that the message is from the user and a second portion representing natural language user input; a second message format corresponding to a message from another LM agent, the second message format including a third portion identifying the other LM agent and a fourth portion representing natural language generated by said other LM agent; and a third message format corresponding to a delegation request to be sent to another LM agent, the third message format including a fifth portion identifying the delegated LM agent and a sixth portion representing a natural language message to said delegated LM agent.

[0211] 13. A first computer system comprising: at least one processor; and at least one memory, the at least one memory including instructions that, when executed by the at least one processor, cause the first computer system to: receive first input data from a first computer system corresponding to a first language model (LM) agent; generate first LM output data using the first input data and a first LM corresponding to the first LM agent, the first LM output data representing a natural language request to delegate a first task to a second LM agent different from the first LM agent and an indication that the natural language request is from the first LM agent; transmit the first LM output data to a second computer system corresponding to the second LM agent; receive first data from the second computer system in response to the first LM output data; generate second LM output data using the first data and the first LM, the second LM output data representing a response to the first input data; and transmit the second LM output data to a first system component.

[0212] 14. The first computer system as described in Clause 13, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second data representing natural language instructions on how the first LM agent should handle a task, the second data indicating: a first instruction to determine whether the second LM agent is more capable of handling the task; and a second instruction to delegate the task to the second LM agent in response to determining that the second LM agent is more capable of handling the task; and to determine a first LM prompt using the first input data and the second data, wherein generating the first LM output data includes processing the first LM prompt using the first LM.

[0213] 15. The first computer system as described in Clause 13 or 14, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second data representing an identifier corresponding to the second LM agent and a natural language description of the capabilities corresponding to the second LM agent; and use the first input data and the second data to determine a first LM prompt, wherein generating the first LM output data includes using the first LM to process the first LM prompt.

[0214] 16. The first computer system as described in Clauses 13, 14, or 15, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second data representing an identifier corresponding to a software component and a natural language description of a capability corresponding to the software component; and use the first input data and the second data to determine a first LM prompt, wherein generating the first LM output data includes processing the first LM prompt using the first LM.

[0215] 17. A first computer system as described in clauses 13, 14, 15, or 16, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: send second data to the second LM agent representing natural language instructions on how the second LM agent should handle a task, the second data indicating: a first instruction for the second LM agent to determine whether it is able to handle a task indicated in a message from another LM agent; a second instruction to generate a first response to the other LM agent in response to determining that the second LM agent is able to handle the task by processing the message using a second LM corresponding to the second LM agent; and a third instruction to generate a second response to the other LM agent in response to determining that the second LM agent is unable to handle the task, indicating that the second LM agent is unable to handle the task.

[0216] 18. A first computer system as described in clauses 13, 14, 15, 16, or 17, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: send second data to the second LM agent representing a natural language request for a description of a capability corresponding to the second LM agent before generating the first LM output data; receive third data representing a natural language description of a capability corresponding to the second LM agent; and determine a first LM prompt using the first input data and the third data, wherein generating the first LM output data includes processing the first LM prompt using the first LM.

[0217] 19. A first computer system as described in clauses 13, 14, 15, 16, 17, or 18, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second input data; use the first LM to generate third LM output data representing a second task to be delegated to a second LM agent upon detecting an event; use the third LM output data to determine second data corresponding to the second task; detect the occurrence of the event; and, in response to detecting the occurrence, send the second data to the second LM agent, the second LM agent performing the second task in response to receiving the second data.

[0218] 20. A first computer system as described in clauses 13, 14, 15, 16, 17, 18, or 19, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second data representing: a first message format corresponding to a message from a user, the first message format including a first portion indicating that the message is from the user and a second portion representing natural language user input; a second message format corresponding to a message from another LM agent, the second message format including a third portion identifying the other LM agent and a fourth portion representing natural language generated by the other LM agent; and a third message format corresponding to a delegation request to be sent to another LM agent, the third message format including a fifth portion identifying the delegated LM agent and a sixth portion representing a natural language message to the delegated LM agent.

[0219] 21. A computer-implemented method comprising: receiving first input data by a first computer system corresponding to a first language model (LM) agent, the first input data representing a first natural language request to perform a first task and a first indication that the first natural language request is from a second LM agent different from the first LM agent; determining context data for processing the request from the second LM agent; determining a first LM prompt using the first input data and the context data; generating first LM output data by processing the first LM prompt using the first LM agent corresponding to the first LM agent, the first LM output data representing a second natural language request to delegate a subtask of the first task; sending the first LM output data to a second computer system corresponding to the second LM agent; receiving second input data from the second computer system representing a first natural language response to the second natural language request; determining a second LM prompt using the first input data and the second input data; generating second LM output data by processing the second LM prompt using the first LM agent, the second LM output data representing a second natural language response to the first natural language request and a second indication that the second LM output data is from the first LM agent; and sending the second LM output data to the second computer system.

[0220] 22. The computer-implemented method of claim 21, further comprising, before receiving the first input data: receiving first text data representing natural language instructions for how the first LM agent should handle a task, the first text data indicating: a first message format corresponding to a message from another LM agent, the first message format including a first portion identifying the other LM agent and a second portion representing a natural language message generated by said other LM agent; a second message format corresponding to a response to a message from the other LM agent, the second message format including a third portion identifying the other LM agent and a fourth portion representing a natural language response to the message; a natural language description of the capabilities corresponding to the first LM agent; a first instruction for determining whether it is capable of handling the task indicated by the natural language message from the other LM agent; a second instruction for generating a first response to the other LM agent by processing the natural language message using the first LM agent in response to determining that the first LM agent is capable of handling the task; and a third instruction for generating a second response to the other LM agent indicating that the first LM agent is unable to handle the task in response to determining that the first LM agent is unable to handle the task; and using the first text data to determine the context data.

[0221] 23. A computer-implemented method as described in clause 21 or 22, further comprising, before receiving the first input data: sending to the second LM agent first text data representing natural language instructions for how the second LM agent should handle a task, the first text data indicating: a first message format corresponding to a message from a user, the first message format including a first portion indicating that the message is from a user and a second portion representing natural language user input; a second message format corresponding to a message from another LM agent, the second message format including a third portion identifying the other LM agent and a fourth portion representing natural language generated by said other LM agent; and a third message format corresponding to a delegation request to be sent to another LM agent, the third message... The message format includes a fifth part indicating a delegation request, a sixth part identifying the delegated LM agent, and a seventh part representing a natural language message to the delegated LM agent; a first description of the first LM agent, the first description including a natural language description of a first capability corresponding to the first LM agent; a second description of the second LM agent, the second description including an identifier corresponding to the second LM agent and a natural language description of a second capability corresponding to the second LM agent; a first instruction for determining whether the first LM agent is more capable of handling the task for the second LM agent; and a second instruction for delegating the task to the first LM agent in response to determining that the first LM agent is more capable of handling the task.

[0222] 24. A computer-implemented method as described in clauses 21, 22, or 23, further comprising: receiving from a second computer system third input data representing a second task to be performed by a first LM agent; determining a third LM prompt using the third input data and the context data; generating third LM output data by processing the third LM prompt using the first LM, the third LM input data representing the second task and an instruction to perform the second task in response to a command from the second LM agent; determining a third LM prompt corresponding to the second task using the third LM output data; determining a task identifier corresponding to the second task; sending the task identifier to the second computer system in response to the third input data; receiving fourth input data representing the command and the task identifier; processing the third LM prompt using the first LM in response to receiving the fourth input data to generate fourth LM output data; and sending the fourth LM output data to the second computer system.

[0223] 25. A computer-implemented method comprising: receiving first input data by a first computer system corresponding to a first language model (LM) agent; determining that the first input data represents a natural language request to perform a first task; determining that the first input data includes a first indication that the natural language request is from a second LM agent different from the first LM agent; using the first input data and context data to determine a first LM prompt for processing the request from the second LM agent; generating first LM output data by processing the first LM prompt using the first LM agent corresponding to the first LM agent, the first LM output data representing a natural language response to the natural language request and a second indication that the first LM output data is from the first LM agent; and sending the first LM output data to a second computer system corresponding to the second LM agent.

[0224] 26. The computer-implemented method as described in Clause 25, further comprising: receiving second data representing natural language instructions for how the first LM agent should handle a task, the second data indicating: a first instruction for determining whether the first LM agent is capable of handling a task indicated by a natural language message from another LM agent; a second instruction for generating a first response to the other LM agent by processing the natural language message using the first LM agent in response to determining that the first LM agent is capable of handling the task; and a third instruction for generating a second response to the other LM agent indicating that the first LM agent is unable to handle the task in response to determining that the first LM agent is unable to handle the task; and using the second data to determine the context data.

[0225] 27. A computer-implemented method as described in Clause 25 or 26, further comprising: receiving second input data representing a second task to be performed by the first LM agent; using the second input data and the first LM to generate second LM output data representing the second task and an instruction to perform the second task in response to a command; sending a task identifier corresponding to the second task; receiving third input data representing the command and the task identifier; generating third LM output data using the second LM output data and the first LM in response to receiving the third input data; and performing an action relative to the third LM output data.

[0226] 28. A computer-implemented method as described in clauses 25, 26, or 27, further comprising: receiving, before receiving the first input data, second data representing a natural language request for a description of a capability corresponding to the first LM agent; using the second data and the first LM to generate second LM output data representing a natural language description of the capability corresponding to the first LM agent; and sending the second LM output data to the second LM agent.

[0227] 29. A computer-implemented method as described in clauses 25, 26, 27 or 28, further comprising: receiving second data representing an identifier corresponding to a third LM agent and a natural language description of capabilities corresponding to the third LM agent; and using the second data to determine the context data.

[0228] 30. A computer-implemented method as described in clauses 25, 26, 27, 28, or 29, further comprising: receiving second data representing an identifier corresponding to a software component and a natural language description of a capability corresponding to the software component; and using the second data to determine the context data.

[0229] 31. A computer-implemented method as described in clauses 25, 26, 27, 28, 29, or 30, further comprising: receiving second input data; determining a second LM prompt using the second input data and the context data; generating second LM output data by processing the second LM prompt using a first LM, the second LM output data representing a natural language request to delegate a second task; sending the second LM output data to a second computer system; receiving from the second computer system third input data representing a natural language response to the second LM output data; determining a third LM prompt using the second input data and the third input data; generating third LM output data by processing the third LM prompt using the first LM, the third LM output data representing a natural language response to the third input data; and sending the third LM output data to the second computer system.

[0230] 32. A computer-implemented method as described in clauses 25, 26, 27, 28, 29, 30, or 31, further comprising: sending to the first LM agent a first text data representing natural language instructions for how the first LM agent should handle a task, the first text data indicating: a first message format corresponding to a message from another LM agent, the first message format including a first portion identifying the other LM agent and a second portion representing a natural language message generated by the other LM agent; and a second message format corresponding to a response to a message from the other LM agent, the second message format including a third portion identifying the other LM agent and a fourth portion representing natural language generated by the other LM agent.

[0231] 33. A first computer system comprising: at least one processor; and at least one memory, the at least one memory including instructions that, when executed by the at least one processor, cause the first computer system to: receive first input data, the first computer system corresponding to a first language model (LM) agent; determine that the first input data represents a natural language request to perform a first task; determine that the first input data includes a first indication that the natural language request is from a second LM agent different from the first LM agent; use the first input data and context data to determine a first LM prompt for processing the request from the second LM agent; generate first LM output data by processing the first LM prompt using the first LM agent corresponding to the first LM agent, the first LM output data representing a natural language response to the natural language request and a second indication that the first LM output data is from the first LM agent; and send the first LM output data to a second computer system corresponding to the second LM agent.

[0232] 34. A first computer system implemented by a computer as described in Clause 33, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second data representing natural language instructions for how the first LM agent should handle a task, the second data indicating: a first instruction for determining whether the first LM agent is capable of handling a task indicated by a natural language message from another LM agent; a second instruction for generating a first response to the other LM agent by processing the natural language message using the first LM agent in response to determining that the first LM agent is capable of handling the task; and a third instruction for generating a second response to the other LM agent indicating that the first LM agent is unable to handle the task in response to determining that the first LM agent is unable to handle the task; and using the second data to determine the context data.

[0233] 35. A first computer system as described in clause 33 or 34, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second input data representing a second task to be performed by the first LM agent; use the second input data and the first LM to generate second LM output data representing the second task and an instruction to perform the second task in response to a command; send a task identifier corresponding to the second task; receive third input data representing the command and the task identifier; generate third LM output data using the second LM output data and the first LM in response to receiving the third input data; and perform an action relative to the third LM output data.

[0234] 36. A first computer system as described in clauses 33, 34, or 25, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second data representing a natural language request for a description of a capability corresponding to the first LM agent before receiving the first input data; use the second data and the first LM to generate second LM output data representing a natural language description of the capability corresponding to the first LM agent; and send the second LM output data to the second LM agent.

[0235] 37. A first computer system as described in clauses 33, 34, 35, or 36, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second data representing an identifier corresponding to a third LM agent and a natural language description of capabilities corresponding to the third LM agent; and use the second data to determine the context data.

[0236] 38. A first computer system as described in clauses 33, 34, 35, 36, or 37, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second data representing an identifier corresponding to a software component and a natural language description of a capability corresponding to the software component; and use the second data to determine the context data.

[0237] 39. A first computer system as described in clauses 33, 34, 35, 36, 37, or 38, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: receive second input data; determine a second LM prompt using the second input data and the context data; generate second LM output data by processing the second LM prompt using the first LM, the second LM output data representing a natural language request to delegate a second task; send the second LM output data to the second computer system; receive from the second computer system third input data representing a natural language response to the second LM output data; determine a third LM prompt using the second input data and the third input data; generate third LM output data by processing the third LM prompt using the first LM, the third LM output data representing a natural language response to the third input data; and send the third LM output data to the second computer system.

[0238] 40. A first computer system as described in clauses 33, 34, 35, 36, 37, 38, or 39, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: send to the first LM agent a first text data representing natural language instructions for how the first LM agent should handle a task, the first text data indicating: a first message format corresponding to a message from another LM agent, the first message format including a first portion identifying the other LM agent and a second portion representing a natural language message generated by the other LM agent; and a second message format corresponding to a response to a message from the other LM agent, the second message format including a third portion identifying the other LM agent and a fourth portion representing natural language generated by the other LM agent.

[0239] The foregoing aspects of this disclosure are intended to be illustrative. They were chosen to explain the principles and applications of this disclosure and are not intended to be exhaustive or limiting. Many modifications and variations can be made to the disclosed aspects by those skilled in the art. Those skilled in the art of computer and speech processing will recognize that the components and process steps described herein can be interchanged with other components or steps, or combinations of components or steps, and still achieve the benefits and advantages of this disclosure. However, those skilled in the art will understand that this disclosure can be practiced without some or all of the specific details disclosed herein. Furthermore, unless expressly stated to the contrary, features / operations / components, etc., from one embodiment discussed herein may be combined with features / operations / components, etc., from another embodiment discussed herein.

[0240] Aspects of the disclosed system can be implemented as computer-implemented methods or articles of art, such as memory devices or non-transitory computer-readable storage media. The computer-readable storage medium may be computer-readable and may include instructions for causing a computer or other device to perform the processes described in this disclosure. The computer-readable storage medium may be implemented as volatile computer memory, non-volatile computer memory, hard disk drive, solid-state memory, flash memory drive, removable disk drive, and / or other media. Additionally, components of the system may be implemented as firmware or hardware.

[0241] Unless otherwise specifically stated, or otherwise understood as used in the context herein, the conditional language used herein (such as, among others, "can," "may," "may," "can," "for example," etc.) is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not include certain features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining, with or without further input or prompting, whether such features, elements, and / or steps are included or to be performed in any particular embodiment. The terms "comprise," "include," "have," etc., are synonymous and used inclusively in an open form, and do not exclude additional elements, features, actions, operations, etc. Furthermore, the term "or" is used in an inclusive sense (rather than in an exclusive sense), such that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list.

[0242] Unless otherwise specifically stated, disjunctive language such as the phrase "at least one of X, Y, or Z" is understood in context to generally represent that an item, term, etc., can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such antisemantic connective language is generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, respectively.

[0243] As used in this disclosure, unless otherwise specifically stated, the term "a" or "an" may include one or more items. Furthermore, unless otherwise specifically stated, the phrase "based on" is intended to mean "at least in part based on".< / o> < / m> < / o> < / m> < / m>

Claims

1. A computer-implemented method, comprising: The first input data is received by a first computer system corresponding to the first language model (LM) agent; The first input data and the first LM corresponding to the first LM agent are used to generate first LM output data, which represents a natural language request to delegate the first task to a second LM agent that is different from the first LM agent and that the natural language request is an instruction from the first LM agent. The first LM output data is sent to the second computer system corresponding to the second LM agent; Receive first data from the second computer system in response to the first LM output data; The first data and the first LM are used to generate second LM output data, which represents the response to the first input data; as well as The second LM output data is sent to the first system component.

2. The computer-implemented method as described in claim 1, further comprising: Receive second data representing natural language instructions on how the first LM agent should handle the task, the second data indicating: The first instruction given by the first LM agent to determine whether the second LM agent is more capable of handling the task, and A second instruction is used in response to determining that the second LM agent is more capable of handling the task and delegating the task to the second LM agent; as well as The first input data and the second data are used to determine the first LM prompt, wherein generating the first LM output data includes using the first LM to process the first LM prompt.

3. The computer-implemented method as described in claim 1 or 2, further comprising: Receive second data representing an identifier corresponding to the second LM agent and a natural language description of the capabilities corresponding to the second LM agent; as well as The first input data and the second data are used to determine the first LM prompt, wherein generating the first LM output data includes using the first LM to process the first LM prompt.

4. The computer-implemented method as described in claim 1, 2, or 3, further comprising: Receive second data representing an identifier corresponding to a software component and a natural language description of the capabilities corresponding to the software component; as well as The first input data and the second data are used to determine the first LM prompt, wherein generating the first LM output data includes using the first LM to process the first LM prompt.

5. The computer-implemented method as described in claim 1, 2, 3, or 4, further comprising: Send second data to the second LM agent, representing natural language instructions on how the second LM agent should handle the task, the second data indicating: The second LM agent determines whether it is capable of handling the task indicated in a message from another LM agent, based on the first instruction. A second instruction is used to generate a first response to the other LM agent in response to determining that the second LM agent is capable of handling the task, by using the second LM corresponding to the second LM agent to process the message. A third instruction is used to generate a second response to the other LM agent in response to determining that the second LM agent is unable to handle the task, indicating that the second LM agent is unable to handle the task.

6. The computer-implemented method as described in claim 1, 2, 3, 4, or 5, further comprising: Before generating the first LM output data, a second data request in natural language representing a description of the capabilities corresponding to the second LM agent is sent to the second LM agent; Receive third data representing a natural language description of the capabilities corresponding to the second LM agent; as well as The first LM prompt is determined using the first input data and the third data, wherein generating the first LM output data includes processing the first LM prompt using the first LM.

7. The computer-implemented method as described in claim 1, 2, 3, 4, 5, or 6, further comprising: Receive the second input data; The first LM is used to generate third LM output data representing a second task to be delegated to the second LM agent when an event is detected; The third LM output data is used to determine the second data corresponding to the second task; Detect the occurrence of the event; as well as In response to detecting the occurrence, the second data is sent to the second LM agent, and the second LM agent performs the second task in response to receiving the second data.

8. The computer-implemented method as described in claim 1, 2, 3, 4, 5, 6, or 7, further comprising: Receive the following second data: A first message format corresponding to a message from a user, the first message format including a first part indicating that the message is from a user and a second part representing natural language user input. A second message format corresponding to messages from other LM agents, the second message format including a third part identifying the other LM agent and a fourth part representing the natural language generated by said other LM agent, and A third message format corresponding to a delegation request to be sent to another LM agent, the third message format including a fifth part identifying the delegated LM agent and a sixth part representing a natural language message to the delegated LM agent.

9. A first computer system comprising: At least one processor; as well as At least one memory, the at least one memory including instructions that, when executed by the at least one processor, cause the first computer system to: The first input data is received by a first computer system corresponding to the first language model (LM) agent; The first input data and the first LM corresponding to the first LM agent are used to generate first LM output data, which represents a natural language request to delegate the first task to a second LM agent that is different from the first LM agent and that the natural language request is an instruction from the first LM agent. The first LM output data is sent to the second computer system corresponding to the second LM agent; Receive first data from the second computer system in response to the first LM output data; The first data and the first LM are used to generate second LM output data, which represents the response to the first input data; as well as The second LM output data is sent to the first system component.

10. The first computer system of claim 9, wherein the at least one memory further includes instructions, which, when executed by the at least one processor, further cause the first computer system to: Receive second data representing natural language instructions on how the first LM agent should handle the task, the second data indicating: The first instruction given by the first LM agent to determine whether the second LM agent is more capable of handling the task, and A second instruction, in response to determining that the second LM agent is more capable of handling the task, is to delegate the task to the second LM agent; and The first input data and the second data are used to determine the first LM prompt, wherein generating the first LM output data includes using the first LM to process the first LM prompt.

11. The first computer system of claim 9 or 10, wherein the at least one memory further includes instructions, which, when executed by the at least one processor, further cause the first computer system to: Receive second data representing an identifier corresponding to the second LM agent and a natural language description of the capabilities corresponding to the second LM agent; and The first input data and the second data are used to determine the first LM prompt, wherein generating the first LM output data includes using the first LM to process the first LM prompt.

12. The first computer system of claim 9, 10, or 11, wherein the at least one memory further includes instructions, which, when executed by the at least one processor, further cause the first computer system to: Receive second data representing an identifier corresponding to a software component and a natural language description of the capabilities corresponding to the software component; and The first input data and the second data are used to determine the first LM prompt, wherein generating the first LM output data includes using the first LM to process the first LM prompt.

13. The first computer system of claim 9, 10, 11, or 12, wherein the at least one memory further includes instructions, which, when executed by the at least one processor, further cause the first computer system to: Send second data to the second LM agent, representing natural language instructions on how the second LM agent should handle the task, the second data indicating: The second LM agent determines whether it is capable of handling the task indicated in a message from another LM agent, based on the first instruction. A second instruction is used to generate a first response to the other LM agent in response to determining that the second LM agent is capable of handling the task, by using the second LM corresponding to the second LM agent to process the message. A third instruction is used to generate a second response to the other LM agent in response to determining that the second LM agent is unable to handle the task, indicating that the second LM agent is unable to handle the task.

14. The first computer system as claimed in claim 9, 10, 11, 12, or 13, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the first computer system to: Before generating the first LM output data, a second data request in natural language representing a description of the capabilities corresponding to the second LM agent is sent to the second LM agent; Receive third data representing a natural language description of the capabilities corresponding to the second LM agent; as well as The first LM prompt is determined using the first input data and the third data, wherein generating the first LM output data includes processing the first LM prompt using the first LM.

15. The first computer system as claimed in claim 9, 10, 11, 12, 13, or 14, wherein the at least one memory further includes instructions, which, when executed by the at least one processor, further cause the first computer system to: Receive the second input data; The first LM is used to generate third LM output data representing a second task to be delegated to the second LM agent when an event is detected; The third LM output data is used to determine the second data corresponding to the second task; Detect the occurrence of the event; as well as In response to detecting the occurrence, the second data is sent to the second LM agent, and the second LM agent performs the second task in response to receiving the second data.

Images