Natural language processing

Abstract

Techniques are described for determining one or more responses associated with one or more components in response to a user input. A system receives user input and causes one or more components to generate one or more responses associated with the user input. The system determines that one or more of the responses are responsive to the user input, causes one or more actions associated with the responses to be performed, and outputs a natural language summary of the one or more responses. If the system determines that no one of the responses is responsive to the user input and / or ambiguous with respect to the user input, the system can generate a request for additional information that can be used to address the ambiguity, the request for additional information can be sent to another component of the system and / or output to a user providing the user input.

Description

[0001] Cross-referencing of relevant application data This application claims the benefit and priority of U.S. Patent Application No. 18 / 456,949, filed August 28, 2023, entitled “NATURALLANGUAGE PROCESSING”, by Xing Fan et al. The entire patent application is incorporated herein by reference. Background Technology

[0002] Natural language processing systems have evolved to the point where humans can interact with computing devices using their voice and natural language text input. These systems employ techniques to recognize words spoken and written by human users based on received input data of varying quality. Speech recognition combined with natural language understanding processing enables voice-based user control of computing devices to perform tasks based on the user's verbal input. This processing can be used by computers, handheld devices, telephone computer systems, self-service terminals, and a wide variety of other devices to improve human-computer interaction. Attached Figure Description

[0003] To gain a more complete understanding of this disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0004] Figure 1 This is a conceptual diagram illustrating example components according to an embodiment of the present disclosure and a process for determining one or more components configured to perform actions associated with a task.

[0005] Figure 2 This is an example component and a conceptual diagram illustrating a response arbitration component and a process according to an embodiment of this disclosure.

[0006] Figure 3 This is a conceptual diagram illustrating an example route for output data by a response arbitration component according to an embodiment of this disclosure.

[0007] Figure 4 This is an example component and a conceptual diagram illustrating the planned generation of components and processes according to an embodiment of this disclosure.

[0008] Figure 5 This is a conceptual diagram illustrating an example component and process of an LLM candidate list component according to an embodiment of this disclosure.

[0009] Figure 6 This is a conceptual diagram of the components of a system according to an embodiment of this disclosure.

[0010] Figure 7 This is a conceptual diagram illustrating an example processing of an arbitrator component according to an embodiment of this disclosure.

[0011] Figure 8This is a conceptual diagram illustrating components that may be included in an apparatus according to an embodiment of the present disclosure.

[0012] Figure 9 It is a block diagram that conceptually illustrates example components of an apparatus according to an embodiment of the present disclosure.

[0013] Figure 10 It is a block diagram that conceptually exemplifies example components of a system according to an embodiment of the present disclosure.

[0014] Figure 11 Examples of computer networks used in conjunction with the overall system according to embodiments of this disclosure are illustrated. Detailed Implementation

[0015] Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves transforming audio data associated with speech into word units or other text representations of that speech. Similarly, natural language understanding (NLU) is a field of computer science, artificial intelligence, and linguistics that involves enabling computers to derive meaning from natural language input, such as spoken input. ASR and NLU are often used together as part of a system's language processing component. Text-to-speech (TTS) is a field of computer science that involves transforming text and / or other data into audio data, which is then synthesized to resemble human speech. Natural language generation (NLG) is a field of artificial intelligence that involves automatically transforming data into natural language (e.g., English) content. Language modeling (LM) uses various statistical and probabilistic techniques to determine the probability of a given sequence of words appearing in a sentence.

[0016] Some systems can be configured to respond to user input in natural language (e.g., spoken or typed). For example, in response to a user inputting "what is today's weather?", the system can output weather information for the user's location. As another example, in response to a user inputting "what are today's top stories?", the system can output one or more news reports. For a further example, in response to a user inputting "tell me a joke?", the system can output a joke to the user. As another example, in response to a user inputting "book me a flight to Seattle?", the system can book a flight to Seattle and output information about the booked flight. For a further example, in response to a user inputting "lock the front door?", the system can actuate the "front door" smart lock to the locked position.

[0017] The system can receive user input in the form of speech. For example, the user can speak the input into the device. The device can then send audio data representing the spoken input to the system. The system can perform ASR processing on the audio data to generate ASR data representing the user input (e.g., text data, word data, etc.). The system can process the ASR data to determine the action in response to the user input.

[0018] In some cases, a system may be configured to process ASR data using one or more language models (e.g., one or more large language models (LLMs)) to determine one or more components (e.g., generating potential responses / actions in response to user input) that are configured to perform one or more functions potentially in response to user input. For example, in response to the user input “Please plan a 4-person trip to [Location] from [Date 1] to [Date 2]”, the language model may determine one or more components (e.g., API, skill component, LLM agent component, etc.) that are configured to book flights and hotels. To select one or more components to respond to user input, the system may request data from one or more components that includes potential responses to the user input. Potential responses may include natural language data that can be used to respond to user input. Potential responses may also, or alternatively, include a description of the potential actions that the component is configured to / will perform relative to the user input.

[0019] The language model processes data returned from components to determine whether a potential response is in response to user input. If one or more potential responses are in response to user input, the system may select more than one potential response to output, or may generate a summary of the selected potential responses (e.g., given the user input "what is the capital of France", if the selected potential responses are "the capital of France is Paris" and "Paris became the capital of France in 987", the system may output "the capital of France is Paris", "Paris became the capital of France in 987", or a summary of the selected responses: "Paris has been the capital of France since 987").

[0020] If one or more of the selected potential responses include a description of a potential action, the system may further cause the corresponding component to perform the corresponding action (e.g., given a user input "please turn on the light," if the selected potential response includes the potential action of the first component turning on the first light, the system will further cause the first component to turn on the first light). In some embodiments, the system may cause the component corresponding to the selected response to perform the corresponding action before outputting natural language data to the user (e.g., given a user input "please turn on the light" and the above-selected response, the system may first cause the first component to turn on the first light, and then may present the natural language output "The light is on" via a device display and / or synthesized speech).

[0021] If a potential response does not address the user input and / or the system determines that there is ambiguity relative to the user input, the system may send a request for additional information that can be used to resolve the ambiguity. The request for additional information may be presented to the user and / or to components configured to determine the additional information. For example, given the user input “book aflight to [destination]”, the system may determine that the departure point is not included in the user input and may therefore request such information from components (e.g., contextual data indicating the user’s current location or home location) or request such information from the user.

[0022] This disclosure provides techniques for selecting from one or more potential responses provided by one or more different types of components using one or more language models. The system is configured to receive and process potential responses from different types of components (such as APIs, skill components, and LLM-based proxy components) to perform an action in response to user input. The system may process to determine one or more components configured to generate a response associated with a user request, and receive potential responses from the components. The system may process the potential responses and contextual information associated with the user input to select one or more potential responses that respond to the user request. In some cases, the system may generate a summary of one or more selected responses, and / or, if one or more selected responses include a potential action, cause the corresponding component to perform the potential action. For example, in response to receiving the user input "What's the weather like today?", the system can process to determine one or more components (e.g., a weather skills component, an LLM agent fine-tuned for weather inquiries) configured to generate potential responses associated with the user input, and receive from one or more components the potential response "It is currently 70 degrees, with a high of 75 and a low of 68" from a first component, and the potential response "The weather for today is expected to be mostly sunny, but with a chance of rain in the late afternoon" from a second component. The system can determine that potential responses from both components are in response to user input and generate a summary of the response, such as "It is expected to be mostly sunny today, with a high of 75 and a low of 68, but with a chance of rain in the late afternoon," which can be output to the user (e.g., as audio or video information).For a further example, in response to receiving the user input "Please turn on the outside lights," the system may process to determine one or more components (e.g., device control components, LLM agents, etc.) configured to generate potential responses associated with the user input, and receive potential actions from one or more components: "Turn on porch light A" from a first component, "Turn on garage light B" from a second component, and "Turn on hallway light C" from a third component. The system may determine that the responses from the first and second components are in response to the user input and cause the first and second components to perform the corresponding actions (e.g., turn on porch light A and garage light B). The system may also output a summary of the response to the user input selection: "The porch light and the garage light are now on." As another example, in response to a user input “Please plan a 4-person trip to [Date 1] to [Date 2]”, the system may determine one or more components (e.g., a travel skills component, an LLM agent fine-tuned for travel inquiries, etc.) configured to generate potential responses associated with the user input (e.g., booking flights and booking hotels), and receive potential responses from one or more components, including potential actions from a first component such as “book flight ticket for flight [flight number] through [airline companyname] on [Date 1]” and potential actions from a second component such as “book a hotel room at [Hotel company name] in [Location] from [Date 1] to [Date 1]”.The system can identify potential responses from the first and second components in response to user input and can generate a summary of the selected response, such as "I found flight [flight number] to [Location] through [airline company] on [Date 1] and a hotel room at [Hotel company name] from [Date 1] to [Date 2], want me to book them?", to output to the user. The system can also cause the first and second components to perform actions (e.g., book the identified flight and hotel), such as in response to user authorization.

[0023] If the system determines that none of the potential responses are in response to user input and / or ambiguous with respect to the user request, the system may generate a request for additional information that can be used to resolve the ambiguity and / or perform an action in response to the user request. The request for additional information may be output to the user and / or sent to another component of the system that may return various user-specific information associated with the user, including dialogue information (e.g., one or more previous user inputs and / or system-generated responses to the current interaction between the user and the system), user preferences, and user behavior information (e.g., information on one or more typical behaviors associated with the user (e.g., the user turns on outdoor lights after 7 p.m., user preference [music streaming service 1], etc.)). The system may use the additional information to resolve any ambiguity in the user request, arbitrate between potential responses (and / or further responses generated based on the additional information) to select one or more potential responses that are in response to the user input, and present an output corresponding to one or more selected responses and / or cause one or more components to perform the corresponding action.

[0024] Among other things, the teachings of this disclosure provide an improved user experience by offering a system capable of selecting among potential responses (natural language data and / or actions) determined by various components. These components include different types such as API components, skill components, LLM-based agents, etc. The techniques of this disclosure allow the system to use multiple different types of components in response to user input, rather than being limited to a single type of component. Additionally, the system is capable of responding using potential responses / actions from more than one component (e.g., by combining or summarizing responses selected from two LLM-based agents or an LLM-based agent and skill components, etc.). The system can respond to user input using selected responses (or portions of selected responses) from multiple components. Subsequently, where one or more of the selected responses include potential actions to be performed, the system can cause the corresponding component to perform the action.

[0025] Systems according to this disclosure will generally be configured to consolidate user licenses and perform the activities disclosed herein only with user approval. Thus, the systems, apparatus, components, and techniques described herein will generally be configured to restrict processing where appropriate and to process user data only in a manner that ensures compliance with all applicable laws, regulations, standards, etc. The system and techniques may be implemented on a geographical basis to ensure compliance with the laws of the respective jurisdictions and agencies where the system components and / or users are located.

[0026] Figure 1 An example is given of a system 100 for determining actions in response to user input using one or more language models. For instance... Figure 1 As shown, the system may include a user device 110 local to user 105, which communicates with system component 120 via network 199. Network 199 may include the Internet and / or any other wide area network or local area network, and may include wired, wireless and / or cellular network hardware.

[0027] System component 120 may include various components such as a Large Language Model (LLM) orchestrator component 130, a personalization context component 165, an action plan execution component 145, an API provider component 150, an LLM agent component 152, a skill component 154, and a TTS component 156. The LLM orchestrator component 130 may include a plan generation component 135, an LLM candidate list component 140, and a response arbitration component 160. In some embodiments, the response arbitration component 160 may exist in a location within system component 120 other than the LLM orchestrator component 130.

[0028] Language modeling (LM) uses various statistical and probabilistic techniques to determine the probability of a given sequence of words appearing in a sentence. Language models analyze the main body of text data to provide a basis for their word predictions. Language models are generative models. In some implementations, a language model can be an LLM. An LLM is an advanced artificial intelligence system designed to process, understand, and generate human-like text based on large amounts of data. In some implementations, an LLM can also be designed to process, understand, and generate multimodal data including audio, text, images, and / or video. LLM models can be built using deep learning techniques such as neural networks and can be trained on extensive datasets for natural language processing, including text (or other types of data, such as multimodal data including text, audio, images, video, etc.) from a wide range of sources, such as books and websites. Compared to language models, LLMs use massive training datasets and can include a large number of parameters (ranging from billions), hence they are referred to as “large” language models. In some implementations, one or more of the language models (and their corresponding operations discussed below) can be the same language model.

[0029] One or more of the language models in this implementation are some embodiments of LLM, where the language models can be Transformer-based seq2seq models involving an encoder-decoder architecture. In an encoder-decoder architecture, the encoder uses bidirectional encoding to produce a representation of the input (e.g., audio, text, image, video, etc.), and the decoder uses that representation to perform a task. In some such implementations, one or more of the language models can be a multilingual seq2seq model with (approximately) 20 billion parameters, pre-trained on a combination of denoising and causal language modeling (CLM) tasks in various languages ​​(e.g., English, French, German, Arabic, Hindi, Italian, Japanese, Spanish, etc.), and the language models can be pre-trained on approximately one trillion lexical units. After training on the CLM task, one or more language models may be able to perform context learning. An example of this LLM is the Alexa Teacher Model (Alexa™).

[0030] One or more of the language models in this implementation are other implementations of LLM, and the one or more language models can be decoder-only architectures. Decoder-only architectures can use left-to-right (unidirectional) encoding of inputs (e.g., audio, text, images, video, etc.). Examples of such LLMs are the generative pre-trained Transformer 3 (GPT-3) and other versions of GPT. GPT-3 has a capacity of (approximately) 175 billion machine learning parameters.

[0031] Other examples of LLMs include BigScience Large Open Science Open Access Multilingual Language Model (BLOOM), Language Model for Dialogue Applications (LaMDA), Bard, Large Language Model Meta AI (LLaMA), Titan Foundation Model, and others.

[0032] In some implementations, the system may include one or more machine learning models in addition to one or more language models. Such machine learning models may receive text and / or other types of data as input (e.g., audio, images, video, etc.) and output text and / or other types of data. These models may be neural network-based models, deep learning models, classifier models, autoregressive models, seq2seq models, etc.

[0033] In one or more implementations of a language model that is an LLM, the input to the LLM can be in the form of prompts. Prompts can be natural language input, such as instructions, for the LLM to generate output based on the prompts. The output generated by the LLM can be natural language output in response to the prompts. In some implementations, the output can be another type of data, such as audio, images, video, etc. Prompts and output can be text in a specific language (e.g., English, Spanish, German, etc.) and / or other types of data, such as audio, images, video, etc. For example, for the example prompt "How do I cook rice?", the LLM can output a recipe for cooking rice (e.g., a step-by-step process represented by text, audio, images, video, etc.). As another example, for the example prompt "I am hungry. What restaurants in the area are open?", the LLM can output a list of restaurants near the user that are open at that time.

[0034] Language models can be configured using various learning techniques. For example, in some implementations, language models can be configured using few-shot learning. In few-shot learning, the model learns how to solve a given problem. In this approach, the model is provided with a limited number of examples (i.e., “few samples”) from a new task, and the model uses this information to adapt to the task and perform well on it. Few-shot learning may require less training data compared to implementing other fine-tuning techniques. As a further example, in some implementations, language models can be configured using one-shot learning, which is similar to few-shot learning but provides the model with only a single example. As another example, in some implementations, language models can be configured using zero-shot learning. In zero-shot learning, the model solves a given problem based solely on the model’s training dataset without examples of how to solve a particular / similar problem. In this approach, the model is provided with data sampled from categories not observed during training, and the model learns to classify the data.

[0035] In some implementations, the LLM orchestrator component 130 can generate cue data representing prompts for input to the language model. For example... Figure 1 As shown, system component 120 receives user input data 127, which can be provided to LLM orchestrator component 130. In some cases, user input data 127 may correspond to various data types, such as text (e.g., text or lexicalized representation of user input), audio, images, video, etc. For example, when the user input is typed natural language user input, the user input data may include input text (or lexicalized) data. For a further example, before LLM orchestrator component 130 receives user input data 127, another component of system 100 (e.g., Automatic Speech Recognition (ASR) component 650) may receive audio data representing user input. ASR component 650 may perform ASR processing on the audio data to determine ASR data corresponding to the user input, which may correspond to the transcription of the user input. (See below for more details.) Figure 6As described, ASR component 650 can determine ASR data including an optimal list of ASR N, which includes multiple ASR hypotheses representing what the user might have said and corresponding confidence scores. ASR hypotheses may include textual data, terminology data, ASR confidence scores, etc., to represent the input utterance. The confidence score of each ASR hypothesis indicates the level of confidence that the ASR component 650 believes the corresponding hypothesis represents what the user said. ASR component 650 can also determine a lexical score corresponding to each term / word of the ASR hypothesis, where the lexical score indicates the level of confidence that the ASR component 650 believes the corresponding term / word was spoken by the user. When the corresponding term is associated with an entity, the lexical score can be identified as an entity score. In some cases, user input data 127 may include the highest-scoring ASR hypothesis of the ASR data. As a further example, in some implementations, user input may correspond to the actuation of a physical button, data representing the selection of a button displayed on a graphical user interface (GUI), image data of gestural user input, combinations of different types of user input (e.g., gestures and button actuation), etc. In such implementations, system 100 may include one or more components configured to process such user input to generate a textual or lexical representation of the user input (e.g., user input data 127).

[0036] User input data 127 may be received at the LLM orchestrator component 130 of system component 120. This LLM orchestrator component may be configured to generate (e.g., one or more) lists of tasks (e.g., steps / actions) to be performed in response to user input, and to select from the list of tasks the task to be performed first (e.g., in the current iteration of processing in system 100), as described below. Figure 4Detailed description. In cases where the planning generation component 135 generates more than one task to be completed in order to perform an action in response to user input, the planning generation component 135 may further maintain and prioritize the task list as the system 100 processes the user input. In other words, as the system 100 processes to complete the task list, the planning generation component 135 may (1) incorporate potential responses associated with completed tasks into data provided to other components of the system 100; (2) update the task list to indicate completed (or attempted, in progress, etc.) tasks; (3) generate updated priorities for tasks still to be completed (or tasks to be retried); and / or (4) determine the current tasks to be completed. The planning generation component 135 may generate task processing data 137 representing the selected task to be completed, as well as various other information required for further processing relative to that task (e.g., user input data 127, indication of the selected task, potential responses associated with previous tasks, remaining tasks, and contextual data associated with user input data 127, as described below regarding...). Figure 4 (detailed description) and send them to LLM candidate list component 140.

[0037] LLM candidate list component 140 may be configured to identify one or more components (e.g., API, skill component 154, LLM agent component 152, TTS component 156, etc.) configured to perform actions related to user input or the current task. LLM candidate list component 140 may also be configured to generate requests (e.g., API calls, incomplete API calls / API call formats, instructions for actions to be performed by components, etc.) that require one or more components to provide potential responses to user input or the current task (e.g., responses to user-provided questions, paragraphs from websites, etc.) and cause those requests to be executed. This may also include potential actions that components are configured to / will perform relative to user input or the current task (e.g., descriptions of potential actions, such as turning on a light, booking a flight, ordering pizza, etc.). Such requests may be reflected in action plan data 142 sent to action plan execution component 145. Action plan execution component 145 can identify requests in action plan data 142, generate executable API calls corresponding to the requests, and cause corresponding components (e.g., API provider component 150, LLM proxy component 152, skill component 154, and / or TTS component 156) to generate action response data 158a-n representing the requested potential response. Individual action response data 158a may be provided by / correspond to a specific response component (one of API provider component 150, LLM proxy component 152, skill component 154, and / or TTS component 156). Action response data 158a-n may include various data types, including audio, text, images, video, etc. In some embodiments, action response data 158a-n may include identifiers of the components providing the data (e.g., component names, alphanumeric values ​​associated with the component, etc.). LLM candidate list component 140 receives and processes action response data 158a-n, and generates potential response data 143a-n representing potential responses (e.g., relevant potential responses, selected potential responses, ranked potential responses, etc.) for further processing (e.g., as described below). Figure 5 (Detailed description). If the LLM candidate list component 140 determines that there are no remaining tasks available for generating a potential response, the LLM candidate list component 140 may send the potential response data 143a-n to the response arbitration component 160.

[0038] In some implementations, the potential response data 143a-n may be determined based on potential responses received from various different components that may be associated with responding to user input data 127. For example, the potential response data 143a-n may include a first potential response from a first component configured to perform a first task determined by the planning generation component 135, a second potential response from a second component configured to perform a second task determined by the planning generation component 135, and so on. The potential response data 143a-n may include more than one potential response associated with a single task. In some implementations, the potential response data 143a-n may be natural language data. In other implementations, the potential response data 143a-n may be multimodal data, such as audio, images, text, video, etc.

[0039] Response arbitration component 160 processes potential response data 143a-n to determine whether potential responses generated for one or more tasks respond to user input. Response arbitration component 160 processes potential response data 143a-n (at least representing generated potential responses) and selects one or more potential responses determined to respond to user input and / or determines that none of the actions respond to user input. For example, response arbitration component 160 may process potential response data 143a-n to determine whether one or more potential responses (e.g., potential responses and / or potential actions) that can be executed by an API respond to the current task. In some embodiments, response arbitration component 160 may generate a natural language summary of one or more of the selected responses and output the natural language summary to user 105. In some embodiments, response arbitration component 160 may also output other types of data to the user, such as audio, images, video, etc., which may be included in or associated with the selected responses.

[0040] If the response arbitration component 160 determines that none of the potential responses responds to the user input, the response arbitration component 160 may send an instruction to the personalization context component 165 to generate additional information (e.g., personalized context data 167) in response to the user input. Alternatively or additionally, the response arbitration component 160 may generate a natural language question to output to the user 105 who requested the additional information. In some embodiments, the response arbitration component 160 may also output other types of data to the user, such as audio, images, video, etc., which may be included in or associated with the selected response. In such cases, system 100 (e.g., the planning generation component 135, the LLM candidate list component 140, and / or the response arbitration component 160) may process, as further described herein with respect to additional information (e.g., personalized context data 167 and / or user-provided additional information), to perform an action in response to the user input.

[0041] Figure 2 Example components and processing of the response arbitration component 160 are shown. Figure 2 As shown, the response arbitration component 160 may include a response prompt generation component 210, a response language model 220, a compliance component 230, an output routing component 240, a self-learning component 250, and a self-learning data retrieval component 270. As discussed above, the response arbitration component 160 processes potential response data 143a-n (representing potential responses generated by one or more components and determined to be associated with user input) to determine whether one or more of the potential responses generated by the system 100 are in response to user input.

[0042] like Figure 2 As shown, the response arbitration component 160 receives potential response data 143a-n (output by the LLM candidate list component 140) at the response prompt generation component 210. The response prompt generation component 210 may also receive personalized context data 167 (from the LLM candidate list component 140 or the personalized context component 165) and context data 242. In some embodiments, the context data 242 may correspond to various contextual information associated with the user input (e.g., conversation history data, historical user input data, weather data, time of day, user ID, device information associated with the device that sent the user input data 127 (e.g., device ID, device status, historical device interaction data, etc.)). As discussed below, the response arbitration component 160 may also receive additional information from the LLM candidate list component 140, such as potential responses to previous tasks associated with the user input (e.g., previous action response data) and the processing performed on the user input data 127.

[0043] Personalized context data 167 may represent one or more contextual signals associated with user 105, such as information associated with user 105's user profile (e.g., user ID, user behavior information, user preferences, age, gender, historical user interaction data, devices associated with the user profile, etc.), which may be determined using, for example, user identification component 695. In some embodiments, indications of user 105 and / or user profile may be included in user input data 127 (e.g., as included in the output of ASR component 650). In some embodiments, personalized context data 167 may include dialogue history data representing one or more user inputs and corresponding system-generated responses in the current interaction between user 105 and system 100.

[0044] As used herein, a “dialogue” can refer to multiple related user inputs and system 100 outputs between the system and the user (e.g., via device 110), which may have been initiated by a single user input that started the dialogue. Therefore, data associated with the dialogue can be associated with the same dialogue identifier, which components of the entire system 100 can use to correlate information throughout the dialogue. Subsequent user inputs within the same dialogue may or may not begin with the user uttering a wake word. Each natural language input can be associated with a different natural language input identifier, and each natural language input identifier can be associated with a corresponding dialogue identifier. Furthermore, depending on the context of the input, other non-natural language inputs (e.g., image data, gestures, button presses, etc.) can be associated with a specific dialogue. For example, a user can verbally initiate a dialogue with the system 100 to request food delivery, and the system can respond by displaying images of available food items, and the user can respond verbally (e.g., “Item 1” or “that one”) or by gesture (e.g., pointing to an item on the screen or giving a thumbs up) or by touching the desired item on the screen. Non-voice input (e.g., gestures, screen touches, etc.) can be part of the conversation, and the associated data can be linked to the conversation's dialogue identifier.

[0045] The response prompt generation component 210 can process potential response data 143a-n, context data 242, and personalized context data 167 (and optionally further information received from the LLM candidate list component 140) to generate prompt data 215 representing a prompt to input to the response language model 220. In some embodiments, the prompt data 215 may be an instruction for the response language model 220 to determine whether one or more of the potential responses represented in the potential response data 143a-n respond to the user input, given other information included in the prompt data 215 (e.g., personalized context data 167, context data 242, potential responses associated with a previous task associated with the user input (e.g., previous action response data), and user input data 127). The prompt data may also be an instruction for the response language model 220 to cause the execution of one or more corresponding actions (e.g., one or more potential actions included in the selected response) and / or to cause the system 100 to inform the user 105 of one or more selected responses, provided that the response language model 220 determines that one or more of the potential responses respond to the user input. For example, in some implementations, the prompt data 215 may further instruct the response language model 220 to generate a natural language summary of a response determined to be in response to one or more choices made by the user input. The prompt data 215 may instruct the response language model 220 so that the system 100 outputs a natural language summary to the user 105.

[0046] In some implementations, the prompt data 215 may also be instructions for the response language model 220 to generate system 100 components and / or user 105's request for additional information if the response language model 220 determines that none of the potential responses respond to the user input. As discussed above, the additional information may be any information that can be used to determine and / or perform actions in response to the user input (e.g., resolving ambiguities associated with the user input and / or tasks associated with the user input).

[0047] In some embodiments, the response prompt generation component 210 may also include in the prompt data 215 a sample processing format to be used by the response language model 220 when processing the prompt. In some embodiments, the response prompt generation component 210 may generate the prompt data 215 according to a template format. For example, the prompt data 215 may conform to a template format that includes: { "You are a conversational AI agent that communicates with users to satisfy their request or ask clarification questions. If applicable, summarize the responses that satisfy the user's request. If applicable, call the corresponding APIs to perform the potential actions that satisfy the user's request. If no response is needed, indicate that."

[0048] This is the user's request. [User input data 127] Here are the potential responses: [Potential response data 143a-n]} In some implementations, a template format may instruct the response language model 220 on how to process and determine whether one or more potential responses are in response to user input. In some implementations, the format may also include a label such as "User:" indicating the following string / lexicon as user input. In some implementations, the format may also include a label "Thought:" instructing the response language model 220 to generate output indicating whether one or more potential responses are determined to be in response to user input or whether additional information is required. In some implementations, the format may also include a "Response:" instruction instructing the response language model 220 to indicate responses determined to be in response to one or more choices of user input, generate a summary of one or more choices of responses, and / or generate a request for additional information.

[0049] Following such a template format and for example, given the user input "How's the weather today?" and the corresponding potential responses output by the LLM candidate list component 140, the response prompt generation component 210 can generate the following example prompt data 215a: { "You are a conversational AI agent that communicates with users to fulfill their requests or ask clarification questions. If no response is required, indicate this."

[0050] This is the user's request: How is the weather today These are the potential responses and potential actions. Skill Component A: The current temperature is 70 degrees Celsius, with a high of 75 degrees and a low of 68 degrees Celsius. Skill Component B: The weather for today is expected to be mostly sunny, but with a possibility of rain in the late afternoon. } For a further example, and for the example user input “please order some pizza for dinner” and the corresponding potential response output by the LLM candidate list component 140, the response prompt generation component 210 can generate the following example prompt data 215b: { "You are a conversational AI agent that communicates with users to fulfill their requests or ask clarification questions. If no response is required, indicate this."

[0051] This is the user's request: Please order some pizzas for dinner. These are the potential responses and potential actions: Component A: User ordered Brooklynstyle pizza from [Company 1 name] API A: Use [Application 1 name] to order pizza from [Company 1 name] API B: Use [Application 2 name] to order pizza from [Company 2 name] } In some implementations, the response prompt generation component 210 may also include instructions in the prompt data to output a response that meets specific conditions. Such conditions may involve generating a response that is unbiased, harmless, and free of profanity (towards a protected group, such as gender, race, age, etc.). For example, prompt data 215 may include “Please generate a polite, respectful, and safe response that does not violate the protected group policy.”

[0052] The response language model 220 processes the prompt data 215 to generate model output data 225 (e.g., response output data 162 or query output data 163) representing a response determined to be in response to one or more choices of user input, a natural language summary of the response to one or more choices, or a request for additional information. Similar to the potential response data 143a-n, the model output data 225 may include various types of data, including audio, text, images, video, etc.

[0053] If the response language model 220 determines one or more potential responses to the user input, the response language model 220 may generate model output data 225 representing the responses selected by one or more choices, or natural language summaries of the responses selected by one or more choices, to output to the user. For example, based on processing the first example prompt data above, the response language model 220 may select one of the potential responses determined to be in response to the user input (e.g., potential responses from skill component A (e.g., the weather skill component)) to generate model output data 225a: {"The current temperature is 70 degrees, with a high of 75 degrees and a low of 68 degrees"}, etc. For a further example, based on processing the first example prompt data provided above, the response language model 220 may select more than one of the potential responses determined to be in response to the user input (e.g., potential responses from both skill component A and skill component B), and generate a summary of the selected responses to generate model output data 225b: {"Today is expected to be mostly sunny, with a high of 75 degrees and a low of 68 degrees, but there may be rain later in the afternoon"}, etc.

[0054] As another example, based on processing the second example prompt data provided above, the response language model 220 may select one of the potential responses identified in response to user input (e.g., a potential response from component A (e.g., personalization context component 165) indicating that the user ordered a Brooklyn-style pizza from [Company 1 name]) to generate model output data 225a: {“OK, I will place an order for Brooklyn-style pizza from [Company 1 name]”} etc. As a further example, based on processing the second example prompt data provided above, the response language model 220 may select more than one of the potential responses identified in response to user input (e.g., potential responses from both component A and API A), and generate a summary of the selected responses to generate model output data 225b: {“OK, I will place an order for Brooklyn-style pizza from [Company name] using [Application 1 name]”} etc.

[0055] Thus, the response language model 220 can select from one or more potential responses from one or more different components (e.g., for the first example cue data, potential responses from skill component A and skill component B, and for the second example cue data, potential responses from component A, API A, and API B) to determine a subset of potential responses that respond to the user input. Thereafter, the response language model 220 can cause the selected response (e.g., a subset of potential responses) or a natural language summary of the selected response to be output to the user.

[0056] In some implementations, the response arbitration component 160 may also generate instructions and send those instructions to components (e.g., APIs, components, agents, etc., as described below). Figure 5(As discussed below), these components are configured to perform potential actions included in the selected response to cause the potential action to be performed (or another component of system 100 configured to cause a component to perform a potential action, such as action planning execution component 145, which is discussed in more detail below). For example, if the selected response includes a potential action to be performed, response language model 220 may further cause the corresponding component to perform the potential action (e.g., cause API A to order a Brooklyn-style pizza from [Company Name] using [Application Name]). In other embodiments, system 100 may not generate and / or send instructions until it receives approval from user 105 to perform the action.

[0057] If the response language model 220 determines that none of the potential responses are in response to the user input and / or ambiguous with respect to one or more of the user input and / or the determined task, the response language model 220 may generate model output data 225 representing the request to be output to the user and / or the personalization context component 165. For example, based on processing the first example prompt data provided above, the response language model 220 may determine that there is ambiguity with respect to the size of the pizza to be ordered, and may generate model output data 225c such as: {“What size pizza should I order?”}, {“What size pizza does the user usually order?”}, etc., to be output to the user and / or sent to the personalization context component 165.

[0058] As further discussed below, one or more of the components discussed herein (e.g., plan generation component 135 and / or LLM candidate list component 140) may be able to determine whether there is ambiguity in the user input or the current task, and may determine that additional information is needed. In response to such determination, the components may also be configured to send a request for such additional information to a response arbitration component 160, which may process the request as described herein to generate a request for additional information to be sent to the personalization context component 165 or output to the user 105 to solicit additional information. In some embodiments, the response arbitration component 160 may send the request for additional information to the action plan execution component 145, which may cause the request to be output to the user 105 to solicit additional information.

[0059] The Response Language Model 220 may send model output data 225 to a compliance component 230, which is configured to determine whether the model output data generated by the Response Language Model 220 is suitable for output to user 105. In other words, the compliance component 230 processes the model output data 225 to determine whether the model output data 225 includes any inappropriate / sensitive information (e.g., confidential information, offensive language, etc.) that should not be output to user 105. In some embodiments, the compliance component 230 may be configured to compare the model output data 225 with one or more words that are determined to be inappropriate / sensitive and should not be output to user 105. In some embodiments, the compliance component 230 may include / implement an ML model. For example, the ML model may process the model output data 225 to determine whether the model output data 225 includes any inappropriate / sensitive information. During training, the ML model may treat multiple training natural language inputs as input, where the task of the ML model is to classify the natural language inputs as including or excluding inappropriate / sensitive information. The output of the ML model, obtained by processing the natural language input for training (e.g., 0, 1, values ​​between 0 and 1), can be compared with corresponding labels indicating whether the natural language input contains inappropriate / sensitive information. Based on the comparison, one or more parameters of the ML can be configured. In some implementations, the ML model can be a classifier.

[0060] If the compliance component 230's output indicates that the model output data 225 includes information unsuitable for output to user 105, the compliance component 230 may cause downstream components to suspend further processing of the model output data 225. In some embodiments, the response arbitration component 160 may cause the response language model 220 to generate new model output data 225 for evaluation by the compliance component 230. For example, the response arbitration component 160 may cause the response prompt generation component 210 to generate new prompt data, which may include prompt data 215, model output data 225, and an indication that the model output data 225 is unsuitable for output to user 105. The new prompt data may be an instruction to generate new model output data suitable for output to user 105.

[0061] If the output of compliance component 230 indicates that model output data 225 is suitable for output to a user, then compliance component 230 may send model output data 225 to output routing component 240. Output routing component 240 processes model output data 225 to determine one or more components that are prompted to process in response to model output data 225. In other words, output routing component 240 parses model output data 225 to determine one or more components to which model output data 225 should be routed (or prompted to process based on model output data 225).

[0062] For example, in the case where the response language model 220 determines one or more potential responses to user input and generates model output data 225 including one or more selected responses (or natural language summaries of one or more selected responses) / requests for additional information, the output routing component 240 may parse the model output data 225 to determine the selected response / natural language summary and send response output data 162 or query output data 163 corresponding to the selected response / natural language summary to a component configured to generate the corresponding data for output to user 105. For example, the output routing component 240 may send response output data 162 / query output data 163 to a TTS component (e.g., TTS component 156), which may be processed as described herein to generate output audio data including synthesized speech corresponding to response output data 162 / query output data 163, which the system 100 may send to user device 110 for output to user 105. In some implementations, system 100 may further include a component configured to generate visual output data (e.g., output image and / or video data) corresponding to response output data 162 / query output data 163, which may be sent to user device 110 for output to a user. Similar to model output data 225, the corresponding response output data 162 / query output data 163 may include various types of data, including audio, text, images, video, etc.

[0063] For a further example, in an implementation where model output data 225 includes a response (e.g., response output data 162) containing a selection of one or more potential actions to be performed, output routing component 240 may process as described above to determine one or more selected responses / natural language summaries and send response output data 162 to one or more components associated with the selected responses. In such implementations, response output data 162 may also include instructions for one or more components to perform a potential action corresponding to the selected response. For example, in some implementations, the component corresponding to a potential response included in potential response data 143a-n may pause processing required to perform the potential action included in the potential response after the potential response is generated and wait for instructions from system 100 to perform the potential action. Thus, output routing component 240 may include instructions in response output data 162 to cause the component to perform the potential action. In some implementations, output routing component 240 may generate API calls configured to cause the component to perform an action.

[0064] In some implementations, where model output data 225 includes a response containing a selection of one or more potential actions to be performed, response output data 162 may further request authorization from user 105 to perform one or more potential actions in response to user input. Upon receiving a request for authorization (e.g., via subsequent user input), response arbitration component 160 may generate and send a corresponding instruction (or API call) to perform one or more potential actions in response to user input. In some implementations, system 100 may store data indicating prior authorization to perform one or more potential actions in response to user input (or one or more actions similar to one or more potential actions), in which case response arbitration component 160 may use such data as authorization to perform one or more potential actions. For example, user 105 may have previously authorized a set of actions (e.g., turning on outdoor lights). Subsequently, system 100 may determine one or more potential actions to be performed in response to user input data 127. If system 100 determines that one or more actions are included in the set of actions previously authorized by user 105, system 100 may not request further authorization before allowing one or more potential actions to be performed.

[0065] For a further example, when the response language model 220 generates model output data 225 including a request for additional information (in response to the response language model 220 determining that none of the potential responses are in response to user input and / or ambiguous with respect to user input and / or one or more tasks) (this may be determined by the output routing component 240 based on, for example, the model output data 225 including a question), the output routing component 240 may parse the model output data 225 to determine whether the request for additional information should be sent to the personalization context component 165 and / or output to the user 105. In some embodiments, the response language model 220 may include in the model output data 225 an indication of whether the request for additional information should be sent to the personalization context component 165 and / or output to the user 105. In some embodiments, unless otherwise indicated in the model output data 225, the output routing component 240 may determine to send the request for additional information to the personalization context component 165 before outputting the request for additional information to the user 105. If the personalization context component 165 cannot resolve the ambiguity (or the components of system 100 cannot resolve the ambiguity using the personalization context data generated by the personalization context component 165), the output routing component 240 may determine that the request for additional information should be output to user 105.

[0066] Figure 3 Example components are shown where response output data 162 and / or query output data 163 can be sent. For example... Figure 3As shown, query output data 163 can be sent to personalization context component 165. As discussed above, query output data 163 can be sent to personalization context component 165 based on model output data 225 including requests for additional information and output routing component 240 determining that the request for additional information should be sent to personalization context component 165. Personalization context component 165 can process the data to generate personalized context data 325 associated with query output data 163 (e.g., requests for additional information). Personalization context data 325 can then be sent to another component of system 100, such as planning generation component 135, LLM candidate list component 140, and / or response arbitration component 160, which can be processed as discussed above.

[0067] Personalization context component 165 can be configured to determine and return contextual information associated with user input. In some embodiments, personalization context component 165 may query various components and / or stores (e.g., profile store 670) to obtain contextual information. In some embodiments, personalization context component 165 may include a store containing one or more portions of contextual information. In other embodiments, personalization context component 165 may be / implement an LLM. In such embodiments, personalization context component 165 may be fine-tuned based on personalization information for one or more users, as discussed in more detail below. Furthermore, in such embodiments, personalization context component 165 (or system 100) may include a personalized context prompt generation component (not illustrated) that can be configured to generate prompts including user input data 127 (or a representation of user input intent) to be entered into the LLM. The prompt may be an instruction for the LLM to determine one or more portions of the contextual data associated with the prompt (e.g., personalized contextual data 167).

[0068] like Figure 3 As further shown, response output data 162 and / or query output data 163 may be sent to one or more of the TTS components 156. Response output data 162 may also be sent to the skill component 154, the LLM agent component 152, and / or the API provider component 150 (e.g., the device controller component).

[0069] The TTS component 156 can be configured to process text or lexicalized input to generate audio data representing synthesized speech corresponding to the text or lexicalized input spoken by the synthesized speech. The following section discusses... Figure 6 The processing of TTS component 156 is discussed in detail.

[0070] Skill component 154 can be software similar to a software application running on system component 120. That is, skill component 154 enables system component 120 to perform specific functionalities to provide data or generate output for some other request. As used herein, "skill component" can refer to software that can be placed on a machine or virtual machine (e.g., software that can be launched in a virtual instance when invoked). Skill components can be software customized to perform one or more actions, such as those instructed by a business entity, device manufacturer, user, etc. Things described herein as skill components may be referred to using many different terms (such as action, robot, app, etc.). System component 120 may be configured with more than one skill component 154. For example, a weather service skill component enables system component 120 to provide weather information, a car service skill component enables system component 120 to book trips relative to taxi or ride-sharing services, and a restaurant skill component enables system component 120 to order pizzas relative to a restaurant's online ordering system. Skill component 154 can cooperate between system component 120 and other devices (such as user device 110) to perform certain functions. Skill component 154 may include hardware, software, firmware, etc. that may be dedicated to a specific skill component 154 or shared among different skill components 154.

[0071] LLM agent component 152 may correspond to one or more LLM agents. LLM agent component 152 may correspond to a custom instantiation of an LLM (and other components) configured to handle user input related to a specific domain / functionality. In some implementations, LLM agent component 152 may be configured to handle specific use cases via specific prompt generation, LLM fine-tuning, etc. For example, LLM agent component 152a may be configured to handle user input / tasks related to information retrieval, LLM agent component 152b may be configured to handle user input / tasks related to shopping, LLM agent component 152c may be configured to handle user input / tasks related to ordering food from various restaurants, LLM agent component 152d may be configured to handle user input / tasks related to ordering food from a specific restaurant (e.g., a specific pizza restaurant), LLM agent component 152e may be configured to handle user input / tasks related to booking hotels, LLM agent component 152f may be configured to handle user input / tasks related to booking flights, etc.

[0072] For example, TTS component 156 may process as discussed above to generate output audio data corresponding to response output data 162 and / or query output data 163 (e.g., output audio data corresponding to one or more actions / natural language summaries in response to user input, requests for additional information, etc.). System 100 may then send the output audio data to device 110 for output to user 105. As a further example, LLM agent component 152 may process as discussed above to cause one or more corresponding actions to be performed in response output data 162. As another example, the device controller component of API provider component 150 may cause the device to perform an action corresponding to response output data 162. For example, if the action represented by response output data 162 is to turn on a living room light (e.g., “turn_on_device(device="living room light")”), the device controller component may identify the corresponding living room light and instruct it to turn on (e.g., change its state to {state: ON}).

[0073] In some implementations, response output data 162 and / or query output data 163 may be sent to action plan execution component 145, which may then send the response output data 162 and / or query output data 163 to the corresponding component / API (e.g., TTS component 156, skill component 154, LLM agent component 152, API provider component 150, etc.).

[0074] In some implementations, the response arbitration component 160 may be configured to further process data representing potential responses to user input, which are generated by one or more other components of system 100 not included in the LLM orchestrator component 130. For example, the response arbitration component 160 may also process data from orchestrator component 630 (hereinafter referred to as...). Figure 6 (Discussed in detail) Receives data representing potential responses to user input (e.g., the output of skill component 154), wherein the orchestration of processes performed to generate the potential responses is performed by orchestrator component 630, not by LLM orchestrator component 130. In such embodiments, response arbitration component 160 may also be configured to arbitrate between a first potential response received as a result of processing by LLM orchestrator component 130 and a second potential response received as a result of processing by orchestrator component 630. As discussed above, response arbitration component 160 may select one or more portions (e.g., potential actions, potential responses, etc.) of the first and / or second potential responses determined to be in response to user input, and cause one or more portions (or a summary representation of one or more portions) to be output and / or perform a potential action corresponding to the selected response.

[0075] In some implementations, data received from orchestrator component 630 may be included in potential response data 143a-n. For example, orchestrator component 630 may be configured to perform a function potentially related to user input (e.g., causing another component to perform a function), such that LLM candidate list component 140 may cause orchestrator component 630 to generate potential responses potentially responding to user input, which may be included in potential response data 143a-n sent to response arbitration component 160. Further details regarding the processing by which LLM candidate list component 140 generates potential response data 143a-n are provided below. Figure 5 Let's have a discussion.

[0076] As discussed above, the response arbitration component may include a self-learning component 250. The self-learning component 250 may be configured to collect, store, and distribute various feedback related to the processing of one or more components discussed herein with respect to user input. The self-learning component 250 may use the feedback to enable one or more components to be updated / trained based on the various feedback. In some embodiments, the self-learning component 250 may be located elsewhere in system 100 besides the response arbitration component 160.

[0077] For example, the self-learning component 250 may collect and store various information associated with processing relative to user input (e.g., feedback signals 255a-n), such as the identified task associated with the execution of an action in response to user input, the selected task, the priority of the task, the selected API, the potential response generated by the API, interaction history data, dialogue history data, or as described below. Figures 4 to 5Any other data generated during the processing of the discussion. The self-learning component 250 may also collect information associated with user satisfaction with the processing of system 100 (e.g., feedback signals 255a-n). The self-learning component 250 may determine such user satisfaction information based on implicit and explicit feedback signals (e.g., feedback signals 255a-n). For example, explicit feedback signal 255a may be a subsequent user input associated with a response generated by system 100 (e.g., “Add milk, please.”), response arbitration component 160 receiving different outcomes of the processing performed by orchestrator component 630 and LLM orchestrator component 130 (e.g., a first potential response from orchestrator component 630 includes the potential action “add milk to your grocery list,” while a second potential response from LLM orchestrator component 130 includes a request for additional information “can you specify the list?”), requests for additional information output to the user and the user’s corresponding response, the quality of the generated system-determined responses to requests for additional information, etc.

[0078] For a further example, implicit feedback signal 255b could be subsequent user input associated with the response generated by system 100 (e.g., “Please add milk.”), response arbitration component 160 receiving different results from the processing performed by orchestrator component 630 and LLM orchestrator component 130 (e.g., a first potential response from orchestrator component 630 includes the potential action “add milk to your grocery list,” while a second potential response from LLM orchestrator component 130 includes the potential action “add milk to your shopping list”), subsequent user input resulting from the user interrupting the output of the system-generated response (e.g., the user provides interrupting user input “no, add it to my grocery list” before completing the output of the system-generated response “add milk to your grocery list”), and system-determined quality of the system-generated response attempting to preempt subsequent user input (e.g., a preemptive system-generated response “add milk to your shopping list” might receive a better response than the preemptive system-generated response “do you want to add milk?”). (to your shopping list?) (lower quality score), etc.

[0079] Various data collected by the self-learning component 250 (e.g., feedback signals 255a-n) can be used to update / train one or more components of the arbitration component. For example, if a user previously provided subsequent user input requesting that future outputs be kept to a minimum number of words, the self-learning component 250 can receive the subsequent user input as an explicit feedback signal 255a and can use the explicit feedback signal 255a to update the response prompt generation component 210. Figure 2 As shown, the self-learning component 250 can generate self-learning data 260a / 260b representing training data including explicit feedback signals 255a, and send self-learning data 260a to the response prompt generation component 210 and / or send self-learning data 260b to the response language model 220. The response prompt generation component 210 can be updated / trained based on the self-learning data 260a, such that for user input associated with a user providing subsequent user input, the response prompt generation component 210 can include in the prompt data an indication that the response language model 220 should generate a short and concise response for the user. The response language model 220 can be updated / trained based on the self-learning data 260b, such that the response language model 220 is better configured to generate short and concise responses. In some embodiments, various data (e.g., feedback signals 255a-n) collected by the self-learning component 250 can be used by the system 100 to update / train one or more components of the system 100. In such implementations, self-learning component 250 may send self-learning data 260c to another component of system 100 to update / train that component. For a further example, if a user previously responded to a system-generated response to the user input "Add eggs to my list" by providing subsequent user input "Please add milk," self-learning component 250 may receive the subsequent user input as an explicit feedback signal 255b and may use the explicit feedback signal 255b to update the user profile associated with the user (e.g., embodied in profile storage 670) and / or update the storage / index of personalization context component 165. Self-learning component 250 may generate self-learning data 260c representing training data including the explicit feedback signal 255b and send self-learning data 260c to profile storage 670 and / or personalization context component 165. For example, the personalized context component 165 can be updated / trained based on self-learning data 260c, such that processing a future input like "add eggs to my list" can cause the personalized context component 165 to generate personalized context data indicating that the user has previously added milk to their list as well. System 100 can use this personalized context data to generate the response "Would you also like me to add milk to your list?".

[0080] In some implementations, system 100 may update / train one or more components of system 100 during offline operation. In such implementations, the system may store feedback signals 255a-n in storage (e.g., memory storage discussed herein) until system 100 ceases online operation. Thereafter, system 100 (e.g., response arbitration component 160) may cause self-learning data retrieval component 270 to retrieve stored feedback data 275a-n representing the feedback signals 255a-n received during online operation. In some implementations, the stored feedback data 275a-n may correspond to self-learning data associated with updating / training a specific component. The stored feedback data 275a-n may be sent to a self-learning component, which may be processed as described above, such that one or more components of system 100 are updated / trained.

[0081] Figure 4 An example of the planned generation of component 135 is shown. Figure 4 As shown, the plan generation component 135 may include a plan prompt generation component 410, a plan generation language model 420, a task selection prompt generation component 430, and a task selection language model 440.

[0082] like Figure 4As further shown, user input data 127 is received at the plan prompt generation component 410. The plan prompt generation component 410 processes the user input data 127 to generate prompt data 415 representing prompts for input to the plan generation language model 420. In some embodiments, the plan prompt generation component 410 may also receive indications of one or more remaining tasks to be completed relative to the user input data 127. The tasks to be completed may correspond to tasks for which the system 100 has not yet generated potential responses (e.g., API provider component 150, LLM agent component 152, skill component 154, and / or TTS component 156 have not yet generated action response data). Similarly, completed tasks may correspond to tasks for which the system 100 has generated potential responses (API provider component 150, LLM agent component 152, skill component 154, and / or TTS component 156 have generated action response data). For example, if the current processing iteration relative to user input data 127 is a subsequent processing iteration (e.g., the system previously determined that more than one task needs to be completed to perform an action in response to user input data 127, and a potential response has previously been generated for at least the first of the more than one task), the planning prompt generation component 410 may also receive indications of remaining tasks to be completed. In such embodiments, the planning prompt generation component 410 may also receive indications of completed tasks and / or potential responses to processes. The planning prompt generation component 410 may also receive context data 242 representing various context signals associated with user input data 127, such as weather information, time of day, device information associated with the device that sent user input data 127 (e.g., device ID, device status, historical device interaction data, etc.). Such prompt data 415 may be generated based on a combination of user input data 127 and context data 242 (and, in some embodiments, indications of remaining tasks, completed tasks, and / or potential responses). In some embodiments, prompt data 415 may be further generated based on personalized context data 167.

[0083] The prompt generation component 410 may receive personalized context data 167 from the personalization context component 165. As discussed above, the personalization context component 165 may be configured to determine contextual information associated with user input and return it to the prompt generation component 410, which may combine the contextual information with the user input data 127 to generate prompt data 415.

[0084] As discussed above, the personalized context component 165 can be made to generate and return personalized context data 167 based on system 100's determination that additional information is needed to generate a potential response to a task associated with the user input. For example, one or more components of system 100 (e.g., planning generation language model 420, task selection language model 440, candidate list language model 540, response arbitration component 160) can determine that there is ambiguity in the user input (or data determined / generated as a result of processing relative to the user input). In such examples, the personalization context component 165 may receive user input, the current task, and / or model output data indicating ambiguity / the need to determine additional information (e.g., model output data indicating "Does the user prefer to use [Music Streaming Service 1] or [Music Streaming Service 2] for playing music?", "I need to determine whether the user prefers [Music Streaming Service 1] or [Music Streaming Service 2] for playing music?", etc.). The personalization context component 165 may process the data as described above to generate personalized context data 167 (e.g., "The user prefers [Music Streaming Service 1]").

[0085] In some implementations, the planned prompt generation component 410 (or another component of system 100) may process personalized context data 167, user input data 127, and / or potential responses associated with user input data 127 to generate a natural language representation of the user input (represented by user input data 127), which is updated to include contextual information from the personalized context data 167 (e.g., context rewriting of the user input). The planned prompt generation component 410 may then process the updated user input data to generate prompt data 415.

[0086] In some implementations, the prompt data 415 may be instructions for the planning generation language model 420 to determine, in light of other information included in the prompt data 415 (e.g., personalized context data 167, indications of remaining tasks, indications of completed tasks, and / or corresponding potential responses), to perform one or more tasks (e.g., steps / actions) in response to user input.

[0087] In some embodiments, the planned prompt generation component 410 may also include in the prompt data 415 a sample processing format to be used by the planned prompt generation language model 420 when processing the prompt. In some embodiments, the planned prompt generation component 410 may generate the prompt data 415 according to a template format. For example, the prompt data 415 may conform to the following template format: { If necessary, create a new task to help complete the request for [user input data 127 (or a representation of a determined intent of the user input data 127)]. Here are the completed tasks, potential responses, user inputs, and context so far. [completed tasks, the potential responses, dialog history, context data 242, personalized context data 167] These are the remaining tasks to be completed. [Remaining task data] Based on the result, create new tasks to be completed, if necessary. Returns the tasks as an array. } In some implementations, the template format may instruct the planning generation language model 420 on how to process the data to generate one or more tasks (e.g., steps) to be completed. In some implementations, the format may also include instructions, such as a label “User:”, indicating the following string / lexicon as user input. In some implementations, the format may also include a “Thinking:” label, instructing the planning generation language model 420 to generate output representing the planning generation language model 420’s determined interpretation of the user input and / or the action to be taken (e.g., the user is requesting [the intent of the user input], the user is attempting [the intent of the user input], information needed to determine [the information required to properly process the user input], etc.). In some implementations, the format may also include an instruction “Observation:”, indicating the following string / lexicon as the result of the action determined by the planning generation language model 420 / an interpretation of the result of the action determined by the planning generation language model 420 (e.g., task completed and / or its potential response). In some implementations, the format may also include a “response:” instruction that guides the planned language model 420 to generate a response to the prompt (e.g., one or more tasks to be completed).

[0088] Following such a template format and given the user input "turn on all of the lights except the garage", the planned prompt generation component 410 can generate the following example prompt data 415a: { If necessary, create a new task to assist in fulfilling the request to turn on all the lights except those in the garage. Here are the completed tasks, their potential responses, user inputs, and context so far. [] These are the remaining tasks to be completed: [] Based on the results, create new tasks to be completed if necessary.

[0089] Returns tasks as an array.

[0090] } As an example of user input associated with more than one task, system 100 may receive the user input “Please order some pizza for dinner” and determine a list of tasks for “identify user pizza preference” and “find application that enables ordering of pizza”. System 100 may then process the input as described below to select and complete the “identify user pizza preference” task. The prompt generation component 410 may process the user input, corresponding context data, the remaining task list, and potential responses (e.g., user pizza preference, for example, determined by the personalization context component 165) to generate the following example prompt data 415a: { If necessary, create a new task to help fulfill the request to order some pizza for dinner. This is a list of tasks completed to date, their potential responses, user input, and context: Tasks completed: Identify user pizza preference: The user ordered a Brooklyn-style pizza from [Company name] These are the remaining tasks to be completed: Find apps to order pizza Based on the results, create new tasks to be completed if necessary.

[0091] Returns tasks as an array.

[0092] } In some implementations, the planned prompt generation component 410 may also include instructions in the prompt data to output a response that meets specific conditions. Such conditions may involve generating a response that is unbiased, harmless, and free of profanity (towards a protected group, such as gender, race, age, etc.). For example, prompt data 415 may include “Please generate a polite, respectful, and safe response that does not violate protected group policies.”

[0093] The planned language model 420 processes the prompt data 415 to generate model output data 425, which represents one or more prediction tasks to be performed in response to user input. For example, based on processing the first example prompt data provided above, the planned language model 420 may output model output data 425a: {"turn on all of the lights except the garage light"), etc. For a further example, as discussed above, based on processing the prompt data corresponding to the user input "Please order some pizza for dinner," the planned language model 420 may output model output data 425b: {"Identify user pizza preferences"; "Find an app that can order pizza"}, etc. After the first task "Identify user pizza preferences" is completed, and based on processing the second example prompt data provided above, the planned language model 420 may also output model output data 425c: {"Find an application to order pizza", "Find API to order [Company name] pizza"}, etc. In some implementations, thresholds for determining one or more tasks may encourage the planned generative language model 420 to generate multiple prediction tasks for a given user input, wherein the system 100 may parse and filter the task list during downstream processing (e.g., during processing by the task selection language model 440). For example, based on processing the first example prompt data provided above, the planned generative language model 420 may output model output data 425d: {"Turn on all lights except the garage light", "Turn on all lights", "Identify which garage light", "Turn on all lights then turn off the garage light", "Turn on all lights where the user is located", "Turn on kitchen lights, living room lights, dining room lights, hallwayslights", "Turn on all lights on the first floor"}, etc.

[0094] Model output data 425 is sent to task selection prompt generation component 430, which processes the model output data 425 to generate prompt data 435 representing prompts for input to task selection language model 440. In some embodiments, such prompt data 435 may be generated based on a combination of user input data 127, personalized context data 167, prompt data 415, and / or model output data 425. In some embodiments, planning generation component 135 may include another component that parses the model output data 425 to determine one or more tasks and may send representations of one or more tasks to task selection prompt generation component 430.

[0095] In some embodiments, the prompt data 435 may be an instruction for the task selection language model 440 to select the task to be completed first (e.g., during the current processing iteration) from among one or more tasks, given information included in the prompt data 435 (e.g., user input data 127, personalized context data 167, and one or more tasks). In some embodiments, the prompt data 435 may also include instructions for the task selection language model 440 to determine the priority of one or more tasks (e.g., an ordered list indicating the order in which one or more tasks should be completed). As discussed above with respect to the planning prompt generation component 410, in some embodiments, the task selection prompt generation component 430 may also include in the prompt data 435 a sample processing format to be used by the task selection language model 440 when processing the prompt. Similarly, in some embodiments, the task selection prompt generation component 430 may generate the prompt data 435 according to a template format, such as: { Given the ultimate goal of [user input data 127 (or a representation of a determined intent included in the user input data 127)], select the top-priority task. Here are the completed tasks, their potential responses, and user inputs so far. [completed tasks, potential responses associated with the tasks, dialog history, context data, personalized context data 167] The following are the task candidates: [Remaining tasks] Returns to your selected task, or None if the goal has been achieved, or indicates any ambiguities. } In some implementations, the template format may guide the task selection language model 440 on how to select tasks and / or prioritize one or more tasks. In some implementations, as discussed above, the format may also include instructions for "User:", "Thinking:", "Action:", "Observation:", and / or "Response:" indicators.

[0096] Following such a template format, and given the first example user input "turn on all lights except the garage" provided above, the task selection prompt generation component 430 can generate the following example prompt data 435a: { Given the ultimate goal of turning on all the lights except the garage, select the top-priority task. This is a list of tasks completed to date, their potential responses, user input, and context: [] The following are the task candidates: Turn on all lights except the garage light. Returns to the task you selected, or None if the goal has been achieved, or indicates that there is ambiguity.

[0097] } For a further example, given the second example above where the user inputs "Please order some pizza for dinner," the task selection prompt generation component 430 can generate the following example prompt data 435b: { Given the ultimate goal of "Please order some pizza for dinner," select the top-priority task. This is a list of tasks completed to date, their potential responses, user input, and context: Task completed: Identifying user pizza preferences: The user ordered a Brooklyn-style pizza from [Company Name]. The following are the task candidates: Find an application that sells pizza. Find APIs that sell pizzas for [Company Name] Returns to the task you selected, or None if the goal has been achieved, or indicates that there is ambiguity.

[0098] } In some implementations, the task selection prompt generation component 430 may also include instructions in the prompt data to output responses that meet specific conditions. Such conditions may involve generating responses that are unbiased, harmless, and free of profanity (towards a protected group, such as gender, race, age, etc.). For example, the prompt data may include "Please generate a polite, respectful, and safe response that does not violate protected group policies."

[0099] The task selection language model 440 processes the prompt data 435 to generate model output data representing the priority of the task to be completed first and / or one or more tasks. For example, based on processing the first example prompt data provided above, the task selection language model 440 may output model output data such as: {"1. Turn on all lights except the garage light"}. For a further example, based on processing the second example prompt data provided above, the task selection language model 440 may output model output data such as: {"1. Find the API that sells [Company Name] pizza"}. In some implementations, during the process of the task selection language model 440 selecting one or more tasks and / or prioritizing one or more tasks, the task selection language model 440 may update the task list to remove any redundant and / or conflicting tasks. For example, for the second example prompt data, the task selection language model 440 may determine that the remaining tasks "Find the API that sells [Company Name] pizza" and "Find the API that sells [Company Name] pizza" are redundant, and that "Find the API that sells [Company Name] pizza" has a higher priority. Therefore, the task selection language model 440 may remove the task "Find the API that sells [Company Name] pizza" from the remaining task list. Subsequently, the planning generation component 135 (or another component of the planning generation component 135) processes the model output data of the task selection language model 440 to determine task processing data 137 representing user input data 127, personalized context data 167, and / or tasks selected by the task selection language model 440 as the first tasks to be completed. In some embodiments, the task processing data 137 may include one or more remaining tasks and / or may indicate the priority of one or more tasks as determined by the task selection language model 440. The task processing data 137 may be sent to the LLM candidate list component 140, as described below. Figure 5 Provide a detailed description.

[0100] Figure 5 Example processing of LLM candidate list component 140 is shown. Figure 5 As shown, the LLM candidate list component 140 may include an index store 530, an API candidate list component 520, a candidate list suggestion generation component 510, and a candidate list language model 540.

[0101] like Figure 5As further shown, task processing data 137 is received at the candidate list prompt generation component 510. The candidate list prompt generation component 510 processes the task processing data 137 to generate prompt data 515 representing prompts for input to the candidate list language model 540. In some embodiments, such prompt data 515 may be generated based on a combination of task processing data 137 (e.g., user input data 127, the selected task, remaining tasks, potential responses associated with one or more previous tasks, etc.) and related API data 535 representing one or more APIs associated with user input data 127 and / or the current task.

[0102] Relevant API data 535 can be generated by API candidate list component 520, which can be configured to retrieve one or more (e.g., the top k) relevant APIs associated with user input data 127 or the current task. In some implementations, APIs may correspond to various components. For example, components may correspond to rule-based components, ML-based components, LLM-based components, etc. (such as personalization context component 165, skill component 154, LLM agent component 152, TTS component 156, orchestrator component 630, etc.). In some implementations, APIs may correspond to components.

[0103] API candidate list component 520 may use a retrieval-based method to retrieve one or more relevant APIs from index store 530, which may store various information associated with multiple APIs, such as API descriptions, API parameters (e.g., parameter inputs / outputs), identifiers of components providing the APIs (e.g., personalization context component 165, skill component 154, LLM proxy component 152, TTS component 156), etc. For example, API candidate list component 520 may compare one or more APIs included in index store 530 with user input or the current task to determine one or more APIs (top k) corresponding to the user input or the current task (e.g., APIs semantically similar to the user input or the current task, APIs capable of performing the current task, etc.). In some embodiments, API candidate list component 520 (or another component of API candidate list component 520) may determine an encoded representation of the user input or the current task and compare that encoded representation with an encoded representation of the API's API description (e.g., using cosine similarity) to determine whether the API is semantically similar to the user input or the current task. An API description may correspond to a description of one or more functions that the API is configured to perform and / or other information associated with the API (e.g., API call formatting structure, including input parameters, historical accuracy / defect rate, historical latency values, etc.). In some embodiments, the API description may also include one or more paradigms associated with the use of the API (e.g., example user input, corresponding API call, and example API output). If the semantic similarity value reaches or exceeds a threshold, the API (and optionally the API description) may be included in the relevant API data 535. In some embodiments, the API candidate list component 520 may further use contextual information to determine the relevant API data 535, including personalized contextual data 167, accuracy / defect rate values ​​associated with the API, and / or historical latency values ​​associated with the API (e.g., which may be included in the API description). In some embodiments, an index store 530 may be included in the API candidate list component 520. Similar processing may be performed to determine one or more components that are semantically similar to user input or the current task, which may be included in the relevant API data 535. API retrieval can send relevant API data 535 to the candidate list prompt generation component 510.

[0104] In some implementations, the prompt data 515 may be instructions for the candidate list language model 540 to determine, given information (e.g., determining one or more APIs to be processed relative to the user input or the current task, such as personalized context data 167, current task, and relevant API data 535), to perform processing on the APIs. As discussed above with respect to the plan prompt generation component 410 and the task selection prompt generation component 430, in some implementations, the candidate list prompt generation component 510 may also include in the prompt data 515 a sample processing format to be used by the candidate list language model 540 when processing the prompts. Similarly, the candidate list prompt generation component 510 may generate the prompt data 515 according to a template format, such as: { You are an AI agent that finds and executes an API to complete the task of [user input data 127 / current task]. Here is a list of relevant APIs available. [API data 535] Use the following format: Thought: Think about what to do. API: API calls compatible with the task. Observation: The result of the API call. Summary: Summary results from the API call. If no appropriate API is found, it is summarized as "Nothing was found". } Following a template format such as this, and for the selected task "Turn on all lights except the garage light" and the corresponding API data, the candidate list suggestion generation component 510 can generate the following example suggestion data 515a: { You are an AI agent that finds and executes an API to complete the task of turning on all the lights except the garage light. Here is a list of available APIs: Let's Chat API Classic NLU API Smart Home Skills Use the following format: Thinking: Thinking about what to do API: Task-compatible API calls Observation: Results of API calls Abstract: Summary of API call results If no suitable API is found, the result is summarized as "Nothing found".

[0105] } In some implementations, the candidate list prompt generation component 510 may also include instructions in the prompt data to output responses that meet specific conditions. Such conditions may involve generating responses that are unbiased, harmless, and free of profanity (towards the protected group, such as gender, race, age, etc.). For example, the prompt data may include "Please generate a polite, respectful, and safe response that does not violate the protected group policy."

[0106] Candidate list language model 540 processes cue data 515 to generate one or more API calls, which correspond to requests for a corresponding API to return a potential response to user input / current task and / or an API configured / to perform a potential action (e.g., a natural language description of the potential action) relative to user input and / or current task. Thus, in some embodiments, candidate list language model 540 may generate API calls for a subset of APIs represented in cue data 515. Candidate list language model 540 may generate one or more API calls (including required input parameters) by applying contextual learning (e.g., one-shot / few-shot learning) to cold-start APIs. For example, in embodiments where associated API data 535 includes an API description, candidate list language model 540 may use one or more paradigms included in the API description (included in cue data 515) to determine one or more input parameters for the API call. In some embodiments, candidate list language model 540 may be fine-tuned on such paradigms (e.g., during offline or runtime processing) such that candidate list language model 540 is able to determine one or more input parameters for a given API call.

[0107] During the processing of the candidate list language model 540 and after the generation of one or more requests, the candidate list language model 540 may cause one or more requests to be executed. For example, as Figure 5 As shown, the candidate list language model 540 can send action plan data 142 representing one or more requests to the action plan execution component 145, thereby causing the execution of one or more requests included in the action plan data 142. For example, the action plan execution component 145 can process the action plan data 142 to generate action data 147a-n. Action data 147a may represent, for example, instructions for a specific API to process relative to user input and / or the current task (e.g., an executable API call determined / generated based on the action plan data 142). In embodiments where the action plan data 142 includes a request corresponding to an incomplete API call, the API call format, and an indication of an action to be performed by one or more components, the action plan execution component 145 can be configured to generate an executable API call corresponding to the action plan data 142. In some embodiments, the action plan execution component 145 can generate action data 147a-n to represent instructions to provide a response including a potential response / potential action relative to user input and / or the current task.

[0108] Action plan execution component 145 can send action data 147a-n to API provider component 150, LLM agent component 152, skill component 154, and / or TTS component 156. API provider component 150 may include one or more components (e.g., rule-based components, ML-based components, LLM-based components, etc.) that can process the action data 147a-n (e.g., using API calls generated by LLM candidate list component 140).

[0109] As discussed above, system 100 may include TTS component 156, skill component 154, and LLM agent component 152.

[0110] As discussed above, API provider component 150 may include various components that enable it to perform actions using action data 147a-n. For example, API provider component 150 may include an entity recognition (ER) component that can be configured to process text or lexicalized input to link one or more entity references included in the text or lexicalized input to specific corresponding entities known to system 100. For example, based on text or lexicalized input (e.g., the context of the text or lexicalized input), the ER component may determine that a reference to “Neil Armstrong” refers to an American astronaut. In some embodiments, action data 147a-n may include indications (e.g., slots) of one or more entities included in user input, as determined by one or more of language models 420, 440, 540, in which case the ER component may process them to link one or more entities to specific entities known to system 100.

[0111] In other implementations, the ER component may be configured to process action data 147a-n to determine one or more entities included in the user input and to link one or more determined entities to a specific entity(s) known to the system 100. For example, the ER component may include one or more identifiers. Each identifier may include a Named Entity Recognition (NER) component. The NER component applies grammatical and lexical information (received from storage) associated with a domain (associated with the identifier implementing the NER component) to determine the mention of one or more entities in the text data. In this way, the NER component identifies “slots” (each slot corresponding to one or more specific words in the text data) that may be useful for later processing. The NER component may also label each slot with a type (e.g., noun, place, city, artist name, song title, etc.). The ER component then links the slots in the text data to specific entities known to the system. To perform entity resolution, the ER component may utilize geospecies dictionary information stored in an entity library store. The geospecies dictionary information can be used to match text data (representing a portion of the user input) with text data representing known entities (such as song titles, contact names, etc.). A place name dictionary can be linked to a user (e.g., a specific place name dictionary can be associated with a specific user's music collection), linked to certain fields (e.g., shopping, music, video, etc.), or organized in various other ways.

[0112] For a further example, API provider component 150 may include a search component that can be configured to query a store (e.g., a database, repository, knowledge base, etc.) to obtain information that can be used to generate a response to user input. For example, if action data 147a-n represents a request for information on "Who won the game between [Team 1 Name] and [Team 2 Name]", the search component may query a store (or other source, such as the Internet) to retrieve the information "[Team 1 Name] won the game between [Team 1 Name] and [Team 2 Name]".

[0113] As a further example, API provider component 150 may include the device controller component discussed above herein, which may also be configured to cause the device to perform actions corresponding to action data 147a-n.

[0114] In some implementations, API provider component 150 may include a domain service component that can be configured to interact with one or more services defined by a specific user (such as a developer, expert, etc.) (e.g., to receive information (such as a response or comment) to elicit an action).

[0115] API provider component 150, LLM agent component 152, skill component 154, and / or TTS component 156 may send action response data 158a-n, representing one or more potential responses (e.g., potential responses and / or potential actions) generated by one or more APIs corresponding to action data 147a-n, to action planning execution component 145. For example, in response to an API call by skill component 154 associated with user input to turn on a light, action response data 158a may correspond to potential actions such as “turn on the light”, “turn_on_device(“light”, [device ID]([device ID]))”. As a further example, in response to an API call by skill component 154 associated with user input to order pizza from a specific restaurant, action response data 158b may correspond to potential actions such as “order medium pizza from [restaurant name]”, “order_pizza(“medium”, “pizza”, [restaurant name]([restaurant name])”)”. The action plan execution component 145 can send action response data 158a-n to the candidate list language model 540.

[0116] In some implementations, the candidate list language model 540 can process action response data 158a-n to generate a natural language summary of the action response data (e.g., potential response data 143a-n). In some implementations, the potential response data 143a-n may include an association between action response data 158a (or a summarized representation of action response data 158a) and indications of the API / component that generated the action response data 158a (e.g., component identifier, API description, etc.). In some implementations, the candidate list language model 540 may be configured to filter and / or sort the action response data 158a-n based on its relevance to the current task. In some implementations, the candidate list language model 540 may be configured to filter and / or sort the action response data 158a-n based on the confidence level of the component providing the action response data, wherein the confidence level may indicate the likelihood that the component can respond (e.g., over a period of time), that the component can perform a potential action corresponding to the current task, etc. In some implementations, action response data 158a-n may indicate whether the corresponding component is able to respond (e.g., action response data 158a may include Boolean values ​​such as "yes" or "no" or other similar indications). In some implementations, the candidate list language model 540 may filter and / or sort action response data 158a-n based on information included in cue data 515 (e.g., user input data 127, relevant API data 535, context data 242, personalized context data 167, cue data 415, etc.). For example, potential response data 143a-n may include a subset of action response data 158a-n (or a summary representation of action response data 158a-n), and may also include a representation of the confidence associated with action response data 158a (or a summary representation of action response data 158a). Thus, potential response data 143a-n may also include data representing the confidence level of the relevance of action response data 158a to the current task. In some implementations, the candidate list language model 540 may consider ratings associated with the component that provides action response data 158a, wherein such ratings may be user satisfaction ratings provided by multiple different users of system 100, user satisfaction ratings provided by user 105 associated with user input data 127, system-generated ratings based on the number of past tasks handled by the component, accuracy ratings based on the number of past tasks that the component has correctly handled / provided the expected response, etc.

[0117] LLM candidate list component 140 may send potential response data 143a-n for further processing. If plan generation component 135 determines that more than one task needs to be completed, LLM candidate list component 140 may send potential response data 143a-n to plan generation component 135, which may process the data as described above to maintain and prioritize the task list based on the potential response data 143a-n and select new tasks to be completed. If plan generation component 135 determines that only one task needs to be completed, or if LLM candidate list component 140 determines that no remaining tasks need to be completed, LLM candidate list may send potential response data 143a-n along with potential responses associated with previously completed tasks (e.g., previous action response data) to response arbitration component 160 for processing as discussed above. LLM candidate list component 140 may also send user input data 127, context data 242, personalized context data 167, etc., to plan generation component 135 and / or response arbitration component 160.

[0118] In some embodiments, the LLM orchestrator component 130 may further include a memory storage (not illustrated) that stores various information associated with processing performed during one or more previous processing iterations of the user input data 127 by the LLM orchestrator component 130 (e.g., user input data 127, prompt data 415, context data 242, personalized context data 167, model output data 425, prompt data 435, task processing data 137, relevant API data 535, prompt data 515, action plan data 142, action response data 158a-n, potential response data 143a-n, etc.). Thus, after the LLM candidate list component 140 generates the potential response data 143a-n, the LLM orchestrator component 130 may send the aforementioned data to the memory storage. In some embodiments, the aforementioned data may be sent to the memory storage when generated by the system 100.

[0119] In such implementations, one or more of the prompt generation components discussed herein may be configured to include (e.g., append) one or more portions of data included in memory storage in the data of the corresponding language model (e.g., the generated prompt). For example, during a subsequent processing iteration, the prompt generation component 410 may receive one or more portions of data included in memory storage (which was generated during one or more previous processing iterations performed relative to user input data 127) and include those one or more portions of data in the prompt data 415.

[0120] As discussed above, the candidate list language model 540 can be configured to determine whether additional information is needed to complete the current task (e.g., if there is ambiguity in the user input data 127 or the current task, if the current task is to resolve ambiguity, if API parameters are missing in the user input or other available data, etc.). In this case, the candidate list language model 540 can send data representing a request for such additional information to the response arbitration component 160. In some implementations, action plan data 142 can represent a request for additional information, and the action plan execution component 145 can be configured to send the corresponding action data 147 to the personalization context component 165. For example, for the example provided above regarding ordering pizza, the candidate list language model 540 can determine that, in order to resolve ambiguity relative to the user input data 127 or the current task (e.g., based on whether the current task is to resolve ambiguity or to determine that the current task cannot be completed due to ambiguity), the system 100 must "recognize user pizza preferences," etc. System 100 may send a request to personalization context component 165 to “identify user pizza preferences”, and personalization context component 165 may process the data as described above to return ambiguous personalization context data (e.g., the user’s pizza preference may be determined to be cheese pizza or escargot pizza).

[0121] In some implementations, language models 420, 440, 540, and 220 can be fine-tuned to perform specific tasks. Fine-tuning of language models 420, 440, 540, and 220 can be performed using one or more techniques. One example fine-tuning technique is transfer learning, which involves reusing the weights and architecture of a pre-trained model for new tasks. Pre-trained models can be trained on large, general-purpose datasets, and transfer learning methods allow for efficient and effective adaptation to specific tasks. Another example fine-tuning technique is sequential fine-tuning, where the pre-trained model is fine-tuned sequentially on multiple related tasks. This allows the model to learn more nuanced and complex language patterns across different tasks, resulting in better generalization and performance. Yet another fine-tuning technique is task-specific fine-tuning, where the pre-trained model is fine-tuned on a specific task using a task-specific dataset. Yet another fine-tuning technique is multi-task learning, where the pre-trained model is fine-tuned simultaneously on multiple tasks. This approach enables the model to learn and utilize shared representations across different tasks, resulting in better generalization and performance. Another fine-tuning technique is adapter training, which involves training lightweight modules inserted into a pre-trained model, allowing fine-tuning on a specific task without affecting the performance of the original model on other tasks.

[0122] In some implementations, one or more components of the system 100 discussed herein may be configured to begin processing relative to data once data or a portion of data becomes available to the one or more components. Some components of system 100 are generating components / models that can begin processing relative to a portion of data as soon as the data portion becomes available, rather than waiting until the entire data becomes available. In other words, system 100 may be configured to stream portions of data associated with processing relative to user input to one or more components such that once the data becomes available to the one or more components, the one or more components can begin performing their configured processing relative to that data. For example, if the output of the planned generation language model 420, the task selection language model 440, and / or the candidate list language model 540 indicates that additional information is needed to complete a first task associated with user input, a request for additional information may be sent to the personalization context component 165. Thereafter, the planned generation language model 420, the task selection language model 440, and / or the candidate list language model 540 may continue processing to complete their configured operations. For example, while the personalization context component 165 is processing to determine additional information, system 100 may begin processing relative to a second task associated with user input. Subsequently, the output of the personalization context component 165 can be sent to the response arbitration component 160, so that once the response arbitration component 160 receives the output of the LLM candidate list component 140, it can resolve the ambiguity leading to the request for additional information in order to generate response output data 162. For a further example, if user input data 127 is generated as a natural language representation including the user input, but the processing required to determine the corresponding context signal (e.g., weather data, time data, dialogue history, device information, etc.) has not yet been completed, the planning generation component 135 can begin processing the natural language representation of the user input. Once the corresponding context signal is generated, the planning generation component 135 can begin processing the context signal and can use the processing results relative to the context signal to update downstream components.

[0123] As another example, if the planning generation component 135 determines that more than one task needs to be completed to perform an action in response to user input, and the LLM candidate list component 140 processes the data as described above to generate potential responses relative to a first task among the more than one task, the LLM candidate list component 140 may send the potential responses (along with the user input and a representation of the current task) to the response arbitration component 160 for processing as described above with respect to these potential responses, while the system 100 (e.g., the planning generation component 135 and / or the LLM candidate list component 140) completes processing relative to the remaining tasks among the one or more tasks. Therefore, the response arbitration component 160 may process the data as described above to select among the potential responses associated with the first task, while completing one or more potential responses associated with the remaining tasks. Thus, when the response arbitration component 160 later processes further potential responses associated with further tasks, the response arbitration component 160 may only need to arbitrate among the potential responses associated with the first task that were previously selected by the response arbitration component 160 as responses to the first task.

[0124] As a further example, if API candidate list component 520 determines (e.g., if a confidence value reaches or exceeds a certain threshold) that a particular API or API description should be included in the relevant API data, then API candidate list component 520 may provide the corresponding relevant API data to candidate list suggestion generation component 510, allowing candidate list suggestion generation component 510 to begin processing relative to the relevant API data, while API candidate list component 520 continues to determine one or more further relevant API data. Generally, system 100 is capable of performing this streaming and processing of the data portions discussed herein (e.g., processing relative to user input), and updating downstream components with the processing results of the newly available data portions when data becomes available for processing.

[0125] System 100 can be used as follows Figure 6The various components described operate. These components may reside on the same or different physical devices. Communication between the components may be direct or across network 199. User device 110 may include an audio capture component, such as a microphone or microphone array of user device 110, which captures audio 610 and creates corresponding audio data. Once speech is detected in the audio data representing audio 610, user device 110 can determine whether the speech is directed to user device 110 / system component. In at least some embodiments, this determination may be performed using wake word detection component 620. Wake word detection component 620 may be configured to detect various wake words. In at least some examples, each wake word may correspond to the name of a different digital assistant. An example wake word / digital assistant name is “Alexa”. In another example, system input may be in the form of text data 613, such as as a result of a user typing input in the user interface of user device 110. Other forms of input may include indications such as a user pressing a physical or virtual button on user device 110 or a user making a gesture. User device 110 may also use its camera 918 to capture images and may send image data 621 representing those images to system components. Image data 621 may include raw image data or image data processed by user device 110 before being sent to system components. Different components of the system may use image data 621 in various ways to perform operations, such as determining whether the user is speaking to the system, interpreting user commands, responding to user commands, etc.

[0126] The wake-word detection component 620 of user device 110 can process audio data representing audio 610 to determine whether speech is represented therein. User device 110 can use various techniques to determine whether the audio data includes speech. In some examples, user device 110 can apply voice activity detection (VAD) techniques. Such techniques can determine the presence of speech in audio data based on various quantitative aspects of the audio data, such as the spectral slope between one or more frames of the audio data; the energy levels of the audio data in one or more spectral bands; the signal-to-noise ratio of the audio data in one or more spectral bands; or other quantitative aspects. In other examples, user device 110 can implement a classifier configured to distinguish speech from background noise. The classifier can be implemented using techniques such as linear classifiers, support vector machines, and decision trees. In still some other examples, user device 110 can apply Hidden Markov Model (HMM) or Gaussian Mixture Model (GMM) techniques to compare the audio data with one or more stored acoustic models, which can include models corresponding to speech, noise (e.g., ambient noise or background noise), or silence. Other techniques can be used to determine whether speech is present in audio data.

[0127] Wake word detection is typically performed without language analysis, text analysis, or semantic analysis. Instead, the audio data representing audio 610 is analyzed to determine whether specific characteristics of the audio data match pre-configured acoustic waveforms, audio signatures, or other data corresponding to a wake word.

[0128] Therefore, the wake word detection component 620 can compare audio data with stored data to detect wake words. One method for wake word detection uses a general large-vocabulary continuous speech recognition (LVCSR) system to decode the audio signal, where the wake word search is performed in the resulting lattices or confusion networks. Another method for wake word detection constructs a Hidden Markov Model (HMM) for each wake word and non-wake word speech signal. Non-wake word speech includes other spoken words, background noise, etc. One or more HMMs can be constructed to model the characteristics of non-wake word speech; this is called a filled model. Viterbi decoding is used to search for the optimal path in the decoding graph, and the decoding output is further processed to make a decision about the presence of a wake word. This method can be extended to include discriminative information by incorporating a hybrid DNN-HMM decoding framework. In another example, the wake word detection component 620 can be built directly on a deep neural network (DNN) / recurrent neural network (RNN) structure without involving an HMM. This architecture can estimate the posterior of the wake word using context data by stacking frames within a context window for use with a DNN or by using an RNN. Subsequent posterior threshold adjustment or smoothing is applied to make the decision. Other techniques for wake word detection, such as those known in the art, may also be used.

[0129] Once the wake-word detection component 620 detects a wake-word and / or the input detector detects input, the user device 110 can "wake up" and begin transmitting audio data 611, representing audio 610, to the system component 120. The audio data 611 may include data corresponding to the wake-word; in other embodiments, the portion of the audio corresponding to the wake-word is removed by the user device 110 before the audio data 611 is sent to the system component 120. In the case of touch input detection or gesture-based input detection, the audio data may not include the wake-word.

[0130] In some implementations, system 100 may include more than one system component. System component 120 may respond to different wake words and / or perform different categories of tasks. Each system component may be associated with its own wake word, such that saying a specific wake word causes audio data to be sent to and processed by a specific system. For example, wake word detection component 620 detecting the wake word “Alexa” may cause audio data to be sent to system component a for processing, while wake word detector detecting the wake word “Computer” may cause audio data to be sent to system component b for processing. The system may have separate wake words and systems for different skills / systems (e.g., “DungeonMaster” for game skill / system component c) and / or such skills / systems may be coordinated by one or more skill components 154 of one or more system components 120.

[0131] User device 110 may also include a system-oriented input detector 885. (System components may also include a system-oriented input detector 685, which may operate in a similar manner to system-oriented input detector 885.) System-oriented input detector 885 may be configured to determine whether input to the system (e.g., voice, gesture, etc.) is directed towards the system or not (e.g., towards another user, etc.). System-oriented input detector 885 may work in conjunction with wake word detection component 620. If system-oriented input detector 885 determines that the input is directed towards the system, user device 110 may “wake up” and begin sending captured data for further processing (e.g., processing audio data using speech processing 692 / 892, processing captured image data using image processing component 640 / 840, etc.). If data is being processed, user device 110 may indicate this to the user, for example by activating or changing the color of an illumination output (such as a ring of light-emitting diodes (LEDs), displaying an indicator on the display (such as a light bar on the display), outputting an audio indicator signal (such as a beep), or otherwise informing the user that input data is being processed. If the system orientation input detector 885 determines that the input is not directed towards the system (such as a voice or gesture directed towards another user), the user device 110 may discard the data and take no further action for processing purposes. In this way, the system 100 can prevent the processing of data not directed towards the system, thereby protecting user privacy. However, as an indication to the user, the system may output an audio, visual, or other indication signal when the system orientation input detector 885 determines whether the input is likely directed towards the device. For example, the system may output an orange indication signal when the input is considered, and a green indication signal if a system orientation input is detected. Other such configurations are also possible.

[0132] Upon receipt by system component 120, audio data 611 may be sent to orchestrator component 630 and / or LLM orchestrator component 130. Orchestrator component 630 may include memory and logic enabling it to transmit various data segments and forms of data to various components of the system and perform other operations as described herein. In some embodiments, orchestrator component 630 may optionally be included in system component 120. In embodiments where orchestrator component 630 is not included in system component 120, audio data 611 may be sent directly to LLM orchestrator component 130. Furthermore, in such embodiments, each component in system component 120 may be configured to interact with LLM orchestrator component 130, action plan execution component 145, and / or API provider component 150.

[0133] In some embodiments, system component 120 may include arbitrator component 682, which may be configured to determine whether orchestrator component 630 and / or LLM orchestrator component 130 should be processed relative to audio data 611. In some embodiments, LLM orchestrator component 130 may be selected to process audio data 611 only if the user 105 associated with the audio data 611 (or the user device 110 that captured the audio 610) has previously indicated that LLM orchestrator component 130 may be selected to process user input received from user 105.

[0134] In some implementations, arbitrator component 682 may determine, based on metadata associated with audio data 611, whether orchestrator component 630 and / or LLM orchestrator component 130 should process the audio data 611. For example, arbitrator component 682 may be a classifier configured to process a natural language representation of the audio data 611 (e.g., output by ASR component 650) and classify corresponding user input as to be processed by orchestrator component 630 and / or LLM orchestrator component 130. As a further example, arbitrator component 682 may determine whether the means receiving the audio data 611 is associated with an indicator indicating that the audio data 611 should be processed by orchestrator component 630 and / or LLM orchestrator component 130. As a further example, arbitrator component 682 may determine whether a user receiving audio data 611 (e.g., using data output from user identification component 695) is associated with a user profile that includes an indicator indicating that audio data 611 is to be processed by orchestrator component 630 and / or LLM orchestrator component 130. As another example, arbitrator component 682 may determine whether audio data 611 (or the output of ASR component 650) corresponds to a request indicating that audio data 611 is to be processed by orchestrator component 630 and / or LLM orchestrator component 130 (e.g., a request including "Let's chat" may indicate that audio data 611 is to be processed by LLM orchestrator component 130).

[0135] In some implementations, if the arbiter component 682 is unsure (e.g., whether the confidence score corresponding to whether the arranger component 630 and / or the LLM arranger component 130 should process is below a threshold), the arbiter component 682 may send audio data 611 to both the arranger component 630 and the LLM arranger component 130. In such implementations, the arranger component 630 and / or the LLM arranger component 130 may include further logic for determining a further confidence score during processing, indicating whether the arranger component 630 and / or the LLM arranger component 130 should continue processing, as further discussed herein.

[0136] Arbitrator component 682 may send audio data 611 to ASR component 650. In some embodiments, a component selected to process audio data 611 (e.g., arranger component 630 and / or LLM arranger component 130) may send audio data 611 to ASR component 650. ASR component 650 may transcribe audio data 611 into text data. The text data output by ASR component 650 represents one or more (e.g., in the form of an N-best list) ASR assumptions, which represent the speech presented in audio data 611. ASR component 650 interprets the speech in audio data 611 based on the similarity between audio data 611 and a pre-established language model. For example, ASR component 650 may compare audio data 611 with models of sound (such as acoustic units, such as phonemes, senons, phonemes, etc.) and sound sequences to identify words that match the sound sequences of speech presented in audio data 611. The ASR component 650 sends the resulting text data to the arbitrator component 682, the arranger component 630, and / or the LLM arranger component 130. When text data is sent to the arbitrator component 682, the arbitrator component 682 may send the text data to the component selected to process the audio data 611 (e.g., the arranger component 630 and / or the LLM arranger component 130). The text data sent from the ASR component 650 to the arbitrator component 682, the arranger component 630, and / or the LLM arranger component 130 may include a single highest-scoring ASR hypothesis, or may include an N-best list comprising multiple highest-scoring ASR hypotheses. The N-best list may additionally include a corresponding score associated with each ASR hypothesis represented therein.

[0137] Figure 7 Example components and processing of arbitrator component 682 are shown. Figure 7 As shown, the arbitrator component 682 may include an encoder component 740, a global retrieval component 750, a personalized retrieval component 760, and a sorting component 770. The arbitrator component 682 may communicate with the global index store 720 and the personalized index store 730. The arbitrator component 682 may be configured to perform retrieval-based techniques based on a semantically vectorized representation of user input and historical user input received by system 100 over a period of time (e.g., the past 30 days) to determine whether the orchestrator component 630 or the LLM orchestrator component 130, or both, should be processed relative to the user input.

[0138] User input data 127 may be received at encoder component 740 of arbitrator component 682. Encoder component 740 may process user input data 127 to generate encoded user input data 745, which represents an encoded representation of user input data 127 (e.g., a vectorized representation of user input). Encoder component may send encoded user input data 745 to global retrieval component 750 and personalized retrieval component 760. In some embodiments, encoder component 740 may be trained using techniques associated with deep structured semantic models (DSSM).

[0139] Global retrieval component 750 is configured to identify one or more historical user inputs similar to user input data 127. Global retrieval component 750 queries global index data 725 in global index storage 720 to find one or more historical user inputs that are semantically similar to user input data 127. Global retrieval component 750 may include one or more historical user inputs received from various users over a period of time (e.g., 30 days). In some embodiments, global index data 725 may correspond to an encoded representation of the historical user input. In such embodiments, one or more historical user inputs that are semantically similar to user input data 127 may be determined based on comparing encoded user input data 745 with the encoder representation of historical user inputs (e.g., to determine cosine similarity). Global retrieval component 750 may send global index data 725 to sorting component 770.

[0140] Personalized retrieval component 760 is configured to determine one or more historical user inputs similar to user input data 127, wherein the one or more historical user inputs are associated with a user 105 who provided user input corresponding to user input data 127. Personalized retrieval component 760 queries personalized index data 735 in personalized index storage 730 representing one or more historical user inputs that are semantically similar to user input data 127 and were provided by the same user who provided user input corresponding to user input data 127. Personalized retrieval component 760 may include one or more historical user inputs received from the user corresponding to user input data 127 over a period of time (e.g., 30 days). In some embodiments, personalized index data 735 may correspond to an encoded representation of the historical user inputs. In such embodiments, one or more historical user inputs that are semantically similar to user input data 127 may be determined based on comparing encoded user input data 745 with the encoder representation of historical user inputs (e.g., to determine cosine similarity). Personalized retrieval component 760 may send personalized index data 735 to sorting component 770.

[0141] In some implementations, the global index store 720 and / or the personalized index store 730 may also include metadata associated with historical user input, which may be further included in the global index data 725 and / or the personalized index data 735. For example, the global index store 720 and / or the personalized index store 730 may also include user satisfaction associated with system-generated responses to user input, values ​​indicating the number of times user input was received during that time period, domains (e.g., routines, smart home, shopping, weather, etc.).

[0142] In some implementations, the global retrieval component 750 and / or the personalized retrieval component 760 may use a maximum inner product search solution to retrieve global index data 725 and / or personalized index data 735 that are semantically similar to the encoded user input data 745.

[0143] The sorting component 770 processes global index data 755 and personalized index data 765 to determine whether to send user input data 127 to orchestrator component 630 and / or LLM orchestrator component 130. In some embodiments, the sorting component 770 may make this determination based on metadata included in the global index data 755 and / or personalized index data 765. In some embodiments, the sorting component 770 may be a rule-based component. In other embodiments, the sorting component 770 may be an ML-based component (e.g., decision tree, classifier, LLM, etc.). In embodiments where the sorting component 770 is an LLM, the sorting component 770 may also be configured to determine whether the user input is ambiguous, in which case the sorting component 770 may generate a request for additional information to resolve the ambiguity.

[0144] In some implementations, after determining that the orchestrator component 630 and / or the LLM orchestrator component 130 should be processed relative to the user input data 127, the sorting component 770 may be configured to periodically determine whether the orchestrator component 630 and / or the LLM orchestrator component 130 should continue processing relative to the user input data 127. For example, after specific points in the processing of orchestrator component 630 (e.g., after performing NLU, before determining skill component 154 to process relative to user input data 127, before performing an action in response to user input, etc.) and / or after specific points in the processing of LLM orchestrator component 130 (e.g., after selecting a task to be completed, after receiving action response data from one or more components, after completing a task, before performing an action in response to user input, etc.), orchestrator component 630 and / or LLM orchestrator component 130 may query arbitrator component 682 to determine that orchestrator component 630 and / or LLM orchestrator component 130 should abort processing relative to user input data 127. As discussed above, system 100 may be configured to stream portions of data associated with processing relative to user input to one or more components such that once the data is available to one or more components, the one or more components may begin performing their configured processing relative to the data. Thus, once the data portion associated with user input data 127 (e.g., ASR data, context data, output of user identification component 695) is available, arbitrator component 682 can cause orchestrator component 630 and / or LLM orchestrator component 130 to begin processing relative to the user input. Subsequently, once arbitrator component 682 has sufficient data to perform the processing described above to determine whether orchestrator component 630 and / or LLM orchestrator component 130 should process relative to the user input, arbitrator component 682 can instruct the corresponding component (e.g., orchestrator component 630 and / or LLM orchestrator component 130) to continue / abort processing relative to the user input at one of the logical checkpoints in the processing of orchestrator component 630 and / or LLM orchestrator component 130.

[0145] In some implementations, orchestrator component 630 and / or LLM orchestrator component 130 may periodically confirm whether they should continue processing relative to user input. For example, arbitrator component 682 may also be configured to periodically receive data generated by orchestrator component 630 and / or LLM orchestrator component 130 during processing relative to user input and determine whether orchestrator component 630 and / or LLM orchestrator component 130 should continue processing. Arbitrator component 682 may receive such data at logical checkpoints in the processing of orchestrator component 630 (e.g., after ASR processing is completed, after natural language understanding processing is completed, after selecting a skill component to process relative to user input and before the skill component initiates processing, or before processing any component discussed herein with respect to orchestrator component 630) and / or at logical checkpoints in the processing of LLM orchestrator component 130 (e.g., before processing of LLM candidate list component 140, before starting processing relative to subsequent tasks, or before processing any other component discussed above with respect to LLM orchestrator component 130). Arbitrator component 682 may be configured to process, as described above, the received data to compare with data associated with the processing of previous user input. This allows arbitrator component 682 to make a more informed decision about which component should process the user input (e.g., based on additional data determined during the processing of orchestrator component 630 and / or LLM orchestrator component 130). In some implementations, data may be received at another component of system 100 that is configured to process data as described herein.

[0146] In some implementations, after sending data to the arbitrator component 682, the orchestrator component 630 and / or the LLM orchestrator component 130 may temporarily suspend processing relative to the user input until they receive data from the arbitrator component 682 confirming that they wish to continue processing relative to the user input. As discussed above, in some implementations, the LLM orchestrator component 130 may send data to the arbitrator component 682 prior to processing by the LLM candidate list component 140. In some implementations, the LLM orchestrator component 130 may further include a component configured to process task processing data (e.g., task processing data 137) output by the planning generation component 135 to determine whether the completion of the current task will result in a real-world action (e.g., a change in device state, such as turning on a light, changing a television channel, changing the temperature value on a thermostat, locking a door, etc.). If the component determines that the completion of the current task will result in a real-world action, the LLM orchestrator component 130 may temporarily suspend its processing prior to processing by the LLM candidate list component 140. If the component determines that completion of the current task will not result in a real-world action, the LLM orchestrator component 130 may begin processing of the LLM candidate list component 140 instead of temporarily pausing processing. In some implementations, the orchestrator component 630 may include components with a similar configuration.

[0147] Skill system component 125 can communicate with skill component 154 within system component 120, directly with orchestrator component 630 and / or action plan execution component 145, or with other components. Skill system component 125 can be configured to perform one or more actions. The ability to perform such actions is sometimes referred to as a "skill." That is, a skill enables skill system component 125 to perform a specific function in order to provide data or perform another action requested by the user. For example, a weather service skill enables skill system component 125 to provide weather information to system component 120, a car service skill enables skill system component 125 to book a trip relative to a taxi or ride-sharing service, and a pizza ordering skill enables skill system component 125 to order pizza relative to a restaurant's online ordering system, etc. Additional types of skills include home automation skills (e.g., skills that enable users to control home appliances such as lights, door locks, cameras, thermostats, etc.), entertainment device skills (e.g., skills that enable users to control entertainment devices such as smart TVs), video skills, newsletter skills, and custom skills not associated with any pre-configured type of skill.

[0148] System component 120 may be configured with a skill component 154 specifically for interacting with skill system component 125. Unless otherwise explicitly stated, references to skill, skill device, or skill component may include skill component 154 operated by system component 120 and / or skill operated by skill system component 125. Furthermore, the functionality or skill described herein as a skill may be referred to using many different terms (such as action, robot, app, etc.). Skill component 154 and / or skill system component 125 may return output data to orchestrator component 630.

[0149] Dialogue processing is a field of computer science that involves communication between computing systems and humans via text, audio, and / or other forms of interaction. While some forms of dialogue processing simply involve generating responses based on only the user's latest input (i.e., a single-turn dialogue), more complex dialogue processing involves identifying one or more goals expressed by the user in a multi-turn dialogue and optionally taking action accordingly, such as making a restaurant reservation and / or booking a flight. These multi-turn "goal-oriented" dialogue systems typically require identifying, retaining, and using information collected during more than one input in back-and-forth or "multi-turn" interactions with the user.

[0150] The system components include a TTS component 156. The TTS component 156 can generate audio data from text data (e.g., synthesized speech) using one or more different methods. The text data input to the TTS component 156 may come from the skill component 154, the orchestrator component 630, or another component of the system. In a synthesis method known as unit selection, the TTS component 156 matches the text data against a database of recorded speech. The TTS component 156 selects matching units of the recorded speech and concatenates these units together to form audio data. In another synthesis method known as parametric synthesis, the TTS component 156 modifies parameters (such as frequency, volume, and noise) to create audio data that includes an artificial speech waveform. Parametric synthesis uses a computerized speech generator, sometimes called a vocoder.

[0151] User device 110 may include still images and / or video capture components, such as one or more cameras, that capture one or more images. User device 110 may include circuitry for digitizing images and / or video for transmission as image data to system component 120. User device 110 may also include circuitry for voice command-based control of the camera, allowing user 105 to request the capture of image or video data. User device 110 may process commands locally or send audio data 611 representing the command to system component 120 for processing, after which system component 120 may return output data that causes user device 110 to activate its camera.

[0152] System component 120 may include user identification component 695, which uses various data to identify one or more users. However, this disclosure is not limited thereto, and without departing from this disclosure, user device 110 may include user identification component 895 as a substitute for and / or supplement to user identification component 695 of system component 120. User identification component 895 operates similarly to user identification component 695.

[0153] User identification component 695 may treat audio data 611 and / or text data output by ASR component 650 as input. User identification component 695 may perform user identification by comparing audio characteristics in audio data 611 with stored audio characteristics of the user. User identification component 695 may also perform user identification by comparing biometric data (e.g., fingerprint data, iris data, etc.) received by a system associated with the current user input with stored biometric data of the user, assuming user permission and prior authorization. User identification component 695 may also perform user identification by comparing image data (e.g., including a representation of at least one feature of the user) received by a system associated with the current user input with stored image data (which includes representations of features of different users). User identification component 695 may perform additional user identification processes, including those known in the art.

[0154] User identification component 695 determines a score indicating whether user input originates from a specific user. For example, a first score may indicate the probability that user input originated from a first user, a second score may indicate the probability that user input originated from a second user, and so on. User identification component 695 also determines an overall confidence level regarding the accuracy of the user identification operation.

[0155] The output of user identification component 695 may include a single user identifier corresponding to the most likely user who initiated the user input. Alternatively, the output of user identification component 695 may include an N-best list of user identifiers, where the corresponding score indicates the likelihood that the corresponding user initiated the user input. The output of user identification component 695 can be used to inform the processing of arbitrator component 682, orchestrator component 630, and / or LLM orchestrator component 130, as well as processing performed by other components of the system.

[0156] System component 120 / user device 110 may include an presence detection component that uses various data to determine the presence and / or location of one or more users.

[0157] System 100 (on user device 110, system components, or a combination thereof) may include profile storage for storing various information related to individual users, user groups, devices, etc., who interact with the system. As used herein, a “profile” refers to a set of data associated with a user, user group, device, etc. Profile data may include user, device, etc., specific preferences; device input and output capabilities; internet connectivity information; user bibliographic information; subscription information, and other information.

[0158] Profile storage 670 may include one or more user profiles, each associated with a different user identifier / user profile identifier. Each user profile may include various user identification data. Each user profile may also include data corresponding to the user's preferences. Each user profile may also include user preferences and / or one or more device identifiers representing one or more devices of the user. For example, a user account may include one or more IP addresses, MAC addresses, and / or device identifiers (such as serial numbers) of each attached electronic device associated with the identified user account. When a user logs in to an application installed on user device 110, the user profile (associated with the presented login information) may be updated to include information about user device 110, such as an indication that the device is currently in use. Each user profile may include identifiers of skills that the user has enabled. When a user enables a skill, the user grants permission to system components to allow the skill to be executed relative to the user's natural language user input. If the user has not enabled a skill, the system components may not invoke the skill to execute relative to the user's natural language user input.

[0159] Profile storage 670 may include one or more group profiles. Each group profile may be associated with a different group identifier. A group profile may be specific to a group of users. That is, a group profile may be associated with two or more individual user profiles. For example, a group profile may be a family profile associated with user profiles associated with multiple users in a single household. A group profile may include preferences shared by all user profiles associated with it. Each user profile associated with a group profile may additionally include preferences specific to the user associated with it. That is, each user profile may include preferences unique to one or more other user profiles associated with the same group profile. User profiles may be independent profiles or may be associated with group profiles.

[0160] Profile storage 670 may include one or more device profiles. Each device profile may be associated with a different device identifier. Each device profile may include various device identification information. Each device profile may also include one or more user identifiers representing one or more users associated with the device. For example, a profile for a home device may include user identifiers for home users.

[0161] although Figure 6 The components may be exemplified as part of system component 120, user device 110 or other components, but without departing from this disclosure, these components may be arranged in other devices (e.g., if exemplified in system component 120, they may be arranged in user device 110, or vice versa, or together in other devices). Figure 8 User device 110 with this configuration is illustrated.

[0162] In at least some embodiments, the system component may receive audio data 611 from the user device 110 to identify speech corresponding to spoken input in the received audio data 611, and perform functions in response to the identified speech. In at least some embodiments, these functions involve sending instructions (e.g., commands) from the system component to the user device 110 (and / or other device 110) to cause the user device 110 to perform actions, such as outputting an audible response to spoken input via a speaker, and / or controlling an assistive device by sending control commands to assistive devices in the environment.

[0163] Therefore, when user device 110 is able to communicate with system components via network 199, some or all of the functions that system components can perform can be executed by sending one or more instructions to user device 110 over network 199, which in turn can process the instructions and perform one or more corresponding actions. For example, system components can use remote instructions included in response data (e.g., remote response) to instruct user device 110 to output an audible response to a user's question via the user device 110's (or otherwise associated with) speakers (e.g., using TTS processing performed by TTS component 880 on the device), output content (e.g., music) via the user device 110's (or otherwise associated with) speakers, display content on the user device 110's (or otherwise associated with) display, and / or send instructions to auxiliary devices (e.g., instructions to turn on a smart light). It should be understood that system components may be configured to provide functions other than those discussed herein, such as, but not limited to, providing step-by-step instructions for navigation from the starting point to the destination, conducting e-commerce transactions on behalf of user 105 as part of a shopping function, establishing communication sessions (e.g., video calls) between user 105 and another user, etc.

[0164] Such as about Figure 6The user device 110 may include a wake word detection component 620 configured to compare audio data 611 with a storage model for detecting a wake word (e.g., "Alexa") that instructs the user device 110 to process the audio data 611 (e.g., orchestrator component 835 and / or LLM orchestrator 830, which are similarly configured to orchestrator component 630 and / or LLM orchestrator component 130, respectively). In at least some embodiments, a mixing selector 824 of the user device 110 may send the audio data 611 to the wake word detection component 620. If the wake word detection component 620 detects a wake word in the audio data 611, it may send an indication of such detection to the mixing selector 824. In response to receiving this indication, the mixing selector 824 may send the audio data 611 to the system component and / or ASR component 850. The wake-word detection component 620 can also send an indication to the mixing selector 824 that no wake-word has been detected. In response to receiving such an indication, the mixing selector 824 can avoid sending audio data 611 to system components and can prevent the ASR component 850 from further processing the audio data 611. In this case, the audio data 611 can be discarded.

[0165] User device 110 may use on-device speech processing components (such as ASR component 850) to perform its own speech processing, in a manner similar to that discussed herein with respect to system components ASR component 650. ASR component 850 may operate in a similar manner to ASR component 650. User device 110 may also internally include other components or otherwise have access to other components, such as one or more skill components 890 (which may operate similarly to skill component 154), arbitrator component 882 (which is configured to process in a manner similar to that discussed herein with respect to arbitrator component 682), action plan execution component 877 (which is configured to process in a manner similar to that discussed herein with respect to action plan execution component 145), personalization context component 878 (which is configured to process in a manner similar to that discussed herein with respect to personalization context component 165), API provider component 879 (which is configured to process in a manner similar to that discussed herein with respect to API provider component 150), and LLM agent component 881 (which is configured to process in a manner similar to that discussed herein with respect to LLM agent component 152), user identification component 895, and so on. (It is configured to process in a manner similar to that discussed herein with respect to user identification component 695 of the system components), profile storage 870 (which is configured to store profile data similar to that discussed herein with respect to profile storage 670 of the system components), or other components. In at least some embodiments, profile storage 870 may store only profile data of users or user groups specifically associated with user device 110. Similar to that described above with respect to skill component 154, skill component 890 may communicate with skill system component 125. User device 110 may also have its own TTS component 880, which may operate similarly to TTS component 156.

[0166] In at least some embodiments, the on-device language processing component may not have the same capabilities as the language processing component of the system component. For example, the on-device language processing component may be configured to handle only a subset of natural language user input that the system component can handle. For example, such a subset of natural language user input may correspond to local type natural language user input (such as input that controls devices or components associated with a user's home). In such cases, the on-device language processing component may be able to interpret and respond to local type natural language user input, for example, faster than processing involving the system component. If the user device 110 attempts to process natural language user input that is not necessarily best suited for the on-device language processing component, the language processing result determined by the user device 110 may indicate a low confidence level or other metric that indicates the processing performed by the user device 110 may be less accurate than the processing performed by the system component.

[0167] The hybrid selector 824 of user device 110 may include a hybrid agent (HP) 826 configured to broker traffic to / from system components. For example, HP 826 may be configured to send messages to / from a hybrid execution controller (HEC) 827 of the hybrid selector 824. For example, HP 826 may be used to send command / instruction data received from system components to HEC 827. HP 826 may also be configured to allow audio data 611 to be delivered to system components while also receiving (e.g., intercepting) the audio data 611 and sending the audio data 611 to HEC 827.

[0168] In at least some embodiments, the hybrid selector 824 may also include a Local Request Organizer (LRO) 828 configured to notify the ASR component 850 of the availability of new audio data 611 representing user speech and, in other ways, initiate local language processing operations when the new audio data 611 becomes available. Generally, the hybrid selector 824 may control the execution of local language processing, such as by sending “execute” and “termination” events / instructions. The “execute” event may instruct the component to continue any suspended execution (e.g., by instructing the component to execute according to a previously determined intent to determine the instruction). Simultaneously, the “termination” event may instruct the component to terminate further execution, such as when the user device 110 receives instruction data from the system component and selects to use the remotely determined instruction data.

[0169] Therefore, when audio data 611 is received, HP 826 can allow audio data 611 to be passed to the system component, and HP 826 can also input audio data 611 to the ASR component 850 on the device by routing audio data 611 through HEC 827 of the mixing selector 824, thereby notifying the ASR component 850 of audio data 611 by LRO 828. At this time, the mixing selector 824 can wait for response data from either or both of the system component or the local language processing component. However, this disclosure is not limited thereto, and without departing from this disclosure, in some examples, the mixing selector 824 can send audio data 611 only to the local ASR component 850. For example, user device 110 can process audio data 611 locally without sending audio data 611 to the system component.

[0170] The local ASR component 850 is configured to receive audio data 611 from the mixing selector 824 and identify speech in the audio data 611. User device 110 and / or system components may associate a unique identifier with each natural language user input. User device 110 may include a unique identifier when sending audio data 611 to the system component, and response data from the system component may include a unique identifier to identify the natural language user input corresponding to the response data.

[0171] In at least some embodiments, user device 110 may include or be configured to use one or more skill components 890, which may operate similarly to skill components 154 implemented by system components. Skill components 890 may correspond to one or more domains used to determine how to act on verbal input in a particular manner, such as by outputting instructions that correspond to a determined intent and can be processed to achieve the desired operation. Skill components 890 installed on user device 110 may include, but are not limited to, smart home skill components (or smart home domains) and / or device control skill components (or device control domains) executed in response to verbal input corresponding to an intent of a second device in the control environment, music skill components (or music domains) executed in response to verbal input corresponding to an intent to play music, navigation skill components (or navigation domains) executed in response to verbal input corresponding to an intent to obtain direction, shopping skill components (or shopping domains) executed in response to verbal input corresponding to an intent to purchase items from an electronic marketplace, and / or similar components.

[0172] Alternatively or concurrently, user device 110 may communicate with one or more skill system components 125. For example, skill system component 125 may be located in a remote environment (e.g., a separate location), allowing user device 110 to communicate with skill system component 125 solely via network 199. However, this disclosure is not limited thereto. For example, in at least some embodiments, skill system component 125 may be configured in a local environment (e.g., a home server and / or similar environment), allowing user device 110 to communicate with skill system component 125 via a private network (such as a local area network (LAN)).

[0173] As used in this article, "skill" may refer to skill component 890, skill system component 125, or a combination of skill component 890 and the corresponding skill system component 125.

[0174] Regarding Figure 6 Similarly, local user device 110 can be configured to recognize multiple different wake words and / or perform different categories of tasks based on the wake words. These different wake words can invoke different processing components of local user device 110. Figure 8 (Not illustrated in the text). For example, the wake word detection component 620 detecting the wake word "Alexa" may cause audio data to be sent to some language processing component 892 / skill component 890 for processing, while the wake word detector detecting the wake word "computer" may cause audio data to be sent to a different language processing component 892 / skill component 890 for processing.

[0175] Various machine learning techniques can be used to train and operate models to perform the various steps described in this paper, such as user identification, sentiment detection, image processing, dialogue management, etc. Models can be trained and operated using a variety of machine learning techniques. Such techniques can include, for example, neural networks (such as deep neural networks and / or recurrent neural networks), inference engines, trained classifiers, etc. Examples of trained classifiers include Support Vector Machines (SVMs), neural networks, decision trees, AdaBoost (short for "adaptive boosting") combined with decision trees, and random forests. Taking SVM as an example, SVM is a supervised learning model with associated learning algorithms that analyze data and identify patterns in the data, and is often used for classification and regression analysis. Considering a set of training examples, each labeled as belonging to one of two categories, the SVM training algorithm builds a model that assigns new examples to one category or the other, thus making it a non-probabilistic binary linear classifier. More complex SVM models can be built using training sets that identify more than two categories, where the SVM determines which category is most similar to the input data. Mappable SVM models separate examples of individual classes by clear margins. New examples are then mapped into the same space, and their class is predicted based on which side of the margin they fall on. The classifier provides a "score" indicating which class the data best matches. The score provides an indication of how well the data matches the class.

[0176] In order to apply machine learning techniques, the machine learning process itself needs to be trained. Training machine learning components (such as one of the first or second models in this case) requires establishing "ground truth" for the training examples. In machine learning, the term "ground truth" refers to the accuracy of classification on the training set using supervised learning techniques. Various techniques can be used to train models, including backpropagation, statistical learning, supervised learning, semi-supervised learning, randomized learning, or other known techniques.

[0177] Figure 9 This is a block diagram conceptually illustrating a user device 110 that can be used with the system. Figure 10This is a block diagram conceptually illustrating example components of a remote device, such as a natural language command processing system component that can assist with ASR processing, NLU processing, etc., and a skill system component 125. The system (120 / 125) may include one or more servers. As used herein, "server" may refer to a conventional server as understood in a server / client computing architecture, but may also refer to a variety of different computing components that can assist the operations discussed herein. For example, a server may include one or more physical computing components (such as rack servers) that are physically and / or connected via a network to other devices / components and are capable of performing computing operations. A server may also include one or more virtual machines that simulate a computer system and run on one device or across multiple devices. A server may also include other combinations of hardware, software, firmware, etc., to perform the operations discussed herein. A server may be configured to operate using one or more of the following computing technologies: client-server model, computer office model, grid computing, fog computing, mainframe technology, utility computing, peer-to-peer model, sandboxing, or others.

[0178] While user device 110 can operate locally with the user (e.g., within the same environment, so the device can receive input and play output for the user), the server / system component can be located remotely from user device 110, as its operation may not require proximity to the user. The server / system component can be located in a completely different location from user device 110 (e.g., as part of a cloud computing system, etc.), or it can be located in the same environment as user device 110 but physically separate from it (e.g., a home server or similar device residing in the user's home or office but possibly in a closet, basement, attic, etc.). System component 120 can also be a version of user device 110 that includes processing power different (e.g., larger) than other user devices 110 in the home / office. One benefit of having the server / system component located in the user's home / office is that the data used to process commands / return responses can be stored at the user's home, thus reducing potential privacy concerns.

[0179] The overall system 100 disclosed herein may include multiple system components (120 / 125), such as one or more natural language processing system components 120 for performing ASR processing, one or more natural language processing system components 120 for performing NLU processing, one or more skill system components 125, etc. In operation, each of these systems may include computer-readable and computer-executable instructions residing on the corresponding device (120 / 125), as will be further discussed below.

[0180] Each of these devices (110 / 120 / 125) may include one or more controllers / processors (904 / 1004), each of which may include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (906 / 1006) for storing the data and instructions of the respective device. The memory (906 / 1006) may individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (110 / 120 / 125) may also include a data storage component (908 / 1008) for storing data and controller / processor executable instructions. Each data storage component (908 / 1008) may individually include one or more non-volatile memory types, such as magnetic storage, optical storage, solid-state storage, etc. Each device (110 / 120 / 125) can also be connected to removable or external non-volatile memory and / or storage (such as removable memory cards, memory key drives, network storage, etc.) via the corresponding input / output device interface (902 / 1002).

[0181] The computer instructions for operating each device (110 / 120 / 125) and its various components can be executed by the controller / processor (904 / 1004) of the corresponding device during operation, using memory (906 / 1006) as temporary "working" storage. The computer instructions for the device can be stored in a non-transitory manner in non-volatile memory (906 / 1006), storage (908 / 1008), or an external device. Alternatively, as a supplement or replacement to software, some or all of the executable instructions can be embedded in the hardware or firmware of the corresponding device.

[0182] Each device (110 / 120 / 125) includes an input / output device interface (902 / 1002). Various components can be connected via the input / output device interface (902 / 1002), as will be discussed further below. Additionally, each device (110 / 120 / 125) may include an address / data bus (924 / 1024) for transferring data between components of the respective device. As a supplement (or alternative) to connecting to other components across the bus (924 / 1024), each component within a device (110 / 120 / 125) may also be directly connected to other components.

[0183] refer to Figure 9User device 110 may include an input / output device interface 902 that connects to various components, such as audio output components, such as speaker 912, wired or wireless headphones (not shown), or other components capable of outputting audio. User device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 920 or a microphone array, wired or wireless headphones (not shown), etc. If a microphone array is included, the approximate distance to the origin of the sound can be determined by acoustic localization based on the time and amplitude differences between the sounds captured by the different microphones in the array. User device 110 may additionally include a display 916 for displaying content. User device 110 may also include a camera 918.

[0184] Via antenna 922, input / output device interface 902 can connect to one or more networks 199 via wireless local area network (WLAN) radio (such as Wi-Fi), Bluetooth, and / or wireless network radio, such as radios capable of communicating with wireless communication networks such as LTE networks, WiMAX networks, 3G networks, 4G networks, and 5G networks. Wired connections, such as Ethernet, are also supported. Through network 199, the system can be distributed in a networked environment. The I / O device interfaces (902 / 1002) may also include communication components that allow data exchange between devices (such as different physical servers or other components in a server set).

[0185] Components of device 110, natural language command processing system component, or skill system component 125 may include their own dedicated processor, memory, and / or storage. Alternatively, one or more components of device 110, natural language command processing system component, or skill system component 125 may utilize the I / O interface (902 / 1002), processor (904 / 1004), memory (906 / 1006), and / or storage (908 / 1008) of device 110, natural language command processing system component, or skill system component 125, respectively. Thus, ASR component 650 may have its own I / O interface, processor, memory, and / or storage; and so on for the various components discussed herein.

[0186] As described above, multiple devices can be employed in a single system. In such a multi-device system, each of the devices may include different components for performing different aspects of system processing. Multiple devices may include overlapping components. As described herein, the components of user device 110, natural language command processing system components, and skill system components 125 are exemplary and may be positioned as independent devices or may be included wholly or partially as components of a larger device or system. As will be understood, numerous components may be present on system components and / or user device 110. For example, language processing 692 / 892 (which may include ASR 650 / 850), language output 693 / 893 (which may include NLG 679 / 879 and TTS 680 / 880), etc., for example, Figure 6 and Figure 8 As illustrated. Unless otherwise expressly stated, the system version of such a component may operate similarly to the device version of such a component, and therefore the description of one version (e.g., the system version or the local version) applies to the description of another version (e.g., the local version or the system version), and vice versa.

[0187] like Figure 11 As illustrated, multiple devices (110a-110n, 120, 125) may contain components of the system, and the devices may be connected to network 199. Network 199 may include a local or private network, or may include a wide area network (such as the Internet). Devices may connect to network 199 via wired or wireless connections. For example, voice detection device 110a, smartphone 110b, smartwatch 110c, tablet computer 110d, vehicle 110e, voice detection device with display 110f, monitor / smart TV 110g, washer / dryer 110h, refrigerator 110i, microwave oven 110j, autonomous mobile device 110k (e.g., robot), etc., may connect to network 199 via wireless service providers, via Wi-Fi or cellular network connections, etc. Other devices are included as supporting devices for network connectivity, such as natural language command processing system component 120, skill system component 125, and / or other devices. Supporting devices may connect to network 199 via wired or wireless connections. The networked device may use one or more built-in or connected microphones or other audio capture devices to capture audio, wherein processing is performed by the ASR component, NLU component or other component (such as the ASR component 650 of the Natural Language Command Processing System Component 120) of the same device or another device connected via network 199.

[0188] The concepts disclosed herein can be applied to many different devices and computer systems, including, for example, general-purpose computing systems, speech processing systems, and distributed computing environments.

[0189] The following terms may also be used to understand this material.

[0190] 1. A computer-implemented method, the method comprising: receiving first audio data corresponding to a first spoken input; performing automatic speech recognition (ASR) processing using the first audio data to generate a first transcription of the first spoken input; receiving first latent response data from a first component configured to perform a first action associated with the first spoken input; receiving second latent response data from a second component configured to perform a second action associated with the first spoken input; receiving third latent response data from a third component configured to perform the second action; generating a first prompt including the first latent response data, the second latent response data, the third latent response data, and the first transcription, wherein the first prompt is a first request for a language model to determine a first output in response to the first spoken input; processing the first prompt using the language model to generate first model output data, the first model output data indicating that: the first latent response data is in response to the first spoken input, and the second latent response data is in response to the first spoken input; causing the first component to perform the first action based at least in part on the first model output data; and causing the second component to perform the second action based at least in part on the first model output data.

[0191] 2. The computer-implemented method as described in Clause 1, the method further comprising: receiving second audio data corresponding to a second spoken input; performing ASR processing using the second audio data to generate a second transcription of the second spoken input; receiving fourth potential response data from a fourth component, the fourth potential response data being associated with the second spoken input; receiving fifth potential response data from a fifth component, the fifth potential response data being associated with the second spoken input; generating a second cue including the fourth potential response data, the fifth potential response data, and the second transcription, wherein the second cue is a second request for the language model to determine a second output in response to the first spoken input; processing the second cue using the language model to generate second model output data, the second model output data indicating that the fourth potential response data is in response to the second spoken input, and the fifth potential response data is in response to the second spoken input, wherein the second model output data includes second output data representing a summary of the fourth potential response data and the fifth potential response data; and causing the second output data to be presented in response to the second spoken input.

[0192] 3. A computer-implemented method as described in Clause 1 or 2, wherein the first latent response data is associated with a first task of the first verbal input, the second latent response data and the third latent response data are associated with a second task of the first verbal input, and the method further comprises: receiving, before receiving the second latent response data and the third latent response data, fourth latent response data from a fourth component configured to perform a fourth action associated with the first verbal input, wherein the fourth latent response data is associated with the first task; generating a second prompt including the first latent response data, the fourth latent response data and the first transcription, wherein the second prompt is a second request for the language model to determine a second output in response to the first task of the first verbal input; and processing the second prompt using the language model to generate second model output data, the second model output data indicating that the first latent response data is in response to the first task of the first verbal input, rather than the fourth latent response data.

[0193] 4. A computer-implemented method as described in clauses 1, 2, or 3, the method further comprising: generating a second prompt including first potential response data, second potential response data, third potential response data, and the first transcription before generating the first prompt, wherein the second prompt is a second request for the language model to determine a second output in response to the first spoken input; processing the second prompt using the language model to generate second model output data, the second model output data including: an indication of ambiguity associated with the first spoken input, and a third request for first data associated with the ambiguity; sending the second model output data to a fourth component based at least in part on the second model output data; and receiving first context data associated with the ambiguity from the fourth component, wherein: the first prompt further includes the first context data, and the first model output data is not associated with the ambiguity.

[0194] 5. A computer-implemented method, the method comprising: receiving first input data; receiving first potential response data from a first component configured to perform a first action associated with the first input data; receiving second potential response data corresponding to a second component configured to perform a second action associated with the first input data; determining a first prompt including the first potential response data, the second potential response data, and the first input data, wherein the first prompt is a first request for a language model to determine a first output in response to the first input data; processing the first prompt using the language model to generate first model output data, the first model output data indicating that the first action is in response to the first input data; and causing the first component to perform the first action.

[0195] 6. A computer-implemented method as described in Clause 5, the method further comprising: receiving second input data; receiving third potential response data from a third component, the third potential response data being associated with the second input data; receiving fourth potential response data from a fourth component, the fourth potential response data being associated with the second input data; generating a second prompt including the third potential response data, the fourth potential response data, and the second input data, wherein the second prompt is a second request for the language model to determine a second output in response to the second input data; processing the second prompt using the language model to generate second model output data, the second model output data indicating that the third potential response data is in response to the second input data, and the fourth potential response data is in response to the second input data, wherein the second model output data includes second output data representing a summary of the third potential response data and the fourth potential response data; and causing the second output data to be presented in response to the second input data.

[0196] 7. A computer-implemented method as described in Clause 5 or 6, wherein the first potential response data is associated with a first task of the first input data, the second potential response data is associated with a second task of the first input data, and the method further comprises: receiving, before receiving the second potential response data, third potential response data corresponding to a third component configured to perform a third action associated with the first input data, wherein the third potential response data is associated with the first task; determining a second prompt including the first potential response data and the third potential response data, wherein the second prompt is a second request for the language model to determine a second output in response to the first task of the first input data; and processing the second prompt using the language model to generate second model output data, the second model output data indicating that the first potential response data responds to the first task of the first input data, rather than the third potential response data.

[0197] 8. A computer-implemented method as described in clauses 5, 6, or 7, the method further comprising: generating a second prompt including first potential response data, second potential response data, and first input data before generating the first prompt, wherein the second prompt is a second request for the language model to determine a second output in response to the first input data; processing the second prompt using the language model to generate second model output data, the second model output data including: an indication of ambiguity associated with the first input data, and a third request for first data associated with the ambiguity; sending the second model output data to a third component based at least in part on the second model output data; and receiving first context data associated with the ambiguity from the third component, wherein: the first prompt further includes the first context data, and the first model output data is not associated with the ambiguity.

[0198] 9. A computer-implemented method as described in clauses 5, 6, 7, or 8, wherein the first action corresponds to a first means of performing a third action, and the method further comprises: determining first output data corresponding to the first action, the first output data indicating the first means of performing the first action; and causing the first output data to be presented after causing the first component to perform the first action.

[0199] 10. A computer-implemented method as described in clauses 5, 6, 7, 8, or 9, wherein the first action corresponds to a first means of performing a third action, and the method further comprises: receiving third potential response data corresponding to a third component configured to perform a fourth action associated with the first input data, wherein: the fourth action corresponds to a second means of performing a fifth action, the first prompt further includes the third potential response data, and the first model output data further indicates that the fourth action is in response to the first input data; and causing the third component to perform the fourth action, wherein: causing the third component to perform the fourth action includes causing the second means of performing the fifth action, and causing the first component to perform the first action includes causing the first means of performing the third action.

[0200] 11. A computer-implemented method as described in clauses 5, 6, 7, 8, 9, or 10, the method further comprising: generating a second prompt including first potential response data, second potential response data, and first input data before generating the first prompt, wherein the second prompt is a second request for the language model to determine a second output in response to the first input data; processing the second prompt using the language model to generate second model output data, the second model output data including: an indication of an ambiguity associated with the first input data, and a third request for contextual data associated with the ambiguity; determining second output data corresponding to the second model output data based at least in part on the second model output data; causing the second output data to be presented; and receiving natural language data corresponding to user input associated with the ambiguity, wherein: the first prompt also includes the user input, and the first model output data is not associated with the ambiguity.

[0201] 12. A computer-implemented method as described in Clauses 5, 6, 7, 8, 9, 10 or 11, wherein the first component is a skill and the second component is an agent based on a large language model (LLM).

[0202] 13. A computing system comprising: at least one processor; and at least one memory including instructions that, when executed by the at least one processor, cause the computing system to: receive first input data; receive first potential response data from a first component configured to perform a first action associated with the first input data; receive second potential response data corresponding to a second component configured to perform a second action associated with the first input data; determine a first prompt including the first potential response data, the second potential response data, and the first input data, wherein the first prompt is a first request for a language model to determine a first output in response to the first input data; process the first prompt using the language model to generate first model output data, the first model output data indicating that the first action is in response to the first input data; and cause the first component to perform the first action.

[0203] 14. The computing system of claim 13, wherein the at least one memory further comprises instructions, which, when executed by the at least one processor, further cause the computing system to: receive second input data; receive third potential response data from a third component, the third potential response data being associated with the second input data; receive fourth potential response data from a fourth component, the fourth potential response data being associated with the second input data; generate a second prompt including the third potential response data, the fourth potential response data, and the second input data, wherein the second prompt is a second request for the language model to determine a second output in response to the second input data; process the second prompt using the language model to generate second model output data, the second model output data indicating that the third potential response data is in response to the second input data, and the fourth potential response data is in response to the second input data, wherein the second model output data includes second output data representing a summary of the third potential response data and the fourth potential response data; and cause the second output data to be presented in response to the second input data.

[0204] 15. A computing system as described in Clause 13 or 14, wherein the first potential response data is associated with a first task of the first input data, the second potential response data is associated with a second task of the first input data, and wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing system to: receive, before receiving the second potential response data, a third potential response data corresponding to a third component configured to perform a third action associated with the first input data, wherein the third potential response data is associated with the first task; determine a second prompt including the first potential response data and the third potential response data, wherein the second prompt is a second request for the language model to determine a second output in response to the first task of the first input data; and process the second prompt using the language model to generate second model output data, the second model output data indicating that the first potential response data responds to the first task of the first input data, rather than the third potential response data.

[0205] 16. A computing system as described in Clauses 13, 14, or 15, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing system to: generate a second prompt comprising the first potential response data, the second potential response data, and the first input data before generating the first prompt, wherein the second prompt is a second request for the language model to determine a second output in response to the first input data; process the second prompt using the language model to generate second model output data, the second model output data comprising: an indication of an ambiguity associated with the first input data, and a third request for first data associated with the ambiguity; send the second model output data to a third component based at least in part on the second model output data; and receive first context data associated with the ambiguity from the third component, wherein: the first prompt further includes the first context data, and the first model output data is not associated with the ambiguity.

[0206] 17. A computing system as described in Clauses 13, 14, 15 or 16, wherein the first action corresponds to a first means of performing a third action, and wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing system to: determine first output data corresponding to the first action, the first output data indicating the first means of performing the first action; and, after causing the first component to perform the first action, cause the first output data to be presented.

[0207] 18. A computing system as described in clauses 13, 14, 15, 16, or 17, wherein the first action corresponds to a first means of performing a third action, and wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing system to: receive third potential response data corresponding to a third component configured to perform a fourth action associated with the first input data, wherein: the fourth action corresponds to a second means of performing a fifth action, the first prompt further includes the third potential response data, and the first model output data further indicates that the fourth action is in response to the first input data; and cause the third component to perform the fourth action, wherein: causing the third component to perform the fourth action includes causing the second means of performing the fifth action, and causing the first component to perform the first action includes causing the first means of performing the third action.

[0208] 19. A computing system as described in clauses 13, 14, 15, 16, 17, or 18, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: generate a second prompt comprising first potential response data, second potential response data, and first input data before generating the first prompt, wherein the second prompt is a second request for the language model to determine a second output in response to the first input data; process the second prompt using the language model to generate second model output data, the second model output data comprising: an indication of an ambiguity associated with the first input data, and a third request for contextual data associated with the ambiguity; determine second output data corresponding to the second model output data, at least in part, based on the second model output data; cause the second output data to be presented; and receive natural language data corresponding to user input associated with the ambiguity, wherein: the first prompt further comprises the user input, and the first model output data is not associated with the ambiguity.

[0209] 20. A computing system as described in Clauses 13, 14, 15, 16, 17, 18 or 19, wherein the first component is a skill and the second component is an agent based on a large language model (LLM).

[0210] The foregoing aspects of this disclosure are intended to be illustrative. They were chosen to explain the principles and applications of this disclosure and are not intended to be exhaustive or limiting. Many modifications and variations of the disclosed aspects will be apparent to those skilled in the art. Those skilled in the art of computer and speech processing will recognize that the components and process steps described herein are interchangeable with other components or steps, or combinations of components or steps, and still achieve the benefits and advantages of this disclosure. However, it will be apparent to those skilled in the art that this disclosure can be practiced without some or all of the specific details and steps disclosed herein. Furthermore, unless expressly stated to the contrary, features / operations / components, etc., from one embodiment discussed herein may be combined with features / operations / components, etc., from another embodiment discussed herein.

[0211] Aspects of the disclosed system may be implemented as computer methods or articles of art, such as memory devices or non-transitory computer-readable storage media. The computer-readable storage media may be readable by a computer and may include instructions for causing a computer or other device to perform the processes described in this disclosure. The computer-readable storage media may be implemented as volatile computer memory, non-volatile computer memory, hard disk drive, solid-state memory, flash memory drive, removable disk drive, and / or other media. Furthermore, components of the system may be implemented as firmware or hardware.

[0212] Unless otherwise specifically indicated, or otherwise understood as used in the context herein, the conditional language used herein (such as, among other things, “can,” “may,” “possibly,” “can,” “e.g.,” etc.) is generally intended to convey that certain embodiments include certain features, elements, and / or steps that are not included in other embodiments. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments necessarily include logic for determining, with or without further input or prompting, whether such features, elements, and / or steps are included or to be performed in any particular embodiment. The terms “comprise,” “include,” “have,” etc., are synonymous and used inclusively in an open manner without excluding additional elements, features, actions, operations, etc. Additionally, the term “or” is used in its inclusive sense (but not its exclusive sense) such that when used, for example, to connect elements in a list, the term “or” means one, some, or all of the elements in the list.

[0213] Unless otherwise specifically stated, disjunctive languages ​​such as “at least one of X, Y, or Z” are understood in context to generally represent that items, terms, etc., can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive languages ​​are generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0214] As used in this disclosure, unless otherwise specifically indicated, the term "a" or "an" may include one or more items. Furthermore, unless otherwise explicitly indicated, the phrase "based on" is intended to mean "at least partially based on".

Claims

1. A computer-implemented method, the method comprising: Receive the first input data; Receive first potential response data from a first component configured to perform a first action associated with the first input data; Receive second potential response data corresponding to a second component configured to perform a second action associated with the first input data; Determine a first prompt including the first potential response data, the second potential response data, and the first input data, wherein the first prompt is a first request for the language model to determine a first output in response to the first input data; The language model is used to process the first prompt to generate first model output data, the first model output data indicating that the first action responds to the first input data; as well as This causes the first component to perform the first action.

2. The computer-implemented method as described in claim 1, further comprising: Receive the second input data; Receive third potential response data from a third component, the third potential response data being associated with the second input data; Receive fourth potential response data from the fourth component, the fourth potential response data being associated with the second input data; Generate a second prompt including the third potential response data, the fourth potential response data, and the second input data, wherein the second prompt is a second request for the language model to determine a second output in response to the second input data; The language model is used to process the second prompt to generate second model output data, which indicates: The third potential response data responds to the second input data, and The fourth potential response data responds to the second input data, wherein the second model output data includes second output data representing a summary of the third and fourth potential response data; and The second output data is presented in response to the second input data.

3. The computer-implemented method of claim 1 or 2, wherein the first potential response data is associated with a first task of the first input data, the second potential response data is associated with a second task of the first input data, and the method further comprises: Before receiving the second potential response data, third potential response data corresponding to a third component configured to perform a third action associated with the first input data is received, wherein the third potential response data is associated with the first task; Determine a second prompt including the first potential response data and the third potential response data, wherein the second prompt is a second request for the language model to determine a second output in response to the first task of the first input data; as well as The language model is used to process the second cue to generate second model output data, which indicates that the first potential response data responds to the first task of the first input data, rather than the third potential response data.

4. The computer-implemented method as described in claim 1, 2, or 3, further comprising: Before generating the first prompt, a second prompt is generated, including the first potential response data, the second potential response data, and the first input data, wherein the second prompt is a second request for the language model to determine a second output in response to the first input data; The language model is used to process the second prompt to generate second model output data, which includes: Indications of ambiguity associated with the first input data, and A third request for the first data associated with the ambiguity; The second model output data is sent to the third component, at least in part, based on the second model output data; and Receive first context data associated with the ambiguity from the third component, wherein: The first prompt also includes the first context data, and The output data of the first model is not related to the ambiguity.

5. The computer-implemented method as described in claim 1, 2, 3, or 4, wherein the first action corresponds to a first means of performing the third action, and the method further comprises: Determine first output data corresponding to the first action, wherein the first output data indicates the execution of the first action by the first device; as well as After the first component performs the first action, the first output data is presented.

6. The computer-implemented method as described in claim 1, 2, 3, 4, or 5, wherein the first action corresponds to a first means of performing the third action, and the method further comprises: Receive third potential response data corresponding to a third component configured to perform a fourth action associated with the first input data, wherein: The fourth action corresponds to the second device that performs the fifth action. The first prompt also includes the third potential response data, and The first model output data further indicates that the fourth action is in response to the first input data; and The third component performs the fourth action, wherein: Making the third component perform the fourth action includes making the second device perform the fifth action, and Making the first component perform the first action includes making the first device perform the third action.

7. The computer-implemented method as described in claim 1, 2, 3, 4, 5, or 6, further comprising: Before generating the first prompt, a second prompt is generated, including the first potential response data, the second potential response data, and the first input data, wherein the second prompt is a second request for the language model to determine a second output in response to the first input data; The language model is used to process the second prompt to generate second model output data, which includes: Indications of ambiguity associated with the first input data, and A third request for contextual data associated with the ambiguity; Based at least in part on the output data of the second model, determine the second output data corresponding to the output data of the second model; This results in the presentation of the second output data; and Receive natural language data corresponding to user input associated with the ambiguity, wherein: The first prompt also includes the user input, and The output data of the first model is not related to the ambiguity.

8. The computer-implemented method as described in claim 1, 2, 3, 4, 5, 6 or 7, wherein the first component is a skill and the second component is an agent based on a large language model (LLM).

9. A computing system, the computing system comprising: At least one processor; as well as At least one memory, the at least one memory including instructions that, when executed by the at least one processor, cause the computing system to: Receive the first input data; Receive first potential response data from a first component configured to perform a first action associated with the first input data; Receive second potential response data corresponding to a second component configured to perform a second action associated with the first input data; Determine a first prompt including the first potential response data, the second potential response data, and the first input data, wherein the first prompt is a first request for the language model to determine a first output in response to the first input data; The language model is used to process the first prompt to generate first model output data, the first model output data indicating that the first action responds to the first input data; as well as This causes the first component to perform the first action.

10. The computing system of claim 9, wherein the at least one memory further comprises instructions, which, when executed by the at least one processor, further cause the computing system to: Receive the second input data; Receive third potential response data from a third component, the third potential response data being associated with the second input data; Receive fourth potential response data from the fourth component, the fourth potential response data being associated with the second input data; Generate a second prompt including the third potential response data, the fourth potential response data, and the second input data, wherein the second prompt is a second request for the language model to determine a second output in response to the second input data; The language model is used to process the second prompt to generate second model output data, which indicates: The third potential response data responds to the second input data, and The fourth potential response data responds to the second input data, wherein the second model output data includes second output data representing a summary of the third and fourth potential response data; and The second output data is presented in response to the second input data.

11. The computing system of claim 9 or 10, wherein the first potential response data is associated with a first task of the first input data, the second potential response data is associated with a second task of the first input data, and wherein the at least one memory further includes instructions, which, when executed by the at least one processor, further cause the computing system to: Before receiving the second potential response data, third potential response data corresponding to a third component configured to perform a third action associated with the first input data is received, wherein the third potential response data is associated with the first task; Determine a second prompt including the first potential response data and the third potential response data, wherein the second prompt is a second request for the language model to determine a second output in response to the first task of the first input data; and The language model is used to process the second cue to generate second model output data, which indicates that the first potential response data responds to the first task of the first input data, rather than the third potential response data.

12. The computing system of claim 9, 10, or 11, wherein the at least one memory further comprises instructions, which, when executed by the at least one processor, further cause the computing system to: Before generating the first prompt, a second prompt is generated, including the first potential response data, the second potential response data, and the first input data, wherein the second prompt is a second request for the language model to determine a second output in response to the first input data; The language model is used to process the second prompt to generate second model output data, which includes: Indications of ambiguity associated with the first input data, and A third request for the first data associated with the ambiguity; The second model output data is sent to the third component, at least in part, based on the second model output data; and Receive first context data associated with the ambiguity from the third component, wherein: The first prompt also includes the first context data, and The output data of the first model is not related to the ambiguity.

13. The computing system of claim 9, 10, 11, or 12, wherein the first action corresponds to a first means of performing a third action, and wherein the at least one memory further includes instructions, which, when executed by the at least one processor, further cause the computing system to: Determine first output data corresponding to the first action, the first output data indicating the execution of the first action by the first device; and After the first component performs the first action, the first output data is presented.

14. The computing system of claim 9, 10, 11, 12 or 13, wherein the first action corresponds to a first means of performing a third action, and wherein the at least one memory further includes instructions, which, when executed by the at least one processor, further cause the computing system to: Receive third potential response data corresponding to a third component configured to perform a fourth action associated with the first input data, wherein: The fourth action corresponds to the second device that performs the fifth action. The first prompt also includes the third potential response data, and The first model output data further indicates that the fourth action is in response to the first input data; and The third component performs the fourth action, wherein: Making the third component perform the fourth action includes making the second device perform the fifth action, and Making the first component perform the first action includes making the first device perform the third action.

15. The computing system of claim 9, 10, 11, 12, 13 or 14, wherein the at least one memory further comprises instructions, which, when executed by the at least one processor, further cause the computing system to: Before generating the first prompt, a second prompt is generated, including the first potential response data, the second potential response data, and the first input data, wherein the second prompt is a second request for the language model to determine a second output in response to the first input data; The language model is used to process the second prompt to generate second model output data, which includes: Indications of ambiguity associated with the first input data, and A third request for contextual data associated with the ambiguity; Based at least in part on the output data of the second model, determine the second output data corresponding to the output data of the second model; This results in the presentation of the second output data; and Receive natural language data corresponding to user input associated with the ambiguity, wherein: The first prompt also includes the user input, and The output data of the first model is not related to the ambiguity.

Images