Using large-scale language models when generating automated assistant responses

Abstract

To use large language models to generate automated assistant responses.SOLUTION: As part of an interaction session between a user and an automated assistant, an implementation: receives a stream of audio data capturing utterances containing assistant queries; determines a set of predicted assistant outputs each corresponding to a response to an assistant query, based on processing the stream of audio data; uses the large language model (LLM) output to process the assistant output and context of the interaction session, to generate a set of modified assistant outputs; and facilitates a given modified assistant output to be provided for presentation to the user in response to the utterance, from the set of modified assistant outputs. In some implementations, the LLM output may be generated offline for later use online. In additional or alternative implementations, the LLM output may be generated online when the utterance is received.SELECTED DRAWING: Figure 2

Description

[Background technology]

[0001] Humans may engage in human-to-computer interactions with conversational software applications referred to herein as “automated assistants” (also known as “chatbots,” “conversational personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” etc.). Automated assistants typically rely on a pipeline of components to interpret and respond to utterances. For example, an automated speech recognition (ASR) engine may process audio data corresponding to a user’s utterance to produce an ASR output, such as an ASR hypothesis of the utterance (i.e., a set of terms and / or other tokens). Furthermore, a natural language understanding (NLU) engine may process the ASR output (or touch / typed input) to produce an NLU output, such as a request (e.g., intent) expressed by the user when providing the utterance (or touch / typed input), and optionally, slot values ​​for parameters related to that intent. Finally, the NLU output may be processed by various performance components to produce performance outputs, such as response content that responds to the utterance and / or one or more actions that may be performed in response to the utterance.

[0002] Generally, a conversation session with an automated assistant is initiated by a user providing an utterance, and the automated assistant can respond to the utterance using the pipeline of the aforementioned components. The user can continue the conversation session by providing additional utterances, and the automated assistant can again respond to the additional utterances using the pipeline of the aforementioned components. In other words, these conversation sessions are generally based on a turn, where the turn for providing an utterance in the conversation session is with the user, the turn for responding to an utterance in the conversation session is with the automated assistant, the additional turn for providing an additional utterance in the conversation session is with the user, the additional turn for responding to the additional utterance in the conversation session is with the automated assistant, and so on. However, from the user's perspective, these turn-based conversation sessions may not be natural as they do not reflect the way humans actually converse with each other.

[0003] For example, if a first person provides an utterance to convey an initial thought to a second person during a conversation session (e.g., "I'm going to the beach today"), the second person can consider that utterance within the context of the conversation session when stating a response to the first person (e.g., "sounds fun, what are you going to do at the beach?", "nice, have you looked at the weather?", etc.). In particular, the second person can provide an utterance that keeps the first person involved in the conversation session in a natural way when responding to the first person. In other words, during the conversation session, rather than one person leading the conversation session, both the first person and the second person can provide utterances to facilitate a natural conversation.

[0004] However, if the second person is replaced by an automated assistant in the above example, the automated assistant may not provide responses that keep the first person involved in the conversational session. For example, in response to the first person's utterance, "I'm going to the beach today," the automated assistant could take some action to facilitate the conversational session and / or provide some response, such as proactively asking the first person what they plan to do at the beach, proactively looking up the weather forecast for the beach the first person frequently visits and including the forecast in the response, or proactively making some inferences based on the weather forecast. Instead, it might simply respond with "sound fun" or "nice" without providing any additional responses to facilitate the conversational session. As a result, the responses provided by the automated assistant in response to the first person's utterances may not reflect a natural conversation between multiple people and may therefore not resonate with the first person. Furthermore, the first person may have to provide additional utterances to explicitly request some information that the automated assistant could proactively provide (for example, a beach weather forecast), thus increasing the amount of utterances directed at the automated assistant and wasting the computing resources of the client device used to process these utterances. [Overview of the project] [Means for solving the problem]

[0005] The implementations described herein are intended to enable an automated assistant to engage in natural conversation with a user during a dialogue session. Some implementations can receive a stream of audio data capturing user utterances. The stream of audio data may be generated by one or more microphones on a client device, and the utterances may include assistant queries. Some implementations can further process a set of assistant outputs and the context of the dialogue session to determine a set of automated assistants based on having processed the stream of audio data, and to generate a set of modified assistant outputs using one or more LLM outputs generated using a Large-Scale Language Model (LLM). Each of the one or more LLM outputs may be determined based on at least a portion of the context of the dialogue session and one or more assistant outputs included in the set of assistant outputs. Some implementations can further ensure that a given modified assistant output is provided from the set of modified assistant outputs for presentation to the user. Furthermore, each of the one or more LLM outputs may include, for example, a probability distribution over a set of one or more words and / or phrases spanning one or more vocabulary, and one or more of that set of words and / or phrases may be selected as one or more LLM outputs based on the probability distribution. Furthermore, the context of a dialogue session may be determined based on one or more contextual signals, including, for example, the time of day, the location of the client device, ambient noise detected in the environment of the client device, user profile data, software application data, environmental data about the user's known environment on the client device, the dialogue history of the dialogue session between the user and the automated assistant, and / or other contextual signals.

[0006] In some implementations, the set of assistant outputs may be determined based on processing a stream of audio data using a streaming ASR model to generate a stream of ASR outputs, such as one or more recognized terms or phrases predicted to correspond to an utterance, one or more phonemes predicted to correspond to an utterance, one or more predicted measures associated with each of the one or more recognized terms or phrases and / or one or more predicted phonemes, and / or other automatic speech recognition (ASR) outputs. Furthermore, the ASR outputs may be processed using an NLU model to generate a stream of natural language understanding (NLU) outputs, such as one or more predicted intentions of the user when providing an utterance, and one or more corresponding slot values ​​for one or more parameters associated with each of the one or more predicted intentions. Moreover, the stream of NLU data may be processed by one or more first-party (1P) and / or third-party (3P) systems to generate a set of assistant outputs. In this specification, one or more 1P systems include systems developed and / or maintained by the same entity (e.g., a common publisher) that develops and / or maintains the automated assistants described herein, while one or more 3P systems include systems developed and / or maintained by entities separate from the entity that develops and / or maintains the automated assistants described herein. In particular, the set of assistant outputs described herein includes assistant outputs that are typically considered to respond to utterances. However, by using the claimed technique, the set of assistant outputs produced in the manner described above may be further processed to produce a modified set of assistant outputs. Specifically, the set of assistant outputs may be modified using one or more LLM outputs, and a given modified assistant output may be selected from the modified set of assistant outputs to be provided for presentation to the user in response to receiving an utterance.

[0007] For example, suppose a user on a client device provides the utterance, "Hey Assistant, I'm thinking about going surfing today." In this example, the automated assistant can process the utterance in the manner described above to generate a set of assistant outputs and a set of modified assistant outputs. The assistant outputs included in the set of assistant outputs in this example may include, for example, "That sounds like fun!" and "Sounds fun!". Furthermore, the assistant outputs included in the set of modified assistant outputs in this example may include, for example, "That sounds like fun, how long have you been surfing?" and "Enjoy it, but if you're going to Example Beach again, be prepared for some light showers." In particular, the Assistant Outputs included in the set of Assistant Outputs do not include any Assistant Outputs that drive the conversation session in a way that further engages the user of the client device in the conversation session, whereas the Assistant Outputs included in the modified set of Assistant Outputs include Assistant Outputs that drive the conversation session in a way that further engages the user of the client device in the conversation session by asking contextually relevant questions (e.g., "how long have you been surfing?"), Assistant Outputs that provide contextually relevant information (e.g., "but if you're going to Example Beach again, be prepared for some light showers"), and / or Assistant Outputs that resonate differently with the user of the client device within the context of the conversation session.

[0008] In some implementations, a modified set of assistant responses may be generated using one or more LLM outputs produced online. For example, in response to receiving an utterance, the automated assistant may be configured to generate a set of assistant outputs in the manner described above. Furthermore, also in response to receiving an utterance, the automated assistant may be configured to process a set of assistant outputs, the context of the dialogue session, and / or assistant queries contained in the utterance using one or more LLMs in order to generate a modified set of assistant outputs based on one or more LLM outputs produced using one or more LLMs.

[0009] In additional or alternative implementations, a modified set of assistant responses may be generated using one or more LLM outputs produced in an offline manner. For example, before receiving an utterance, the automated assistant may retrieve multiple assistant queries and the corresponding context of the corresponding previous interaction session for each of the multiple assistant queries from an assistant activity database (which may be a limited set of assistant activities of the user on the client device). Furthermore, the automated assistant may ensure that for a given assistant query of multiple assistant queries, a set of assistant outputs is generated in the manner described above and for the given assistant query. Moreover, the automated assistant may ensure that a set of assistant outputs, the corresponding context of the interaction session, and / or a given assistant query are processed using one or more LLMs in order to generate a modified set of assistant outputs based on one or more LLM outputs produced using one or more LLMs. This process may be repeated for each of the multiple queries and the corresponding context of the previous interaction session retrieved by the automated assistant.

[0010] In addition, the automated assistant can index one or more LLM outputs in memory accessible by the user's client device. In some implementations, the automated assistant can ensure that one or more LLMs are indexed in memory based on one or more terms contained in multiple assistant queries. In additional or alternative implementations, the automated assistant can generate corresponding embeddings (e.g., word2vec embeddings, or other lower-dimensional representations) for each of the multiple assistant queries, map each of the corresponding embeddings to the assistant query embedding space, and index one or more LLM outputs. In additional or alternative implementations, the automated assistant can ensure that one or more LLMs are indexed in memory based on one or more context signals contained in the corresponding previous context. In additional or alternative implementations, the automated assistant can generate corresponding embeddings for each of the corresponding contexts, map each of the corresponding embeddings to the context embedding space, and index one or more LLM outputs. In an additional or alternative implementation, the automated assistant may ensure that one or more LLMs are indexed in memory based on one or more terms or phrases in the assistant outputs that are included in a set of assistant outputs for each of several assistant queries. In an additional or alternative implementation, the automated assistant may generate a corresponding embedding (e.g., a word2vec embedding, or another lower-dimensional representation) for each of the assistant outputs included in the set of assistant outputs, and map each of the corresponding embeddings to the assistant output embedding space to index one or more LLM outputs.

[0011] Therefore, as the utterance continues and is received on the user's client device, the automated assistant can identify one or more previously generated LLM outputs based on the current assistant query corresponding to one or more assistant queries included in multiple queries, the current context corresponding to one or more corresponding previous contexts, and / or one or more previous assistant outputs corresponding to one or more previous assistant outputs. For example, in an implementation where one or more LLM outputs are indexed based on corresponding embeddings for previous assistant queries, the automated assistant can ensure that embeddings for the current assistant query are generated and mapped into the assistant query embedding space. Furthermore, the automated assistant can determine that the current assistant query corresponds to a previous assistant query based on whether the distance between the embedding for the current assistant query and the corresponding embedding for the previous assistant query in the query embedding space satisfies a threshold. The automated assistant can retrieve one or more LLM outputs from memory that were generated based on having processed the previous assistant query and utilize one or more LLM outputs when generating a modified set of assistant outputs. Furthermore, in an implementation where one or more LLMs are indexed based on one or more terms contained in multiple assistant queries, for example, the automated assistant can determine the edit distance between the current assistant query and the multiple previous assistant queries to identify, for example, previous assistant queries corresponding to the current assistant query. Similarly, the automated assistant can retrieve one or more LLM outputs from memory that were generated based on having processed previous assistant queries and utilize one or more LLM outputs when generating a modified set of assistant outputs.

[0012] In some implementations, in addition to one or more LLM outputs, additional assistant queries may be generated based on the processing of the assistant query and / or the context of the interaction session. For example, when processing the assistant query and / or the context of the interaction session, the automated assistant can determine the intent associated with a given assistant query based on a stream of NLU data. Furthermore, based on the intent associated with a given assistant query, the automated assistant can identify at least one related intent related to the intent associated with the assistant query (for example, based on the mapping of intent to at least one related intent in a database or memory accessible to the client device, and / or based on processing the intent associated with a given assistant query using one or more machine learning (ML) models or heuristically defined rules). Moreover, the automated assistant can generate additional assistant queries based on at least one related intent. For example, suppose the assistant query indicates that the user is planning to go to the beach (e.g., "Hey assistant, I'm going to the beach today"). In this example, an additional assistant query might correspond to, for example, "what's the weather at Example Beach?" (for example, to proactively determine the weather information for a beach named Example Beach that the user usually visits). In particular, additional assistant queries do not need to be provided for presentation to the user on the client device.

[0013] Rather, in these implementations, additional assistant outputs may be determined based on the processing of additional assistant queries. For example, the automated assistant may send a structured request to one or more 1P and / or 3P systems to obtain weather information as additional assistant output. Furthermore, suppose the weather information indicates that rain is expected at Example Beach. In some versions of these implementations, the automated assistant may further have additional assistants processed using one or more LLM outputs and / or one or more additional LLM outputs to generate an additional set of modified assistant outputs. Thus, in the initial example given above, a given modified assistant output provided to the user in response to receiving the utterance "Hey Assistant, I'm thinking about going surfing today" from the initial set of modified assistant outputs may be "Enjoy it," and a given additional modified assistant output from the additional set of modified assistant outputs may be "but if you're going to Example Beach again, be prepared for some light showers." In other words, the automated assistant

[0014] In various implementations, when generating a set of modified assistant outputs, each of the one or more LLM outputs used may be generated using a corresponding set of parameters from several separate sets of parameters. Each of the several separate sets of parameters may be associated with a separate personality for the automated assistant. Some versions of these implementations may utilize a single LLM to generate one or more corresponding LLM outputs using the corresponding set of parameters for each of the separate personalities, while other versions of these implementations may utilize multiple LLMs to generate one or more corresponding LLM outputs using the corresponding set of parameters for each of the separate personalities. Thus, when a given modified assistant output is provided for presentation to the user from a set of modified assistant outputs, it can reflect a variety of changing contextual personalities through the prosodic properties of the different personalities (e.g., the intonation, pitch, tone, pauses, tempo, stress, rhythm, etc. of these different personalities).

[0015] In particular, the responses of these personalities described herein may reflect not only the prosodic nature of different personalities, but also the distinct vocabulary and / or distinct speaking styles of different personalities (e.g., verbose speaking style, concise speaking style, etc.). For example, a given modified assistant output provided for presentation to a user may be generated using a first set of parameters that reflect a first personality of the automated assistant with respect to a first vocabulary to be utilized by the automated assistant, and / or a first set of prosodic natures to be utilized when providing the modified assistant output for audible presentation to the user. Alternatively, the modified assistant output provided for presentation to a user may be generated using a second set of parameters that reflect a second personality of the automated assistant with respect to a second vocabulary to be utilized by the automated assistant, and / or a second set of prosodic natures to be utilized when providing the modified assistant output for audible presentation to the user.

[0016] Therefore, the automated assistant can dynamically adapt the personality used when providing modified assistant output for presentation to the user, based on both the vocabulary used by the automated assistant and the prosodic properties used when rendering modified assistant output for audible presentation to the user. In particular, the automated assistant can dynamically adapt these personalities used when providing modified assistant output based on the context of the dialogue session, including previous utterances received from the user, as well as previous assistant output provided by the automated assistant and / or any other contextual signals described herein. As a result, the modified assistant output provided by the automated assistant may resonate more with the user of the client device. Furthermore, it should be noted that the personality used throughout a given dialogue session can be dynamically adapted as the context of the given dialogue session is updated.

[0017] In some implementations, the automated assistant may rank the assistant outputs included in a set of assistant outputs (i.e., not generated using one or more LLM outputs) and a set of modified assistant outputs (i.e., generated using one or more LLM outputs) according to one or more ranking criteria. Thus, when selecting a given assistant output to be provided for presentation to the user, the automated assistant can choose from both the set of assistant outputs and the set of modified assistant outputs. The one or more ranking criteria may include, for example, one or more predictive measures indicating how well each of the assistant outputs included in the set of assistant outputs and the set of modified assistant outputs is expected to respond to the assistant query contained in the utterance (e.g., an ASR measure generated when generating a stream of ASR outputs, an NLU measure generated when generating a stream of NLU outputs, a performance measure generated when generating a set of assistant outputs), one or more intentions contained in the stream of NLU outputs, and / or other ranking criteria. For example, if the user intent of the client device indicates that the user wants a factual answer (for example, based on providing an utterance that includes the assistant query "why is the sky blue?"), the user is likely to want a simple answer to the assistant query, so the automated assistant can rely on one or more assistant outputs from a set of one or more assistant outputs. However, if the user intent of the client device indicates that the user has provided open-ended input (for example, based on providing an utterance that includes the assistant query "what time is it?"), the user is likely to prefer a more conversational approach, so the automated assistant can rely on one or more assistant outputs from a modified set of assistant outputs.

[0018] In some implementations, the automated assistant may even decide whether to generate a set of modified assistant outputs before generating the set of modified assistant outputs. In some versions of those implementations, the automated assistant may even decide whether to generate a set of modified assistant outputs based on one or more of the user's expected intentions when providing an utterance such as one indicated by a stream of NLU data. For example, in an implementation where the automated assistant decides not to generate a set of modified assistant outputs because the utterance is asking the automated assistant to perform a search (e.g., the assistant query "Why is the sky blue?"), the automated assistant may decide not to generate a set of modified assistant outputs because the user is asking for a factual answer. In additional or alternative versions of those implementations, the automated assistant may even decide whether to generate a set of modified assistant outputs based on one or more computational costs associated with modifying one or more of the assistant outputs. One or more computational costs associated with modifying one or more of the assistant outputs may include, for example, battery consumption, processor consumption associated with modifying one or more of the assistant outputs, or latency associated with modifying one or more of the assistant outputs. For example, if the client device is in low-power mode, the automatic assistant may decide not to generate a modified set of assistant outputs in order to reduce the battery consumption of the client device.

[0019] One or more technical advantages can be achieved by using the techniques described herein. In one limited example, the techniques described herein enable an automated assistant to engage in natural conversation with a user during a dialogue session. For example, an automated assistant can generate modified assistant outputs using one or more LLM outputs of a more conversational nature. Thus, the automated assistant can proactively provide contextual information relevant to the dialogue session that was not directly requested by the user (for example, by generating additional assistant queries as described herein, and by providing additional assistant outputs determined based on those additional assistant queries), thereby making the modified assistant outputs resonate with the user. Furthermore, the modified assistant outputs may be generated with varying personalities in terms of both contextually adapted vocabulary throughout the dialogue session and prosodic properties used to render the modified assistant outputs audible, thereby making the modified assistant outputs even more resonant with the user. This results in various technical advantages that conserve computational resources on the client device, allow dialogue sessions to conclude in a faster and more efficient manner, and / or reduce the length of dialogue sessions. For example, by proactively providing contextually relevant information to the user through the automated assistant, the amount of user input received on the client device can be reduced, thereby decreasing the number of situations in which the user must request such information. Furthermore, in implementations where one or more LLM outputs are generated offline and then used online, runtime latency can be reduced.

[0020] In this specification, a “dialogue session” may include a logically self-contained exchange between a user and an automated assistant (and, in some cases, other human participants). The automated assistant may distinguish between multiple dialogue sessions with a user based on various signals, such as the passage of time between sessions, changes in the user’s situation between sessions (e.g., location, before / during / after an scheduled meeting, etc.), detection of one or more interfering interactions between the user and the client device other than the dialogue between the user and the automated assistant (e.g., the user switching applications for a while, the user leaving a standalone voice-operated product and returning later), the locking / sleeping of the client device between sessions, and changes in the client device used to interface with the automated assistant. In particular, during a given dialogue session, the user may interact with the automated assistant using a variety of input methods, including, but not limited to, spoken input, typed input, and / or touch input.

[0021] The above description is provided as an overview of only some of the implementations disclosed herein. These implementations and other implementations are described in further detail herein.

[0022] It should be understood that the techniques disclosed herein may be implemented locally on a client device, remotely by a server connected to the client device via one or more networks, and / or both. [Brief explanation of the drawing]

[0023] [Figure 1] This is a block diagram illustrating various aspects of the present disclosure and of an exemplary environment in which the implementations disclosed herein may be implemented. [Figure 2] This diagram illustrates a process flow that demonstrates the use of large-scale language models when generating assistant output, across various implementation forms. [Figure 3] A flowchart showing an exemplary method of utilizing a large language model when generating assistant output in an offline manner for later use in an online manner according to various implementation forms. [Figure 4] A flowchart showing an exemplary method of utilizing a large language model when generating assistant output based on generating an assistant query according to various implementation forms. [Figure 5] A flowchart showing an exemplary method of utilizing a large language model when generating assistant output based on generating a response of an assistant personality according to various implementation forms. [Figure 6] A diagram showing a non-limiting example of a dialogue session between a user and an automatic assistant, where the automatic assistant utilizes a large language model when generating assistant output according to various implementation forms. [Figure 7] A diagram showing an exemplary architecture of a computing device according to various implementation forms.

Embodiments for Carrying Out the Invention

[0024] Referring now to FIG. 1, there is shown a block diagram of an exemplary environment 100 that illustrates various aspects of the present disclosure and in which implementations disclosed herein may be implemented. The exemplary environment 100 includes a client device 110 and a natural conversation system 120. In some implementations, the natural conversation system 120 may be implemented locally on the client device 110. In additional or alternative implementations, the natural conversation system 120 may be implemented remotely from the client device 110 (e.g., on a remote server), as shown in FIG. 1. In these implementations, the client device 110 and the natural conversation system 120 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks (LANs, including Wi-Fi LAN, mesh network, Bluetooth, near field communication, etc.) or wide area networks (WANs, including the Internet).

[0025] The client device 110 can be one or more of, for example, a desktop computer, a laptop computer, a tablet, a mobile phone, a vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (optionally having a display), a smart appliance such as a smart TV, and / or a user's wearable device including a computing device (e.g., a user's wristwatch having a computing device, a user's glasses having a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0026] The client device 110 can run an automated assistant client 114. An example of the automated assistant client 114 may be an application separate from the operating system of the client device 110 (for example, installed "on top of" the operating system), or alternatively, it may be implemented directly by the operating system of the client device 110. The automated assistant client 114 can interact with a natural conversation system 120, which may be implemented locally on the client device 110, or implemented and invoked remotely via one or more of the networks 199 as shown in Figure 1. The automated assistant client 114 (and optionally through interaction with other remote systems (e.g., servers)) may form what appears to the user as a logical example of an automated assistant 115 with which the user can engage in a human-to-computer interaction. An example of the automated assistant 115 is shown in Figure 1, enclosed by a dashed line containing the automated assistant client 114 and the natural conversation system 120 on the client device 110. Therefore, it should be understood that a user interacting with the automated assistant client 114 running on client device 110 may, in effect, interact with their own logical instance of the automated assistant 115 (or a logical instance of the automated assistant 115 shared among a household or a group of other users). For brevity and simplicity, as used herein, the automated assistant 115 refers to the automated assistant client 114 running locally on client device 110 and / or remotely on one or more remote servers that may implement the natural conversation system 120.

[0027] In various implementations, the client device 110 may include a user input engine 111 configured to detect user input provided by a user of the client device 110 using one or more user interface input devices. For example, the client device 110 may be equipped with one or more microphones for capturing audio data, such as audio data corresponding to user speech or other sounds in the environment of the client device 110. Additionally or alternatively, the client device 110 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected in one or more fields of view of the visual components. Additionally or alternatively, the client device 110 may be equipped with one or more touch-sensing components (e.g., keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons) configured to capture signals corresponding to touch input directed towards the client device 110.

[0028] In various implementations, the client device 110 may include a rendering engine 112 configured to provide content for audible and / or visual presentation to the user of the client device 110 using one or more user interface output devices. For example, the client device 110 may be equipped with one or more speakers that enable content to be provided for audible presentation to the user via the client device 110. Additionally or alternatively, the client device 110 may be equipped with a display or projector that enables content to be provided for visual presentation to the user via the client device 110.

[0029] In various implementations, the client device 110 may include one or more presence sensors 113 configured to provide signals indicating a detected presence, particularly the presence of a person, upon authorization from the corresponding user. In some of these implementations, the automated assistant 115 can identify which client device 110 (or another computing device associated with the user of the client device 110) should satisfy the utterance, at least in part, based on the presence of the user in the client device 110 (or another computing device associated with the user of the client device 110). The utterance can be satisfied by controlling the client device 110 and / or the other computing device associated with the user of the client device 110 by rendering response content (for example, via the rendering engine 112), and / or by causing the client device 110 and / or the other computing device associated with the user of the client device 110 to perform any other actions necessary to satisfy the utterance. As described herein, the automated assistant 115 can leverage data determined based on the presence sensor 113 when determining which client device 110 (or other computing device) is nearby or has recently been near, and provide corresponding commands only to that client device 110 (or those other computing devices).In some additional or alternative implementations, the automated assistant 115 can leverage data determined based on the presence sensor 113 when determining whether any user (any user or a specific user) is currently near the client device 110 (or other computing device), and can optionally suppress the provision of data to and from the client device 110 (or other computing device) based on the user who is near the client device 110 (or other computing device).

[0030] The presence sensor 113 may take various forms. For example, the client device 110 may utilize one or more of the user interface input components described above with respect to the user input engine 111 to detect the presence of a user. In addition or alternatively, the client device 110 may be equipped with other types of light-based presence sensors 113, such as passive infrared ("PIR") sensors that measure infrared ("IR") light emitted from objects in the field of view.

[0031] As an addition or alternative, in some implementations, the presence sensor 113 may be configured to detect other phenomena related to the presence of a person or a device. For example, in some embodiments, the client device 110 may be equipped with a presence sensor 113 that detects various types of wireless signals (e.g., radio waves, ultrasound, electromagnetic waves, etc.) emitted by other computing devices carried / operated by the user (e.g., mobile devices, wearable computing devices) and / or other computing devices. For example, the client device 110 may be configured to emit waves that are imperceptible to humans, such as ultrasound or infrared waves, which can be detected by other computing devices (e.g., via an ultrasound / infrared receiver such as an ultrasound-enabled microphone).

[0032] As an addition or alternative, the client device 110 may emit other types of human-imperceptible waves, such as radio waves (e.g., Wi-Fi, Bluetooth, cellular, etc.), which can be detected by other computing devices (e.g., mobile devices, wearable computing devices, etc.) carried / operated by the user and used to determine the user's specific location. In some implementations, GPS and / or Wi-Fi triangulation may be used to detect a person's location based on GPS and / or Wi-Fi signals to / from the client device 110, for example. In other implementations, other wireless signal characteristics such as time of flight and signal strength may be used by the client device 110, alone or collectively, to determine the location of a particular person based on signals emitted by other computing devices carried / operated by the user. As an addition or alternative, in some implementations, the client device 110 may perform speaker identification (SID) to recognize the user from the user's voice and / or facial identification (FID) to recognize the user from visual data capturing the user's face.

[0033] In some implementations, the speaker's movement may then be determined, for example, by the presence sensor 113 of the client device 110 (and optionally the client device 110's GPS sensor, Soli chip, and / or accelerometer). In some implementations, based on such detected movement, the user's location may be predicted, and this location may be assumed to be the user's location when any content is rendered on the client device 110 and / or other computing devices, at least in part on the proximity of the client device 110 and / or other computing devices to the user's location. In some implementations, the user may simply be assumed to be at the last location where they interacted with the automated assistant 115, especially if not much time has passed since that last interaction.

[0034] Furthermore, the client device 110 and / or the natural conversation system 120 may include one or more memories for storing data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components that facilitate communication over one or more of the networks 199. In some implementations, one or more software applications may be installed locally on the client device 110, while in other implementations, one or more software applications may be hosted remotely (for example, by one or more servers) and accessible by the client device 110 over one or more of the networks 199.

[0035] In some implementations, the operations performed by the automated assistant 115 may be implemented locally on the client device 110 via the automated assistant client 114. As shown in Figure 1, the automated assistant client 114 may include an automated speech recognition (ASR) engine 130A1, a natural language understanding (NLU) engine 140A1, a large language model (LLM) engine 150A1, and a text-to-speech (TTS) engine 160A1. In some implementations, the operations performed by the automated assistant 115 may be distributed across multiple computer systems, such as when the natural conversation system 120 is implemented remotely from the client device 110 as shown in Figure 1. In these implementations, the automated assistant 115 may, as an addition or alternative, utilize the ASR engine 130A2, NLU engine 140A2, LLM engine 150A2, and TTS engine 160A2 of the natural conversation system 120.

[0036] Each of these engines may be configured to perform one or more functions. For example, the ASR engines 130A1 and / or 130A2 may process a stream of audio data generated by the microphone of the client device 110 that captures utterances, using streaming ASR models stored in the machine learning (ML) model database 115A (e.g., recurrent neural network (RNN) models, transformer models, and / or any other type of ML model capable of performing ASR) to generate a stream of ASR output. In particular, a streaming ASR model may be used to generate a stream of ASR output as the stream of audio data is generated. Furthermore, the NLU engines 140A1 and / or 140A2 may process a stream of ASR output using NLU models stored in the ML model database 115A (e.g., long-term short-term memory (LSTM), gated recurrent units (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or grammar-based rules to generate a stream of NLU output. Furthermore, the automated assistant 115 can process the NLU output to generate a stream of performance data. For example, the automated assistant 115 can send one or more structured requests to one or more first-party (1P) systems 191 and / or one or more third-party (3P) systems 192 via one or more of the network 199 (or one or more application programming interfaces (APIs)), and receive performance data from one or more of the 1P systems 191 and / or 3P systems 192 to generate a stream of performance data. One or more structured requests may, for example, include NLU data to be included in the stream of performance data.The performance data stream may correspond to a set of assistant outputs that are expected to respond to assistant queries contained in utterances captured in a stream of audio data processed by the ASR engines 130A1 and / or 130A2.

[0037] Furthermore, the LLM engines 150A1 and / or 150A2 can process a set of assistant outputs that are expected to respond to assistant queries contained in utterances captured in the stream of audio data processed by the ASR engines 130A1 and / or 130A2. As described herein, for example (with respect to Figures 2-6), in some implementations, the LLM engines 150A1 and / or 150A2 can be modified using one or more LLM outputs to generate a modified set of assistant outputs. In some versions of those implementations (as described with respect to Figure 3, for example), the automated assistant 115 can have one or more LLM outputs generated offline (for example, without responding to utterances being received during a dialogue session) and then utilized online to generate a modified set of assistant outputs (for example, when utterances are received during a dialogue session). In additional or alternative implementations of those implementations (as described, for example, with respect to Figures 4 and 5), the automated assistant 115 may generate one or more LLM outputs online (for example, when an utterance is received during a dialogue session). In these implementations, one or more LLM outputs may be generated based on processing a set of assistant outputs (for example, a stream of performance data), the context of the dialogue session in which the utterance is received (for example, based on one or more context signals stored in the context database 110A), the recognized text corresponding to the assistant query contained in the utterance, and / or other information that the automated assistant 115 can utilize when generating one or more LLM outputs.Each of the one or more LLM outputs may, for example, include a probability distribution over a set of one or more words and / or phrases spanning one or more vocabulary, and one or more of that set of words and / or phrases may be selected as one or more LLM outputs based on the probability distribution. In various implementations, one or more LLM outputs may be stored in the LLM output database 150A for later use when modifying one or more assistant outputs included in the set of assistant outputs.

[0038] Furthermore, in some implementations, the TTS engines 160A1 and / or 160A2 may process text data (e.g., text spoken by the automated assistant 115) using TTS models stored in the ML model database 115A to generate synthesized speech audio data, including computer-generated synthesized speech. The text data may correspond to, for example, one or more assistant outputs from a set of assistant outputs included in a stream of performance data, one or more modified assistant outputs from a set of modified assistant outputs, and / or any other text data described herein. In particular, the ML models stored in the ML model database 115A may be on-device ML models stored locally in the client device 110, or they may be shared ML models accessible to both the client device 110 and / or a remote system when the natural conversation system 120 is not implemented locally in the client device 110. In additional or alternative implementations, audio data corresponding to one or more assistant outputs from a set of assistant outputs included in the performance data stream, one or more modified assistant outputs from a set of modified assistant outputs, and / or any other text data described herein may be stored in memory or one or more databases accessible by the client device 110, so that the automated assistant does not need to use the TTS engines 160A1 and / or 160A2 to generate any synthesized speech audio data when the audio data is provided for audible presentation to the user.

[0039] In various implementations, the ASR output stream may include, for example, a stream of ASR hypotheses (e.g., terminology hypotheses and / or transcription hypotheses) predicted to correspond to user utterances captured in the audio data stream, one or more corresponding predicted values ​​(e.g., probability, log-likelihood, and / or other values) for each ASR hypothesis, multiple phonemes predicted to correspond to user utterances captured in the audio data stream, and / or other ASR outputs. In some versions of these implementations, the ASR engines 130A1 and / or 130A2 may select one or more ASR hypotheses as recognized text corresponding to an utterance (e.g., based on the corresponding predicted values).

[0040] In various implementations, the NLU output stream may include a stream of annotated recognized text, for example, one or more annotations of the recognized text for one or more (e.g., all) terms of the recognized text. For example, NLU engines 140A1 and / or 140A2 may include a portion of a speech tagger (not shown) configured to annotate terms in their grammatical role. Additionally or alternatively, NLU engines 140A1 and / or 140A2 may include an entity tagger (not shown) configured to annotate entity references in one or more segments of the recognized text, such as references to people (e.g., including literary figures, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some implementations, data about entities may be stored in one or more databases, such as in a known graph (not shown). In some implementations, the known graph may include nodes representing known entities (and possibly entity attributes), as well as edges connecting the nodes to represent relationships between entities. Entity taggers can annotate references to entities at a high level of granularity (for example, to allow identification of all references to an entity class such as people) and / or at a low level of granularity (for example, to allow identification of all references to a specific entity such as a particular person). Entity taggers may rely on the content of natural language input to resolve a particular entity, and / or, optionally, may communicate with a known graph or other entity database to resolve a particular entity.

[0041] As an addition or alternative, the NLU engines 140A1 and / or 140A2 may include a coreference resolver (not shown) configured to group, or "cluster," references to the same entity based on one or more contextual queues. For example, the coreference resolver might be used to resolve the term "them" in the input "buy them" to "buy theatre tickets" based on the fact that "theatre tickets" is mentioned in a client device notification rendered immediately before receiving the natural language input "buy them". In some implementations, one or more components of the NLU engines 140A1 and / or 140A2 may depend on annotations from one or more other components of the NLU engines 140A1 and / or 140A2. For example, in some implementations, an entity tagger may depend on annotations from a coreference resolver when annotating all references to a particular entity. Furthermore, in some implementations, for example, coreference resolvers may rely on annotations from entity taggers when clustering references to the same entity.

[0042] Figure 1 illustrates a single client device with a single user, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, one or more additional client devices of a user can also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user can form an ecosystem of devices that can utilize the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (for example, via network 199). As another example, a given client device may be utilized by multiple users in a shared setting (e.g., a group of users, a household).

[0043] As described herein, the automated assistant 115 can determine whether to modify a set of assistant responses using one or more LLM outputs, and / or determine one or more sets of modified assistant outputs based on one or more LLM outputs. The automated assistant 115 can make these decisions using the natural conversation system 120. In various implementations, as shown in Figure 1, the natural conversation system 120 may include, as an addition or alternative, an offline output modification engine 170, an online output modification engine 180, and / or a ranking engine 190. The offline output modification engine 170 may include, for example, an assistant activity engine 171 and an indexing engine 172. Furthermore, the online output modification engine 180 may include, for example, an assistant query engine 181 and an assistant personality engine 182. These various engines of the natural conversation system 120 will be described in more detail with respect to Figures 2 to 5.

[0044] Figure 2 shows an exemplary process flow 200 of using LLM when generating assistant output. The stream of audio data 201 generated by one or more microphones of the client device 110 in Figure 1 may be processed by ASR engines 130A1 and / or 130A2 to generate a stream of ASR output 203. Furthermore, the ASR output 203 may be processed by NLU engines 140A1 and / or 140A2 to generate a stream of NLU output 204. In some implementations, the NLU engines 140A1 and / or 140A2 may process the context 202 of the interaction session between the user of the client device 110 and the automated assistant 115, which is at least partially performed on the user's client device 110. In some versions of these implementations, the context 202 of the interaction session may be determined based on one or more context signals generated by the client device 110 (e.g., time, day of the week, location of the client device 110, ambient noise detected in the environment of the client device 110, and / or other context signals generated by the client device 110). In additional or alternative versions of these implementations, the context 202 of the interaction session may be determined based on one or more context signals stored in a context database 110A accessible by the client device 110 (e.g., user profile data, software application data, environmental data about the user's known environment on the client device 110, interaction history of ongoing interaction sessions between the user and the automated assistant 115 and / or past interaction history of one or more previous interaction sessions between the user and the automated assistant 115, and / or other context data stored in the context database 110A).Furthermore, the stream of NLU output 204 may be processed by one or more of the 1P system 191 and / or 3P systems to generate a stream of performance data including a set of one or more assistant outputs 205, each of which is expected to respond to an utterance captured in the stream of audio data 201.

[0045] Typically, in sequence-based dialogue sessions that do not utilize LLM, the ranking engine 190 may process one or more sets of assistant outputs 205 to rank each of the one or more assistant outputs included in one or more sets of assistant outputs 205 according to one or more ranking criteria, and the automated assistant 115 may select one or more given assistant outputs 207 from the set of one or more assistant outputs 205 to be provided for presentation to the user of the client device 110 in response to receiving an utterance. In some implementations, the selected one or more given assistant outputs 207 may be processed by the TTS engines 160A1 and / or 160A2 to generate synthesized speech audio data containing synthesized speech corresponding to the selected one or more given assistant outputs 207, and the rendering engine 112 may enable the synthesized speech audio data to be rendered audibly by the speaker of the client device 110 for audible presentation to the user of the client device 110. In additional or alternative implementations, the rendering engine 112 may cause text data corresponding to one or more selected given assistant outputs 207 to be visually rendered by the display of the client device 110 for visual presentation to the user of the client device 110.

[0046] However, when using the claimed technique, the automated assistant 115 may further have one or more sets of assistant outputs 205 processed by LLM engines 150A1 and / or 150A2 in order to generate one or more sets of modified assistant outputs 206. In some implementations, one or more LLM outputs may be generated offline in advance (for example, before receiving a stream of audio data 201 generated by one or more microphones of the client device 110) using an offline output modification engine 170, and one or more LLM outputs may be stored in an LLM output database 150A. As described with respect to Figure 3, one or more LLM outputs may be pre-indexed in the LLM output database 150A based on the corresponding assistant query and / or the corresponding context of the corresponding dialogue session in which the corresponding assistant query was received. Furthermore, the LLM engines 150A1 and / or 150A2 can determine that an assistant query contained in an utterance captured in the stream of audio data 201 matches a corresponding assistant query, and / or that the context 202 of the dialogue session in which the assistant query is received matches the corresponding context of the corresponding dialogue session in which the corresponding assistant query was received. The LLM engines 150A1 and / or 150A2 can obtain one or more LLM outputs indexed by the corresponding assistant query and / or the corresponding context that matches the assistant query in order to modify one or more sets of assistant outputs 205. Moreover, one or more sets of assistant outputs 205 may be modified based on one or more LLM outputs, thereby obtaining one or more sets of modified assistant outputs 206.

[0047] In additional or alternative implementations, one or more LLM outputs may be generated online (for example, in response to receiving a stream of audio data 201 generated by one or more microphones of the client device 110) using the online output modification engine 180. As described with reference to Figures 4 and 5, one or more LLM outputs may be generated using one or more LLMs stored in the ML model database 115A, based on processing one or more sets of assistant outputs 205, recognized text corresponding to assistant queries contained in utterances captured in the stream of audio data 201 (for example, contained in the stream of ASR output 203), and / or the context 202 of the dialogue session between the user of the client device 110 and the automated assistant 115. Furthermore, one or more sets of assistant outputs 205 may be modified based on one or more LLM outputs, thereby obtaining one or more sets of modified assistant outputs 206. In other words, in these implementations, the LLM engines 150A1 and / or 150A2 can directly generate one or more sets of modified assistant outputs 206 using one or more LLMs stored in the ML model database 115A.

[0048] In these implementations, in contrast to the typical sequence-based dialogue sessions described without LLM, the ranking engine 190 may process one or more sets of assistant outputs 205 and one or more sets of modified assistant outputs 206 to rank each of the one or more assistant outputs contained in both sets of one or more assistant outputs 205 and one or more sets of modified assistant outputs 206 according to one or more ranking criteria. Thus, when selecting one or more given assistant outputs 207, the automated assistant 207 can select from one or more sets of assistant outputs 205 and one or more sets of modified assistant outputs 206. In particular, one or more given assistant outputs 207 may resonate more with the user of the client device 110 because the assistant outputs contained in the set of one or more modified assistant outputs 206 may be generated based on the set of one or more assistant outputs 205 and convey the same or similar information, but may also convey the same or similar information with additional information that is relevant to the context of the dialogue 202 (as described with respect to Figure 4, for example), and / or is more natural and fluent, and / or is more in line with the character of the automated assistant.

[0049] One or more ranking criteria may include, for example, one or more predicted measures indicating how well each of the assistant outputs in one or more sets of assistant outputs 205 and one or more sets of modified assistant outputs 206 is expected to respond to assistant queries contained in utterances captured in the stream of audio data 201 (e.g., an ASR measure generated by ASR engines 130A1 and / or 130A2 when generating the stream of ASR output 203, an NLU measure generated by NLU engines 140A1 and / or 140A2 when generating the stream of NLU output 204, a performance measure generated by one or more 1P systems 191 and / or 3P systems 192), one or more intentions contained in the stream of NLU output 204, a measure derived from a classifier that processes each of the assistant outputs in one or more sets of assistant outputs 205 and one or more sets of modified assistant outputs 206 for determining how natural, fluent, and / or fitting the character of an automated assistant when provided for presentation to the user, and / or other ranking criteria. For example, if the user intent of the client device 110 indicates that the user desires a factual answer (for example, based on providing an utterance that includes the assistant query "why is the sky blue?"), the user is likely to prefer a simple answer to the assistant query, and therefore the ranking engine 190 can favor one or more assistant outputs from a set of one or more assistant outputs 205. However, if the user intent of the client device 110 indicates that the user has provided open-ended input (for example, based on providing an utterance that includes the assistant query "what time is it?"), the user is likely to prefer a more conversational approach, and therefore the ranking engine 190 can favor one or more assistant outputs from a set of one or more modified assistant outputs 206.

[0050] Figures 1 and 2 illustrate a voice-based dialogue session as described herein, but it should be understood that this is for illustrative purposes only and not intended to be limiting. Rather, it should be understood that the techniques described herein can be used regardless of the user's input method. For example, in some implementations where the user provides input typed and / or touch input as an assistant query, the automatic assistant 115 may process the typed input using the NLU engines 140A1 and / or 140A2 to generate a stream of NLU output 204 (e.g., skipping the processing of the stream of audio data 201), and the LLM engines 150A1 and / or 150A2 may utilize the text input corresponding to the assistant query (e.g., derived from the typed and / or touch input) when generating one or more sets of modified assistant outputs 206 in the same or similar manner as described above.

[0051] Figure 3 shows a flowchart illustrating Method 300, an example of using a large language model when generating assistant output in an offline manner for later use in an online manner. For convenience, the operation of Method 300 is described with reference to a system that performs the operations from process flow 200 in Figure 2. This system of Method 300 includes one or more processors, memory, and / or other components of a computing device (e.g., client device 110 in Figure 1, client device 610 in Figure 6, and / or computing device 710 in Figure 7, one or more servers, and / or other computing devices). Furthermore, the operations of Method 300 are shown in a particular order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0052] In block 352, the system retrieves multiple assistant queries directed to the automated assistant, and the corresponding context of the previous interaction session for each of the multiple assistant queries. For example, the system can cause the assistant activity engine 171 of the offline output correction engine in Figures 1 and 2 to retrieve multiple assistant queries and the corresponding context of the previous interaction session in which the multiple assistant queries were received, for example from the assistant activity database 170A shown in Figure 1. In some implementations, the multiple assistant queries and the corresponding context of the previous interaction session in which the multiple assistant queries were received may be limited to those related to the user of the client device (for example, the user of client device 110 in Figure 1). In other implementations, the multiple assistant queries and the corresponding context of the previous interaction session in which the multiple assistant queries were received may be limited to those related to multiple users of each client device (for example, including or not including the user of client device 110 in Figure 1).

[0053] In block 354, the system processes a given assistant query of multiple assistant queries using one or more LLMs to generate one or more corresponding LLM outputs, each of which is expected to respond to the given assistant query. Each of the one or more corresponding LLM outputs may, for example, include a probability distribution over a set of one or more words and / or phrases spanning one or more vocabulary, and one or more of that set of words and / or phrases may be selected as one or more corresponding LLM outputs based on the probability distribution. In various implementations, when generating one or more corresponding LLM outputs for a given assistant query, the system may use one or more LLMs to process, along with the assistant query, the corresponding context of the corresponding previous dialogue session in which the given assistant query was received, and / or a set of assistant outputs expected to respond to the given assistant query (for example, generated based on processing audio data corresponding to the given assistant query using one or more ASR engines 130A1 and / or 130A2, NLU engines 140A1 and / or 140A2, and 1P systems 191 and / or 3P systems 192 as described with respect to Figure 2). In some implementations, the system may process recognized text corresponding to a given assistant query, and in additional or alternative implementations, the system may process audio data capturing an utterance containing the given assistant query. In some implementations, the system can have the LLM engine 150A1 process a given assistant query locally on the user's client device (for example, the user of client device 110 in Figure 1) using one or more LLMs, while in other implementations, the system can have the LLM engine 150A2 process a given assistant query remotely from the user's client device (for example, on a remote server) using one or more LLMs.As described herein, one or more corresponding LLM outputs can reflect a more natural conversational output than a typical assistant output that may be provided by the automated assistant, which allows the automated assistant to lead the conversation session more fluently. Therefore, an assistant output modified based on one or more corresponding LLM outputs is more likely to resonate with the user perceiving the modified assistant output.

[0054] In some implementations, in addition to one or more corresponding LLM outputs, additional assistant queries may be generated using one or more LLM models based on the processing of a given assistant query and / or the corresponding context of the corresponding previous interaction session in which the given assistant query was received. For example, when processing a given assistant query and / or the corresponding context of the corresponding previous interaction session in which the given assistant query was received, one or more LLMs can determine the intent associated with the given assistant query (for example, based on the stream of NLU output 204 generated using NLU engines 140A1 and / or 140A2 in Figure 2). Furthermore, one or more LLMs can identify at least one related intent for the intent associated with a given assistant query (for example, based on the mapping of intent to at least one related intent in a database or memory accessible to the client device 110, and / or based on processing the intent associated with the given assistant query using one or more machine learning (ML) models or heuristically defined rules). Furthermore, one or more LLMs can generate additional assistant queries based on at least one relevant intent. For example, suppose an assistant query indicates that the user has not yet had dinner (a given assistant query, "I'm feeling pretty hungry," received in the evening at the user's physical location, as indicated by the corresponding context of a corresponding previous dialogue session related to the user's intent to eat).In this example, additional assistant queries could be, for example, "What types of cuisine has the user indicated he / she prefers?" (reflecting the user's intent to identify relevant types of cuisine, associated with their intention to eat), "What restaurants nearby are open?" (reflecting the user's intent to search for relevant restaurants, associated with their intention to eat), and / or other additional assistant queries.

[0055] In these implementations, additional assistant outputs may be determined based on the processing of additional assistant queries. In the example above, where the additional assistant query is "what types of cuisine has the user indicated he / she prefers?", user profile data from one or more 1P systems 191 stored locally on client device 110 may be used to determine that the user indicated a preference for Mediterranean and Indian cuisine. Based on the user profile data indicating the user's preference for Mediterranean and Indian cuisine, one or more corresponding LLM outputs may be modified to ask the user whether Mediterranean and / or Indian cuisine sounds good for dinner (for example, "how does Mediterranean cuisine or Indian cuisine sound for dinner?").

[0056] In the example above, where the additional assistant query is "what restaurants nearby are open?", restaurant data from one or more of 1P system 191 and / or 3P system 192 may be used to determine which restaurants near the user's primary residence are open (and optionally limited to restaurants serving Mediterranean and Indian cuisine based on user profile data). Based on the results, one or more corresponding LLM outputs may be modified to provide the user with a list of one or more restaurants open near the user's primary residence (e.g., "Example Mediterranean Restaurant is open until 9:00 PM and Example Indian Restaurant is open until 10:00 PM"). In particular, the additional assistant queries initially generated using LLM (e.g., "what types of cuisine has the user indicated he / she prefers?" and "what types of cuisine has the user indicated he / she prefers?" in the example above) may not be included in one or more corresponding LLM outputs and, consequently, may not be provided for presentation to the user. Rather, additional assistant outputs determined based on additional assistant queries (for example, "how does Mediterranean cuisine or Indian cuisine sound for dinner" and "Example Mediterranean Restaurant is open until 9:00 PM and Example Indian Restaurant is open until 10:00 PM") may be included in one or more corresponding LLM outputs and consequently provided for presentation to the user.

[0057] In additional or alternative implementations, each of one or more corresponding LLM outputs (and optionally, additional assistant outputs determined based on additional assistant queries) may be generated using the corresponding set of parameters from multiple separate sets of one or more parameters of the LLM. Each of the multiple separate sets of parameters may be associated with a separate personality for the automated assistant. Some versions of these implementations may utilize a single LLM to generate one or more corresponding LLM outputs using the corresponding set of parameters for each of the separate personalities, while other versions of these implementations may utilize multiple LLMs to generate one or more corresponding LLM outputs using the corresponding set of parameters for each of the separate personalities. For example, a single LLM could be used to generate a first LLM output using a first set of parameters that reflect a first personality (e.g., the cook's personality in the example above where the given assistant query corresponds to "I'm feeling pretty hungry"), a second LLM output using a second set of parameters that reflect a second personality (e.g., the butler's personality in the example above where the given assistant query corresponds to "I'm feeling pretty hungry"), and the same thing for several other distinct personalities. Furthermore, a first LLM may be used, for example, to generate a first LLM output using a first set of parameters that reflect a first personality (for example, the personality of the cook in the above example where the given assistant query corresponds to "I'm feeling pretty hungry"), and a second LLM may be used to generate a second LLM output using a second set of parameters that reflect a second personality (for example, the personality of the butler in the above example where the given assistant query corresponds to "I'm feeling pretty hungry"), and the same applies to several other separate personalities.Therefore, when the corresponding LLM output is provided for presentation to the user, it can reflect a variety of changing contextual personalities through the prosodic properties of different personalities (e.g., intonation, pitch, tone, pauses, tempo, stress, rhythm, etc. of these different personalities). Additionally or alternatively, the user can specify one or more personalities to be utilized by the automated assistant (e.g., through the settings of the automated assistant application related to the automated assistant described herein) in a consistent manner (e.g., always using the butler personality) and / or in a contextual manner (e.g., using the butler personality in the morning and evening, but a different personality at noon).

[0058] In particular, the responses of these personalities described herein may reflect not only the prosodic nature of different personalities, but also the vocabulary and / or distinct speaking styles of different personalities (e.g., verbose speaking, concise speaking, friendly personality, sarcastic personality, etc.). For example, since the cook personality described above may have a specific cook's vocabulary, the probability distribution over a set of one or more words and / or phrases for one or more corresponding LLM outputs generated using the set of parameters for the cook personality may give more weight to the set of words and / or phrases used by the cook than to other sets of words and / or phrases for other personalities (e.g., the scientist personality, the librarian personality). Therefore, when one or more corresponding LLM outputs are provided for presentation to the user, they can reflect personalities in a variety of changing contexts, not only in terms of the prosodic nature of different personalities, but also in terms of the accurate and realistic vocabulary of different personalities, so that one or more corresponding LLM outputs will resonate more with the user in a variety of contextual scenarios. Moreover, it should be understood that the vocabulary and / or speech patterns of different personalities can be defined with a degree of variability. Continuing with the above example, the chef personality described above may have a unique Mediterranean chef vocabulary when asking about Mediterranean cuisine, based on the additional assistant query relating to Mediterranean cuisine, and may have a unique Indian chef vocabulary when asking about Indian cuisine, based on the additional assistant query relating to Indian cuisine, and so on.

[0059] In block 356, the system indexes one or more corresponding LLM outputs in memory accessible on the client device (e.g., the LLM output database 150A in Figure 1) based on a given assistant query and / or the corresponding context of a corresponding previous dialogue session for the given assistant query. For example, the system can cause the indexing engine 172 of the offline output correction engine in Figures 1 and 2 to index one or more corresponding LLM outputs in the LLM output database 150A. In some implementations, the indexing engine 172 can index one or more corresponding LLM outputs based on one or more terms contained in a given assistant query and / or one or more context signals contained in the corresponding context of a corresponding previous dialogue session in which the given assistant query was received. In additional or alternative implementations, the indexing engine 172 may generate an embedding of a given assistant query (e.g., a word2vec embedding or any other low-dimensional representation) and / or generate an embedding of one or more contextual signals contained in the corresponding context of the corresponding previous interaction session in which the given assistant query was received, one or more of these embeddings may be mapped to an embedding space (e.g., a low-dimensional space). In these implementations, one or more corresponding LLM outputs indexed in the LLM output database 150A may be later used by the online output modification engine 180 to modify a set of assistant outputs (e.g., as described below with respect to blocks 362, 364, and 366 of method 300 in Figure 3).In various implementations, the system may, in addition or alternatively, generate one or more assistant outputs (e.g., one or more assistant outputs 205 as described with respect to Figure 2) that are expected to respond to a given assistant query (but not generated using the LLM engines 150A1 and / or 150A2), and one or more corresponding LLM outputs may, in addition or alternatively, be indexed by one or more assistant outputs.

[0060] In some implementations, as shown in block 358, the system may optionally accept user input to evaluate and / or modify one or more of the corresponding LLM outputs. For example, a human evaluator may analyze one or more corresponding LLM outputs generated using one or more LLM models and modify one or more of the corresponding LLM outputs by changing one or more terms and / or phrases contained in one or more of the corresponding LLM outputs. Alternatively, for example, a human evaluator may re-index, discard, and / or otherwise modify the indices of one or more corresponding LLM outputs. Thus, in these implementations, one or more corresponding LLM outputs generated using one or more LLMs may be selected by a human evaluator to ensure the quality of one or more corresponding LLM outputs. Furthermore, any non-discarded, re-indexed, and / or selected LLM outputs may be used to modify or retrain the LLMs in an offline manner.

[0061] In block 360, the system determines whether there are additional assistant queries included in the multiple assistant queries retrieved in block 352 that have not been processed using one or more LLMs. If, in an iteration of block 360, the system determines that there are additional assistant queries included in the multiple assistant queries retrieved in block 352 that have not been processed using one or more LLMs, the system returns to block 354 and performs additional iterations of blocks 354 and 356 with respect to the additional assistant queries, rather than the given assistant queries. These actions may be repeated for each of the assistant queries included in the multiple assistant queries retrieved in block 352. In other words, the system can index one or more corresponding LLM outputs for each of the corresponding contexts of the corresponding previous interaction session in which the assistant queries, and / or one of the corresponding assistant queries, are received, before those one or more corresponding LLM outputs become available.

[0062] If, in an iteration of block 360, the system determines that there are no additional assistant queries included in the multiple assistant queries obtained in block 352 that have not been processed using one or more LLMs, the system may proceed to block 362. In block 362, the system may monitor a stream of audio data generated by one or more microphones of a client device to determine whether the stream of audio data captures utterances of a user of the client device directed to the automated assistant. For example, the system may monitor one or more specific words or phrases included in the stream of audio data (e.g., one or more specific words or phrases that invoke the automated assistant using a hotword detection model). Alternatively, for example, the system may monitor utterances directed to the client device in addition to one or more other signals of its choice (e.g., one or more gestures captured by the client device's vision sensors, gazes directed to the client device, etc.). If, in an iteration of block 362, the system determines that the stream of audio data did not capture utterances of a user of the client device directed to the automated assistant, the system may continue monitoring the stream of audio data in block 362. In an iteration of block 362, if the system determines that the stream of audio data captures the utterance of a user on a client device directed to the automated assistant, the system may proceed to block 364.

[0063] In block 364, based on its processing of the audio data stream, the system determines that the utterance contains the current assistant query corresponding to one of several assistant queries, and / or that the utterance is received in the current context of the current dialogue session, which corresponds to the corresponding context of the corresponding previous dialogue session for one of the several assistant queries. For example, the system may process the audio data stream (e.g., the audio data stream 201 in Figure 2) using ASR engines 130A1 and / or 130A2 to generate the ASR output stream (e.g., the ASR output stream 203 in Figure 2). Furthermore, the system may process the ASR output stream using NLU engines 140A1 and / or 140A2 to generate the NLU output stream (e.g., the NLU output stream 204 in Figure 2). In addition, based on the ASR output stream and / or the NUL output stream, the system can identify the current assistant query. In some implementations, the system can further determine a set of one or more assistant outputs (for example, one or more assistant outputs 205 in Figure 2) by having one or more 1P systems 191 and / or 3P systems process the stream of ASR outputs and / or the stream of NLU outputs.

[0064] In some implementations of Method 300 in Figure 3, the system can use the online output correction engine 180 to determine that one or more terms or phrases in the current assistant query correspond to one or more terms or phrases in one or more of the LLM outputs that have been indexed for that purpose (using any known technique for determining whether terms or phrases correspond to each other, such as the exact matching technique, the soft matching technique, the edit distance technique, the phonetic similarity technique, or the embedding technique). In response to determining that one or more terms or phrases in the current assistant query correspond to one or more terms or phrases in one or more of the multiple assistant queries, the system can retrieve one or more of the corresponding LLM outputs (for example, from the LLM output database 150A) that have been indexed in an iteration of block 356 and associated with one of the multiple assistant queries. For example, if the current query contains the phrase "I'm hungry", the system can retrieve one or more of the corresponding LLM outputs that are generated in the above example described with respect to a given assistant query. For example, the system may determine that both the current query and the given assistant query, as described above, contain the phrase "I'm hungry" based on comparing the edit distance between the terms of the current query and the terms of a given assistant query. Alternatively, for example, the system may generate an embedding for the current assistant query and map the embedding for the current query to the embedding space described above with respect to block 356. Furthermore, the system may determine that the current assistant query corresponds to the given assistant query based on whether the distance between the generated embedding for the current query and the previously generated embedding for the given assistant query in the embedding space satisfies a distance threshold.

[0065] In an additional or alternative implementation of Method 300 in Figure 3, the system may utilize the online output modification engine 180 to determine that one or more contextual signals detected when the current assistant query is received correspond to one or more corresponding contextual signals when one of a group of assistant queries is received (e.g., received on the same day of the week, received at the same time, received at the same location, the same ambient noise present in the client device environment, received in a particular series of utterances during a dialogue session, etc.). In response to determining that one or more contextual signals associated with the current assistant query correspond to one or more contextual signals of one of the group of assistant queries, the system may retrieve one or more corresponding LLM outputs (e.g., from the LLM output database 150A) that are indexed in an iteration of block 356 and associated with one of the group of assistant queries. For example, if the current query is received in the evening at the user's primary residence, the system may retrieve one or more corresponding LLM outputs generated in the above example described with respect to a given assistant query. For example, the system may determine that both the described current query and a given assistant query are associated with the temporal context signal "evening" and the location context signal "primary residence". Alternatively, the system may generate embeddings for one or more context signals related to the current assistant query and map these embeddings to the current query to the embedding space described above with respect to block 356. Furthermore, the system may determine that one or more context signals related to the current assistant query correspond to one or more context signals related to a given assistant query, based on whether the distance between the generated embeddings for one or more context signals related to the current query and the previously generated embeddings for one or more context signals related to a given assistant query in the embedding space satisfies a threshold distance.

[0066] In particular, the system may leverage either or both the current assistant query and the context of the interaction session in which the current assistant query is received (e.g., one or more contextual signals detected) when determining one or more corresponding LLM outputs to be used when generating one or more current assistant outputs that should be provided for presentation to the user in response to the current assistant query. In various implementations, the system may, as an addition or alternative, leverage one or more assistant outputs generated for the current assistant query when determining one or more corresponding LLM outputs to be used when generating one or more current assistant outputs that should be provided for presentation to the user in response to the current assistant query. For example, the system may, as an addition or alternative, generate one or more assistant outputs (e.g., one or more assistant outputs 205 as described with respect to Figure 2) that are expected to respond to the current assistant query (but are not generated using LLM engines 150A1 and / or 150A2), and determine that one or more of the assistant outputs expected to respond to the current assistant query correspond to one or more previously generated assistant outputs for one of several assistant queries using the various techniques described above.

[0067] In block 366, the system makes the automated assistant available one or more of the corresponding LLM outputs when generating one or more current assistant outputs to be provided for presentation to the user of the client device. For example, the system can rank one or more assistant outputs (e.g., one or more assistant outputs 205 as described with respect to Figure 2) and one or more corresponding LLM outputs (e.g., one or more modified assistant outputs 206 as described with respect to Figure 2) according to one or more ranking criteria. Furthermore, the system can select one or more current assistant outputs from the one or more assistant outputs and one or more corresponding LLM outputs. In addition, the system can make one or more current assistant outputs rendered visually and / or audibly for presentation to the user of the client device.

[0068] The implementation of Method 300 in Figure 3 describes generating one or more corresponding LLM outputs offline (for example, by using the offline output correction engine 170 to generate corresponding LLM outputs using one or more LLMs and indexing one or more corresponding LLM outputs in the LLM output database 150A), and then utilizing one or more corresponding LLM outputs online (for example, by using the online output correction engine 180 to determine which LLM outputs from the LLM output database 150A to use based on the current assistant query), but it should be understood that this is illustrative and not intended to be limiting. For example, as described below with respect to Figures 4 and 5, the online output correction engine 180 may, in addition or alternative to a preferred implementation, utilize LLM engines 150A1 and / or 150A2 online.

[0069] Figure 4 shows a flowchart illustrating an exemplary method 400 of utilizing a large language model when generating assistant output based on the generation of an assistant query. For convenience, the operation of method 400 is described by referring to a system that performs the operations from process flow 200 in Figure 2. This system of method 400 includes one or more processors, memory, and / or other components of a computing device (e.g., client device 110 in Figure 1, client device 610 in Figure 6, and / or computing device 710 in Figure 7, one or more servers, and / or other computing devices). Furthermore, the operations of method 400 are shown in a particular order, but this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0070] In block 452, the system receives a stream of audio data capturing user utterances, which include assistant queries directed to the automated assistant, and the utterances are received during a dialogue session between the user and the automated assistant. In some implementations, the system may simply process the stream of audio data to determine whether to capture an assistant query in response to determining that one or more conditions are met. For example, the system may monitor one or more specific words or phrases contained in the stream of audio data (e.g., one or more specific words or phrases that invoke the automated assistant using a hotword detection model). Alternatively, for example, the system may monitor utterances directed to the client device in addition to one or more other signals of its choice (e.g., one or more gestures captured by the client device's vision sensors, gazes directed to the client device, etc.).

[0071] In block 454, the system determines a set of assistant outputs based on its processing of the audio data stream, with each assistant output in the set responding to an assistant query contained in the utterance. For example, the system can process the audio data stream (e.g., the stream of audio data 201) using ASR engines 130A1 and / or 130A2 to generate a stream of ASR outputs (e.g., ASR output 203). Furthermore, the system can process the ASR output stream (e.g., the stream of audio data 201) using NLU engines 140A1 and / or 140A2 to generate a stream of NLU outputs (e.g., the stream of NLU output 204). In addition, the system can cause one or more 1P systems 191 and / or 3P systems 192 to process the NLU output stream (e.g., the stream of NLU output 204) to generate a set of assistant outputs (e.g., a set of assistant outputs) 205. In particular, the set of assistant outputs may correspond to one or more candidate assistant outputs that the automated assistant may consider using when responding to an utterance, without the use of the techniques described herein (i.e., techniques that do not utilize the LLM engines 150A1 and / or 150A2 when modifying assistant outputs as described herein).

[0072] In block 456, the system processes a set of assistant outputs and the context of the dialogue session to (1) generate a set of modified assistant outputs using one or more LLM outputs, each of which is determined based on the context of the dialogue session and / or one or more assistant outputs included in the set of assistant outputs, and (2) generate additional assistant queries related to the utterance, based at least in part on the context of the dialogue session and at least in part on the assistant queries included in the utterance. In various implementations, each of the LLM outputs may be determined further based on the assistant queries included in the utterance captured in the stream of audio data. In some implementations, when generating the set of modified assistant outputs, one or more LLM outputs may be generated offline in advance (for example, using the offline output modification engine 170 before receiving the utterance, as described above with respect to Figure 3). In these implementations, the system can determine that an assistant query contained in an utterance captured in the audio data stream corresponds to one or more previous assistant queries for which one or more LLM outputs have been previously generated; that the context of the dialogue session corresponds to the previous context of a previous dialogue session in which the previous assistant query was received; and / or that one or more assistant outputs included in the set of assistant outputs determined in block 454 correspond to one or more previous assistant outputs determined based on the previous query. Furthermore, as described with respect to method 300 in Figure 3, the system can retrieve one or more LLM outputs that are indexed based on the previous assistant query, the previous context, and / or one or more previous assistant outputs corresponding to the assistant query, as described, and utilize one or more LLM outputs as a set of modified assistant outputs.

[0073] In additional or alternative implementations, when generating a set of modified assistant outputs, the system may process at least one or more assistant outputs included in the dialogue session context and / or set of assistant outputs online (for example, using the online output modification engine 180 in response to receiving an utterance) in order to generate one or more LLM outputs. For example, the system may cause the LLM engines 150A1 and / or 150A2 to process one or more assistant outputs included in the dialogue session context, assistant queries, and / or set of assistant outputs using one or more LLMs in order to generate a set of modified assistant outputs. One or more LLM outputs may be generated online in the same or similar manner as described above with respect to block 354 of method 300 in Figure 3 with respect to generating one or more LLM outputs offline, but in response to receiving an utterance on the client device. For example, the system may process one or more assistant outputs in a set of assistant outputs using one or more LLMs to generate one or more personality responses for each of the one or more assistant outputs in the set of assistant outputs, as described above with respect to block 354 of method 300 in Figure 3. In other words, each of the assistant outputs in a set of assistant outputs may have a limited vocabulary and a consistent personality with respect to the prosodic properties associated with each of the assistant outputs. However, when processing each of the assistant outputs in a set of assistant outputs to generate a set of modified assistant outputs, each of the modified assistant outputs may have a much larger vocabulary depending on the use of one or more LLMs in generating one or more modified assistant outputs, in which case the variation in the prosodic properties associated with each of the modified assistant outputs is much greater.As a result, each of the modified assistant outputs may correspond to contextually relevant assistant outputs that resonate more with the user involved in the interaction session with the automated assistant.

[0074] Similarly, in some implementations, when generating additional assistant queries, additional assistants may be generated offline in advance (for example, using the offline output correction engine 170 before receiving an utterance, as described above with respect to Figure 3). In these implementations, as described above with respect to obtaining one or more LLM outputs generated offline in advance, the system may obtain additional assistant queries that are indexed based on a previous assistant query, a previous context, and / or one or more previous assistant outputs corresponding to an assistant query, each corresponding to one or more assistant outputs included in a set of assistant queries, contexts, and / or assistant outputs, as described with respect to method 300 in Figure 3, and the previously generated additional assistant queries can be used as additional assistant queries.

[0075] Similarly, in additional or alternative implementations, when generating additional assistant queries, the system may process one or more assistant outputs included in at least the context of the dialogue session and / or the set of assistant outputs online (for example, using the online output modification engine 180 in response to the receipt of an utterance) in order to generate additional assistant queries. For example, the system may cause the LLM engines 150A1 and / or 150A2 to process one or more assistant outputs included in the context of the dialogue session, the assistant queries, and / or the set of assistant outputs, using one or more LLMs, in order to generate additional assistant queries. Additional assistant queries may be generated online in the same or similar manner as described above with respect to block 354 of method 300 in Figure 3 with respect to generating additional assistant queries offline, but in response to the receipt of an utterance on the client device. In some implementations, one or more LLMs described herein may have multiple separate layers dedicated to performing certain functions. For example, one or more first layers of one or more LLMs may be used to generate the personality responses described herein, and one or more second layers of one or more LLMs may be used to generate the additional assistant queries described herein. In additional or alternative implementations, one or more LLMs may communicate with one or more additional layers not included in one or more LLMs when generating the additional assistant queries described herein. For example, one or more layers of one or more LLMs may be used to generate the personality responses described herein, and one or more additional layers of another ML model communicating with one or more LLMs may be used to generate the additional assistant queries described herein. Non-limiting examples of personality responses and additional assistant queries are described in more detail below with reference to Figure 6.

[0076] In block 458, the system determines additional assistant outputs to respond to additional assistant queries based on those additional assistant queries. In some implementations, the system may ensure that additional assistant queries are processed by one or more of the 1P system 191 and / or 3P system 192 in the same or similar manner as described with respect to processing assistant queries in Figure 2, in order to generate additional assistant outputs. In some implementations, the additional assistant outputs may be a single additional assistant output, while in other implementations, the additional assistant outputs may be included in a set of additional assistant outputs (for example, the set of additional assistant outputs 205 in Figure 2). In additional or alternative implementations, additional assistant queries may be directly mapped to additional assistant outputs based on user profile data of the user who provided, for example, utterances and / or any other data accessible to the automated assistant.

[0077] In block 460, the system processes a set of modified assistant outputs based on additional assistant outputs responding to additional assistant queries in order to generate an additional set of modified assistant outputs. In some implementations, the system may add additional assistant outputs to the beginning or end of one or more modified assistant outputs for each of the one or more assistant outputs included in the set of modified assistant outputs generated in block 456. In additional or alternative implementations, as shown in block 460A, the system may process additional assistant outputs and the context of the interaction session in order to generate an additional set of modified assistant outputs, using the LLM outputs used in block 456 and / or, in addition to the one or more LLM outputs used in block 456, one or more additional LLM outputs generated based on at least some of the additional assistant outputs based on at least some of the context of the interaction session. An additional set of modified assistant outputs may be generated in the same or similar manner as described above with respect to generating a set of modified assistant outputs, but based on additional assistant outputs rather than a set of assistant outputs (for example, using one or more LLM outputs generated in an offline manner, and / or using LLM engines 150A1 and / or 150A2 in an online manner).

[0078] In block 462, the system ensures that a given modified assistant output from a set of modified assistant outputs, and / or a given additional modified assistant output from a set of additional modified assistant outputs, are provided for presentation to the user. In some implementations, the system may cause the ranking engine 190 to rank each of the one or more modified assistant outputs included in the set of modified assistant outputs (and optionally, each of the one or more assistant outputs included in the set of assistant outputs) according to one or more ranking criteria, and to select a given modified assistant output from the set of modified assistant outputs (or a given assistant output from the set of assistant outputs). Furthermore, the system may cause the ranking engine 190 to rank each of the one or more additional modified assistant outputs included in the set of additional modified assistant outputs (and optionally, an additional assistant output) according to one or more ranking criteria, and to select a given additional modified assistant output from the set of additional modified assistant outputs (or an additional assistant output as a given additional assistant output). In these implementations, the system can combine a given modified assistant output with a given additional assistant output so that the given modified assistant output and the given additional assistant output are provided for visual and / or audible presentation to the user of the client device involved in the interaction session with the automated assistant.

[0079] Figure 5 shows a flowchart illustrating an exemplary method 500 for utilizing a large language model when generating assistant output based on the generation of assistant personality responses. For convenience, the operation of method 500 is described by referring to a system that performs the operations from process flow 200 in Figure 2. This system of method 500 includes one or more processors, memory, and / or other components of a computing device (e.g., client device 110 in Figure 1, client device 610 in Figure 6, and / or computing device 710 in Figure 7, one or more servers, and / or other computing devices). Furthermore, the operations of method 500 are shown in a particular order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0080] In block 552, the system receives a stream of audio data capturing user utterances, which include assistant queries directed to the automated assistant, and the utterances are received during a dialogue session between the user and the automated assistant. In block 554, based on its processing of the audio data stream, the system determines a set of assistant outputs, each of which responds to an assistant query contained in the utterance. The operations of blocks 552 and 554 of method 500 in Figure 5 can be performed in the same or similar manner as described with respect to blocks 452 and 454 of method 400 in Figure 4, respectively.

[0081] In block 556, the system decides whether to modify one or more assistant outputs included in a set of assistant outputs. The system may decide whether to modify one or more assistant outputs based, for example, the user's intent when providing an utterance (e.g., included in the stream of NLU output 204), one or more assistant outputs included in a set of assistant outputs (e.g., a set of assistant outputs 205), one or more computational costs associated with modifying one or more of the assistant outputs included in the set of assistant outputs (e.g., battery consumption, processor consumption, latency, etc.), the length of time spent interacting with the automated assistant, and / or other considerations. For example, if the user's intent indicates that the user providing the utterance expects a quick and / or factual answer (e.g., "why is the sky blue?", "what's the weather?", "what time is it?"), in some cases the system may decide not to modify one or more assistant outputs in order to reduce the latency and consumption of computational resources when providing content that responds to the utterance. Furthermore, for example, if the user's client device is in power-saving mode, the system may decide not to modify one or more of the assistant outputs in order to conserve battery power. Also, for example, if the user is engaged in an interaction with a session duration exceeding a threshold (e.g., 30 seconds, 1 minute, etc.), the system may decide not to modify one or more of the assistant outputs in order to terminate the interaction session in a faster and more efficient manner.

[0082] If, in an iteration of block 556, the system decides not to modify one or more of the assistant outputs included in the set of assistant outputs, the system may proceed to block 558. In block 558, the system ensures that a given assistant output from the set of assistant outputs is provided for presentation to the user. For example, the system may cause the ranking engine 190 to rank each of the assistant outputs included in the set of assistant outputs according to one or more ranking criteria, and based on the ranking, select a given assistant output that should be provided for visual and / or audible presentation to the user.

[0083] If, in an iteration of block 556, the system decides to modify one or more assistant outputs included in the set of assistant outputs, the system may proceed to block 560. In block 560, the system processes the set of assistant outputs and the context of the dialogue session to generate a set of modified assistant outputs using one or more LLM outputs, each of which is determined based on the context of the dialogue session and / or one or more assistant outputs included in the set of assistant outputs, and each of which reflects a corresponding personality of the automated assistant from among several distinct personalities. As described with respect to block 354 of method 300 in Figure 3, one or more LLM outputs may be generated using a variety of distinct parameters to reflect different personalities of the automated assistant. In various implementations, each of the LLM outputs may be determined further based on assistant queries included in utterances captured in the stream of audio data. In some implementations, the system may process a set of assistant outputs and the context of the interaction session (and optionally the assistant queries) to generate a set of assistant outputs modified using one or more LLM outputs pre-generated offline as described herein; however, in additional or alternative implementations, the system may process a set of assistant outputs and the context of the interaction session (and optionally the assistant queries) to generate a set of assistant outputs modified online as described herein. Non-limiting examples of personality responses and additional assistant queries, as described above with respect to Method 400 in Figure 4, are described in more detail below with respect to Figure 6.

[0084] In block 562, the system ensures that a given assistant output from a set of assistant outputs is provided for presentation to the user. In some implementations, the system may cause the ranking engine 190 to rank each of the one or more modified assistant outputs (and optionally each of the one or more assistant outputs (and optionally) from a set of assistant outputs) that are included in a set of modified assistant outputs according to one or more ranking criteria, and to select a given modified assistant output (or a given assistant output from a set of assistant outputs) from the set of modified assistant outputs. Furthermore, the system may ensure that the given modified assistant output is provided for visual and / or audible presentation to the user.

[0085] Figure 5 does not illustrate the generation of additional assistant queries, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. Rather, it should be understood that generating additional assistant queries in method 400 of Figure 4 is based on determining that there are additional contextually relevant assistant outputs that can be provided to facilitate the dialogue session in relation to providing a more natural conversational experience for the user, thereby making any assistant outputs provided for presentation to the user more person-to-person dialogue session-like and more resonant with the user. Furthermore, Figures 3 and 4 do not illustrate the decision of whether to modify the set of assistant outputs, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. Rather, it should be understood that deciding whether to trigger a modification of the set of assistant responses using one or more LLMs can be performed in any of the illustrative methods in Figures 3, 4, and 5.

[0086] Figure 6 shows a non-restrictive example of a user-automated assistant interaction session in which the automated assistant utilizes one or more LLMs when generating assistant outputs. As described herein, in some implementations, the automated assistant can utilize one or more LLM outputs pre-generated offline to generate a set of modified assistant outputs (for example, as described above with respect to method 300 in Figure 3). For example, the automated assistant can determine that a previous assistant query for which one or more LLM outputs have been pre-generated corresponds to an assistant query contained in the utterance, a previous context of a previous interaction session corresponds to the context of the user-automated assistant interaction session in which the utterance is received, and / or that one or more previous assistant outputs correspond to one or more assistant outputs contained in a set of assistant outputs for an assistant query contained in the utterance. Furthermore, the automated assistant can retrieve one or more LLM outputs that are indexed (for example in the LLM output database 150A) according to one or more of the previous assistant queries, the previous context, and / or previous assistant outputs contained in the previous set of assistant outputs, and utilize one or more LLM outputs as a set of modified assistant outputs. In additional or alternative implementations, the automated assistant may have the assistant query, the context of the interaction session, and / or one or more assistant outputs included in the set of assistant outputs processed using one or more LLMs to generate one or more LLM outputs to be used as a set of modified assistant outputs in an online manner (for example, as described above with respect to Method 400 in Figure 4 and Method 500 in Figure 5). Thus, the non-limiting example in Figure 6 is given to show how the use of LLMs by the techniques described herein results in improved, more natural conversation between the user and the automated assistant.

[0087] The client device 610 (for example, the client device 110 example in Figure 1) may include various user interface components, such as a microphone for generating audio data based on speech and / or other audible input, a speaker for rendering synthesized speech and / or other audible output audibly, and / or a display 680 for visually rendering visual output. Furthermore, the display 680 of the client device 610 may include various system interface elements 681, 682, and 683 (for example, hardware and / or software interface elements) with which the user of the client device 610 can interact in order to cause the client device 610 to perform one or more actions. The display 680 of the client device 610 allows the user to interact with the content rendered on the display 680 by touch input (for example, by directing user input to the display 680 or a part thereof (for example, a text input box (not shown), a keyboard (not shown), or another part of the display 680)) and / or spoken input (for example, by selecting the microphone interface element 684, or simply by speaking without necessarily selecting the microphone interface element 684 on the client device 610 (i.e., the automated assistant may use one or more terms or phrases, gestures, gazes, mouth movements, lip movements, and / or other conditions to enable spoken input)). The client device 610 shown in Figure 6 is a mobile phone, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, the client device 610 may be a standalone speaker with a display, a standalone speaker without a display, a home automation device, an in-vehicle system, a laptop, a desktop computer, and / or any other device capable of running an automated assistant to engage in a human-to-computer interaction session with the user of the client device 610.

[0088] For example, suppose a user of client device 610 provides the utterance 652, "Hey Assistant, what time is it?". In this example, the automated assistant may use ASR engines 130A1 and / or 130A2 to process the audio data capturing the utterance 652 in order to generate a stream of ASR output. Furthermore, the automated assistant may use NLU engines 140A1 and / or 140A2 to process the stream of ASR output in order to generate a stream of NLU output. In addition, the automated assistant may use one or more of the 1P system 191 and / or 3P system 192 to process the stream of NLU output in order to generate one or more sets of assistant outputs. The set of assistant outputs may include, for example, "8:30 AM", "Good morning, it's 8:30 AM", and / or any other output that tells the user of client device 610 the current time.

[0089] In the example in Figure 6, the automated assistant decides to modify one or more of the assistant outputs included in the set of assistant outputs in order to generate a modified set of assistant outputs. For example, the automated assistant may decide that the assistant query contained in utterance 652 requests the automated assistant to provide the current time for presentation to the user. The automated assistant may decide that an example of a previous assistant query for which one or more LLM outputs have been pre-generated requests the automated assistant to provide the current time for presentation to the user corresponds to the assistant query contained in utterance 652 in Figure 6, the previous context of a previous dialogue session corresponds to the context of the dialogue session between the user and the automated assistant in Figure 6 (for example, the user requests the automated assistant to provide the current time in the morning (and optionally, making the request by initiating a dialogue session), the client device 610 is located in a specific location, and / or other contextual signals), and / or that one or more previous assistant outputs correspond to one or more of the assistant outputs included in the set of assistant outputs for the assistant query contained in utterance 652 in Figure 6. Furthermore, the automated assistant can retrieve one or more LLM outputs that are indexed (for example, in the LLM output database 150A) according to one or more previous assistant outputs included in a previous set of assistant outputs, a previous assistant query, a previous context, and / or an earlier set of assistant outputs, and utilize one or more LLM outputs as a modified set of assistant outputs. Alternatively, for example, the automated assistant can ensure that one or more LLM outputs are processed using one or more LLMs in order to generate one or more LLM outputs that should be used as a modified set of assistant outputs in an online manner.

[0090] In the example in Figure 6, the automated assistant decides to provide a modified assistant output 654, “Good morning [User]! It's 8:30 AM. Any fun plans today?” for presentation to the user, and further assumes that the modified assistant output 654 is determined based on one or more LLM outputs. The modified assistant output 654 provided for presentation to the user is personalized or adapted to the context of the user and interaction session of client device 610 in that the modified assistant output 654 greets the user with a contextually appropriate greeting (e.g., “Good Morning”) and refers to the user of client device 610 by name (e.g., “[User]”). In particular, one or more LLM outputs may include one or more corresponding substitutes (e.g., as indicated by “[User]!” in the modified assistant output 654) that can be populated with user profile data accessible to the automated assistant. The example in Figure 6 includes corresponding substitutes for the name of the user of client device 610, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, one or more of the corresponding proxies may be populated with any data accessible to the automated assistant, such as a smart network connected device identifier (e.g., smart lighting, smart TV, smart appliance, smart speaker, smart door lock, etc.), a known location associated with the user of the client device 610 (e.g., a city, state, county, region, country, workplace, the user's office, or the physical address of the user's primary residence on the client device 610), entity criteria (e.g., criteria people, places, things, etc.), software applications accessible on the user's client device 610, and / or any other data accessible to the automated assistant.

[0091] Furthermore, the modified assistant output 654 functions in relation to responding to the assistant query contained in utterance 652 (e.g., "It's 8:30 AM"). However, the modified assistant output 654 is not only personalized or adapted to the user and functions in relation to responding to the assistant query, but also helps to drive the conversational session between the user and the automated assistant by further engaging with the user in the conversational session (e.g., "Any fun plans today?"). Without using the techniques described herein regarding the use of one or more LLM outputs when modifying the original generated set of assistant outputs based on the processing of utterance 652, the automated assistant may simply respond "It's 8:30 AM" without greeting the user of client device 610 (e.g., "Good morning"), without addressing the user of client device 610 by name (e.g., "[User]"), and without further engaging with the user of client device 610 in the conversational session (e.g., "Any fun plans today?"). Therefore, the modified assistant output 654 may resonate more with the user of the client device 610 than any of the assistant outputs included in the original generated set of assistant outputs that do not utilize one or more LLM outputs.

[0092] In the example in Figure 6, we further assume that the user of client device 610 provides the utterance 656, “Yes, I'm thinking about going to the beach.” In this example, the automated assistant may be processed to generate a set of assistant outputs that would otherwise be generated without using one or more LLM outputs. Furthermore, the automated assistant may be processed to generate a modified set of assistant outputs that would otherwise be generated using one or more LLM outputs (for example, in an offline and / or online manner), and optionally to generate additional assistant queries based on the assistant queries.

[0093] In this example, the assistant outputs included in the set of assistant outputs (i.e., those generated without using one or more LLM outputs) may be limited because the assistant query in utterance 656 does not require the automated assistant to take any action. For example, the assistant outputs included in the set of assistant outputs may include "Sounds fun!", "Surf's up!", "That sounds like fun!", and / or other assistant outputs that respond to utterance 656 but do not further engage the user of client device 610 in the dialogue session. In other words, the vocabulary of the assistant outputs included in the set of assistant outputs may be limited because they are not generated using one or more LLM outputs as described herein. Nevertheless, the automated assistant can leverage the assistant outputs included in the set of assistant outputs to determine how to modify one or more of the assistant outputs using one or more LLM outputs.

[0094] Furthermore, as illustrated with respect to Figures 3 and 4, the automated assistant can generate additional assistant queries based on the assistant query and using one or more LLMs or separate ML models communicating with one or more LLMs. For example, in the example in Figure 6, utterance 656 provided by the user of client device 610 indicates that the user plans to go to the beach. Based on identifying that the intent associated with utterance 656 indicates that the user plans to go to the beach, the automated assistant can determine an associated intent, which is associated with checking the weather at a beach frequently visited by the user of client device 610 (for example, an exemplary beach named "Half Moon Bay"). Based on identifying the associated intent, the automated assistant can generate an additional assistant query, "What's the weather?", along with the location parameter "Half Moon Bay", and query one or more of the 1P system 191 and / or 3P system 192 to obtain additional assistant output, for example, "weather" for "Half Moon Bay", indicating that it will rain all day and the temperature will be low at "Half Moon Bay". In some implementations, the automated assistant may process additional assistant outputs and / or the context of the interaction session to generate a set of additional modified assistant outputs determined using one or more LLM outputs and / or one or more additional LLM outputs.

[0095] In the example in Figure 6, the automated assistant can arrange for the set of assistant outputs and the set of modified assistant outputs to be ranked according to one or more ranking criteria, and can select one or more assistant outputs based on the ranking (for example, selecting a given assistant output, "Sounds fun!"). Furthermore, the automated assistant can arrange for the set of additional modified assistant outputs and additional assistant outputs to be ranked according to one or more ranking criteria, and can select one or more assistant outputs based on the ranking (for example, selecting a given additional assistant output, "But if you're going to Half Moon Bay again, expect rain and chilly temps"). Moreover, the automated assistant can combine the selected given assistant output and the selected given additional assistant output to produce a modified assistant output 658, "Sounds fun! But if you're going to Half Moon Bay again, expect rain and chilly temps", so that the modified assistant output 658 is provided for visual and / or audible presentation to the user of the client device 610. Therefore, in this example, the automated assistant can process utterance 656 and provide additional contextual information related to the utterance (for example, the weather at a beach that the user of client device 610 is likely to visit) in order to further engage with the user of client device 610 in the dialogue session.Without the techniques described herein, users of client device 610 may be asked to proactively request weather information from the automated assistant, even though the automated assistant can determine and provide the weather information. This increases the amount of user input, wastes computing resources in client device 610 in processing this increased user input, and increases the cognitive load on the user of client device 610.

[0096] In the example in Figure 6, we further assume that the user of client device 610 provides the utterance 660, "Oh no... thanks for the heads up, can you remind me to check the weather again in two hours?". In this example, the automated assistant may be processed to generate a set of assistant outputs that would otherwise be generated without using one or more LLM outputs. Furthermore, the automated assistant may be processed to generate a modified set of assistant outputs that would otherwise be generated using one or more LLM outputs (for example, in an offline and / or online manner). Based on having processed utterance 660, the automated assistant may decide to set a reminder for 10:30 AM (for example, two hours after the conversation session) to remind the user of client device 610 to check the weather for "Half Moon Bay", or to proactively provide the user of client device 610 with the weather for "Half Moon Bay" at 10:30 AM. Furthermore, the automated assistant may provide a modified assistant output 662, “Sure thing, I set the reminder and hope the weather clears up for you,” from a set of modified assistant outputs (i.e., generated using one or more LLM outputs), for visual and / or audible presentation to the user of the client device 610. In particular, the modified assistant output 662 in the example of Figure 6 is contextual with respect to the interaction session in that it indicates to the user that it hopes the weather will clear up. In contrast, an assistant output included in a set of assistant outputs (i.e., generated without using one or more LLM outputs) may simply provide an indication that a reminder has been set, without considering the context of the interaction session.

[0097] Figure 6 illustrates the use of one or more LLM outputs to generate specific modified assistant outputs based on the context of a particular utterance and dialogue session, and to select specific modified assistant outputs to be provided for presentation to the user, but it should be understood that this is illustrative and not intended to be limiting. Rather, the techniques described herein can be used for any dialogue session between any user and corresponding examples of an automated assistant. Furthermore, it should be understood that the transcript corresponding to the dialogue session between the user and the automated assistant is shown on the display 680 of the client device 610, but this is illustrative and not intended to be limiting. For example, it should be understood that the dialogue session can be performed on any device capable of running an automated assistant, regardless of whether the client device includes a display.

[0098] Furthermore, it should be understood that the assistant output provided for presentation to the user in the dialogue session of Figure 6 may include various personality responses as described herein. For example, modified assistant output 654 may be generated using a first set of parameters that reflect a first personality of the automated assistant with respect to a first vocabulary to be used by the automated assistant and / or a first set of prosodic properties to be used when providing modified assistant output 654 for audible presentation to the user. In addition, modified assistant output 658 may be generated using a second set of parameters that reflect a second personality of the automated assistant with respect to a second vocabulary to be used by the automated assistant and / or a second set of prosodic properties to be used when providing modified assistant output 658 for audible presentation to the user. In this example, the first personality may reflect, for example, the personality of a butler or maid used to provide a morning greeting, respond to a user requesting the current time, and ask the user if they have any appointments for the day. Furthermore, the second personality could reflect the personalities of a weather forecaster, surfer, or lifeguard (or a combination thereof), for example, to indicate that going to the beach sounds fun, but that the weather on that day might not be ideal for the beach. For example, since the user has indicated that they plan to go to the beach, the surfer personality could be used to provide the “Sounds fun!” portion of the modified assistant output 658, and since the automated assistant is providing the user with weather information, the weather forecaster personality could be used to provide “But if you're going to Half Moon Bay again, expect rain and chilly temps.”

[0099] Therefore, the automated assistant can dynamically adapt the personality used when providing the modified assistant output for presentation to the user, based on both the vocabulary used by the automated assistant and the prosodic properties used when rendering the modified assistant output for audible presentation to the user. In particular, the automated assistant can dynamically adapt these personalities used when providing the modified assistant output based on the context of the dialogue session, including previous utterances received from the user, as well as previous assistant outputs provided by the automated assistant and / or any other contextual signals described herein. As a result, the modified assistant output provided by the automated assistant may resonate more with the user of the client device.

[0100] Furthermore, while Figure 6 is illustrated herein in relation to a user providing utterances throughout a dialogue session, it should be understood that this is for illustrative purposes only and not intended to be limiting. For example, the user may, in addition or alternatively, provide typed input and / or touch input throughout the dialogue session. In these implementations, the automated assistant may process the typed input (e.g., using NLU engines 140A1 and / or 140A2) to generate a stream of NLU output (e.g., and optionally, skip any processing using ASR engines 130A1 and / or 130A2), and in the same or similar manner as described above, process the stream of NLU data and text inputs corresponding to assistant queries derived from the typed input and / or touch input (e.g., using LLM engines 150A1 and / or 150A2) when generating one or more sets of modified assistant outputs.

[0101] Figure 7 shows a block diagram of an exemplary computing device 710 that may be optionally used to perform one or more embodiments of the techniques described herein. In some implementations, one or more client devices, cloud-based automated assistant components, and / or other components may comprise one or more components of the exemplary computing device 710.

[0102] The computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices enable user interaction with the computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to a corresponding interface device in another computing device.

[0103] The user interface input device 722 may include a keyboard, and pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into a display, a speech recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 710 or into a communication network.

[0104] The user interface output device 720 may include non-visual displays such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include flat panel devices such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs), projection devices, or several other mechanisms for creating visible images. The display subsystem may also provide non-visual displays, such as through an audio output device. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 710 to the user or to another machine or computing device.

[0105] The storage subsystem 724 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 724 may include logic for performing selected embodiments of the methods disclosed herein and for implementing the various components shown in Figures 1 and 2.

[0106] These software modules are generally executed by processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 may include several types of memory, including main random access memory (RAM) 730 for storing instructions and data during program execution, and read-only memory (ROM) 732 for storing fixed instructions. The file storage subsystem 726 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing functions in several implementation forms may be stored by the file storage subsystem 726 in the storage subsystem 724 or in other machines accessible by processor 714.

[0107] The bus subsystem 712 provides a mechanism for various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative implementations of the bus subsystem 712 may utilize multiple buses.

[0108] The computing device 710 may be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing systems or computing devices. Due to the ever-changing nature of computers and networks, the description of the computing device 710 shown in Figure 7 is intended only as a concrete example to illustrate several implementation forms. Numerous other configurations of the computing device 710 are possible, having more or fewer components than the computing device shown in Figure 7.

[0109] Where the systems described herein may collect or otherwise monitor personal information about a user, or may utilize personal and / or monitored information, the user may be given the opportunity to control whether the program or features collect user information (e.g., the user's social networks, social behavior or activities, occupation, user preferences, or the user's current geographical location), or whether and / or how content is received from content servers to which the user may be more relevant. In addition, some data may be handled in one or more ways before being stored or used so that personally identifiable information is removed. For example, user identification information may be handled in such a way that personally identifiable information about the user cannot be determined, or the user's geographical location may be generalized, in which case the geographical location information is obtained in such a way that the user's specific geographical location cannot be determined (e.g., down to the city, zip code, or state level). Thus, the user can control how information is collected about and / or used about them.

[0110] In some implementations, a method performed by one or more processors is provided as part of a dialogue session between a user of a client device and an automated assistant performed by the client device, and includes the steps of: receiving a stream of audio data capturing the user's utterance, the stream of audio data being generated by one or more microphones generated by the client device, and the utterance including an assistant query; determining a set of assistant outputs based on having processed the stream of audio data, each assistant output in the set of assistant outputs responding to an assistant query contained in the utterance; and generating a set of modified assistant outputs using one or more LLM outputs generated using a Large-Scale Language Model (LLM), each of which is determined based on at least a portion of the context of the dialogue session and one or more assistant outputs included in the set of assistant outputs. The process includes: processing a set of assistant outputs and the context of a dialogue session for generating additional assistant queries related to an utterance, based on at least a portion of the context of the dialogue session and based on at least a portion of the assistant queries; determining additional assistant outputs to respond to the additional assistant queries based on the additional assistant queries; processing additional assistant outputs and the context of a dialogue session for generating a set of additional modified assistant outputs using one or more LLM outputs, or one or more additional LLM outputs generated using LLM, wherein each of the additional LLM outputs is determined based on the context of the dialogue session and at least a portion of the additional assistant outputs; and ensuring that a given modified assistant output from the set of modified assistant outputs and a given additional modified assistant output from the set of additional modified assistant outputs are provided for presentation to the user.

[0111] These and other implementations of the technology disclosed herein may optionally include one or more of the following features:

[0112] In some implementations, determining assistant outputs to respond to assistant queries contained in an utterance based on processing a stream of audio data may include the steps of processing a stream of audio data using an ASR model to generate a stream of automatic speech recognition (ASR) outputs, processing a stream of ASR outputs using an NLU model to generate a stream of natural language understanding (NLU) data, and ensuring that the set of assistant outputs is determined based on the NLU stream.

[0113] In some versions of these implementations, processing a set of assistant outputs and the context of an interaction session to generate a set of modified assistant outputs using one or more LLM outputs generated using LLM may include processing a set of assistant outputs and the context of an interaction session using LLM to generate one or more LLM outputs, and determining a set of modified assistant outputs based on one or more LLM outputs. In some further versions of these implementations, processing a set of assistant outputs and the context of an interaction session to generate one or more LLM outputs using LLM may include processing a set of assistant outputs and the context of an interaction session using a first set of LLM parameters from multiple separate sets of LLM parameters to determine one or more LLM outputs having a first personality from multiple separate personalities. The set of modified assistant outputs may include one or more first personality assistant outputs that reflect the first personality. Further versions of these implementations may include processing the set of assistant outputs and the context of the interaction session using LLM to generate one or more LLM outputs, using a second set of LLM parameters from a multiple separate set of LLM parameters to determine one or more LLM outputs having a second personality from a multiple separate personality. The modified set of assistant outputs may include one or more second personality assistant outputs that reflect a second personality, the second personality being distinct from the first personality.In yet another version of these implementations, one or more first personality assistant outputs, included in the set of modified assistant outputs and reflecting a first personality, may be determined using a first vocabulary associated with the first personality, and one or more second personality assistant outputs, included in the set of modified assistant outputs and reflecting a second personality, may be determined using a second vocabulary associated with the second personality, the second personality being distinct from the first personality on the basis that the second vocabulary is distinct from the first vocabulary. In further additional or alternative versions of those implementations, one or more first personality assistant outputs, included in the set of modified assistant outputs and reflecting a first personality, may be associated with a first set of prosodic properties used when providing a given modified assistant output for audible presentation to the user, and one or more second personality assistant outputs, included in the set of modified assistant outputs and reflecting a second personality, may be associated with a second set of prosodic properties used when providing a given modified assistant output for audible presentation to the user, and the second personality may be distinct from the first personality on the basis that the second set of prosodic properties is distinct from the first set of prosodic properties.

[0114] In some versions of these implementations, processing a set of assistant outputs and the context of an interaction session to generate a set of assistant outputs modified using one or more LLM outputs generated using LLM may include identifying one or more previously generated LLM outputs based on the fact that one or more LLM outputs were previously generated based on a previous assistant query in a previous interaction session corresponding to the assistant query of the interaction session, and / or based on the fact that one or more LLM outputs were previously generated for a previous context in a previous interaction session corresponding to the context of the interaction session, and ensuring that the set of assistant outputs is modified using one or more LLM outputs in order to determine the set of modified assistant outputs. In some further versions of these implementations, identifying one or more previously generated LLM outputs using LLM models may include identifying one or more first LLM outputs among one or more LLM outputs that reflect a first personality among several separate personalities. The set of modified assistant outputs may include one or more first personality assistant outputs that reflect a first personality. In further versions of these implementations, identifying one or more previously generated LLM outputs using the LLM model may include identifying one or more second LLM outputs among one or more LLM outputs that reflect a second personality among several separate personalities. The modified set of assistant outputs may include one or more second personality assistant outputs that reflect a second personality, which may be distinct from the first personality.Further additional or alternative versions of these implementations may further include a step of determining that a previous assistant query in a previous dialogue session corresponds to an assistant query in the current dialogue session, based on the ASR output containing one or more terms in the assistant query that correspond to one or more terms in the previous assistant query of the previous dialogue session. Further additional or alternative versions of these implementations may further include a step of generating an embedding of the assistant query based on one or more terms in the ASR output that correspond to the assistant query, and a step of determining that a previous assistant query in a previous dialogue session corresponds to an assistant query in the current dialogue session, based on comparing the embedding of the assistant query with a previously generated embedding of the previous assistant query in the previous dialogue session. Further additional or alternative versions of these implementations may further include a step of determining that a previous context in a previous dialogue session corresponds to a context in the current dialogue session, based on one or more context signals in the current dialogue session that correspond to one or more context signals in the previous dialogue session. Further additional or alternative versions of these implementations may include one or more of the following: time, day of the week, location of the client device, and ambient noise in the environment of the client device. Further additional or alternative versions of these implementations may include the steps of generating an embedding of the context of a dialogue session based on context signals of the dialogue session, and determining that the previous context of a previous dialogue session corresponds to the context of the dialogue session, based on comparing one or more context signal embeddings with previously generated embeddings of the previous context of a previous dialogue session.

[0115] In some versions of these implementations, processing a set of assistant outputs and the context of a dialogue session to generate additional assistant queries related to an utterance, based at least partly on the context of the dialogue session and at least partly on the assistant queries, may include the steps of: determining the intent related to the assistant queries contained in the utterance based on the NLU output; identifying at least one relevant intent regarding the intent related to the assistant queries contained in the utterance based on the intent related to the assistant queries contained in the utterance; and generating additional assistant queries related to the utterance based on at least one relevant intent. In some further versions of these implementations, determining additional assistant outputs in response to additional assistant queries based on additional assistant queries may include ensuring that additional assistant queries are sent to one or more first-party systems via an application programming interface (API) to generate additional assistant outputs in response to additional assistant queries. In some additional or alternative further versions of those implementation forms, determining additional assistant output in response to an additional assistant query based on an additional assistant query may include sending the additional assistant query to one or more third-party systems over one or more networks, and receiving additional assistant output in response to the additional assistant query being sent to one or more third-party systems.In some additional or alternative versions of those implementation forms, processing the context of the additional assistant output and interaction session to determine one or more additional LLM outputs or to generate a set of additional modified assistant outputs using one or more additional LLM outputs may include processing the context of the additional assistant output and interaction session using LLM to determine one or more additional LLM outputs, and determining a set of additional modified assistant outputs based on one or more additional LLM outputs. In some additional or alternative versions of those implementation forms, processing additional assistant outputs and the context of an interaction session to generate an additional set of modified assistant outputs using one or more LLM outputs or additional LLM outputs determined using LLM may include identifying one or more previously generated additional LLM outputs based on the fact that one or more additional LLM outputs were previously generated based on a previous assistant query in a previous interaction session corresponding to an additional assistant query in the interaction session, and / or based on the fact that one or more additional LLM outputs were previously generated for a previous context in a previous interaction session corresponding to the context of the interaction session, and ensuring that the set of additional assistant outputs is modified using one or more additional LLM outputs in order to determine the set of additional modified assistant outputs.

[0116] In some implementations, the method may further include the steps of ranking a top set of assistant outputs based on one or more ranking criteria, wherein the top set of assistant outputs includes at least a set of assistant outputs and a set of modified assistant outputs, and selecting a given modified assistant output from the set of modified assistant outputs based on the ranking. In some versions of those implementations, the method may further include the steps of ranking an additional top set of assistant outputs based on one or more ranking criteria, wherein the top set of assistant outputs includes at least a set of additional assistant outputs and an additional modified assistant output, and selecting a given additional modified assistant output from the additional modified assistant output set based on the ranking. In some further versions of these implementations, providing a given modified assistant output and a given additional modified assistant output for presentation to the user may include the steps of combining a given modified assistant output and a given additional modified assistant output, processing a given modified assistant output and a given additional modified assistant output using a text-to-speech (TTS) model to generate synthesized speech audio data that includes synthesized speech capturing the given modified assistant output and a given additional modified assistant output, and rendering the synthesized speech audio data audibly for presentation to the user via the client device's speaker.

[0117] In some implementations, the method may further include the steps of ranking a top set of assistant outputs based on one or more ranking criteria, wherein the top set of assistant outputs includes a set of assistant outputs, a set of modified assistant outputs, an additional assistant output, and an additional modified assistant output; and selecting a given modified assistant output from the set of modified assistant outputs and a given additional modified assistant output from the set of additional modified assistant outputs, based on the ranking. In some further versions of those implementations, providing a given modified assistant output and a given additional modified assistant output for presentation to the user may include the steps of processing a given modified assistant output and a given additional modified assistant output using a text-to-speech (TTS) model to generate synthesized speech audio data that includes synthesized speech capturing the given modified assistant output and a given additional modified assistant output; and rendering the synthesized speech audio data audibly for presentation to the user via the client device's speaker.

[0118] In some implementations, generating a set of modified assistant outputs using one or more LLM outputs may further be based on processing at least some of the assistant queries contained in the utterance.

[0119] In some implementations, a method performed by one or more processors is provided as part of a dialogue session between a user of a client device and an automated assistant performed by the client device, and includes the steps of: receiving a stream of audio data capturing the user's utterance, the stream of audio data being generated by one or more microphones of the client device, and the utterance containing an assistant query; determining a set of assistant outputs based on having processed the stream of audio data, each assistant output in the set of assistant outputs responding to an assistant query contained in the utterance; and generating a set of assistant outputs modified using one or more LLM outputs generated using a Large-Scale Language Model (LLM), each of which responds to an assistant query contained in the utterance. The method includes the steps of processing a set of assistant outputs and the context of a dialogue session for generating additional assistant queries related to an utterance, determined at least in part on the context of the dialogue session and one or more assistant outputs included in the set of assistant outputs, and generating additional assistant queries related to an utterance, determined at least in part on the context of the dialogue session and at least in part on the assistant queries; determining additional assistant outputs to respond to additional assistant queries based on the additional assistant queries; processing a set of modified assistant outputs based on the additional assistant outputs to respond to additional assistant queries for generating an additional set of modified assistant outputs; and ensuring that a given additional modified assistant output is provided from the set of additional modified assistant outputs for presentation to the user.

[0120] In some implementations, a method performed by one or more processors is provided as part of a dialogue session between a user of a client device and an automated assistant performed by the client device, and includes the steps of: receiving a stream of audio data capturing the user's utterance, the stream of audio data being generated by one or more microphones of the client device, and the utterance containing an assistant query; determining a set of assistant outputs based on having processed the stream of audio data, the set of assistant outputs being each of the assistant outputs responding to an assistant query contained in the utterance; and processing the set of assistant outputs and the context of the dialogue session to generate a set of assistant outputs modified using one or more LLM outputs generated using a Large-Scale Language Model (LLM), each of which is determined based at least in part on the context of the dialogue session and one or more of the assistant outputs contained in the set of assistant outputs. Generating a set of modified assistant outputs using one or more LLM outputs includes generating a set of first personality responses based on (i) a set of assistant outputs, (ii) the context of the interaction session, and (iii) one or more first LLM outputs that reflect a first personality among several separate personalities. The method further includes the step of providing a given modified assistant output from the set of modified assistant outputs for presentation to the user.

[0121] In some implementations, a method performed by one or more processors is provided as part of a dialogue session between a user of a client device and an automated assistant performed by the client device, and includes the steps of: receiving a stream of audio data capturing the user's utterance, the stream of audio data being generated by one or more microphones of the client device, and the utterance including an assistant query; determining a set of assistant outputs based on having processed the stream of audio data, the set of assistant outputs being a response to an assistant query contained in the utterance; processing a set of assistant outputs and the context of the dialogue session to generate a set of modified assistant outputs using one or more LLM outputs generated using a Large-Scale Language Model (LLM), the set of LLM outputs being determined based at least in part on the context of the dialogue session and one or more assistant outputs contained in the set of assistant outputs; and providing a given modified assistant output from the set of modified assistant outputs for presentation to the user.

[0122] In some implementations, a method performed by one or more processors is provided as part of a dialogue session between a user of a client device and an automated assistant performed by the client device, and includes the steps of: receiving a stream of audio data capturing the user's utterance, the stream of audio data being generated by one or more microphones of the client device, and the utterance including an assistant query; determining a set of assistant outputs based on having processed the stream of audio data, each assistant output in the set responding to an assistant query contained in the utterance; determining whether to modify one or more of the assistant outputs in the set of assistant outputs based on having processed the utterance; processing the set of assistant outputs and the context of the dialogue session in order to generate a set of modified assistant outputs using one or more LLM outputs generated using a Large-Scale Language Model (LLM) in response to having decided to modify one or more of the assistant outputs in the set of assistant outputs, each of the one or more LLM outputs being determined based at least in part on the context of the dialogue session and one or more of the assistant outputs in the set of assistant outputs; and providing a given modified assistant output from the set of modified assistant outputs for presentation to the user.

[0123] These and other implementations of the technology disclosed herein may optionally include one or more of the following features:

[0124] In some implementations, determining whether to modify one or more assistant outputs in a set of assistant outputs based on utterance processing may include processing a stream of audio data using an ASR model to generate a stream of automatic speech recognition (ASR) outputs, processing a stream of ASR outputs using an NLU model to generate a stream of natural language understanding (NLU) data, identifying the user's intent in providing the utterance based on the stream of NLU data, and determining whether to modify the assistant output based on the user's intent in providing the utterance.

[0125] In some implementations, the decision of whether to modify one or more assistant outputs included in a set of assistant outputs may further be based on one or more computational costs associated with modifying one or more assistant outputs. In some versions of those implementations, one or more computational costs associated with modifying one or more assistant outputs may include one or more of the following: battery consumption, processor consumption associated with modifying one or more assistant outputs, or latency associated with modifying one or more assistant outputs.

[0126] In some implementations, a method is provided which is carried out by one or more processors, comprising the steps of: obtaining a set of assistant queries directed to an automated assistant and the corresponding context of the corresponding previous interaction session for each of the set of assistant queries; processing a given assistant query among the set of assistant queries using one or more LLMs to generate a corresponding Large Language Model (LLM) output in response to a given assistant query for each of the set of assistant queries; indexing the corresponding LLM output in memory accessible on the client device based on the given assistant query and / or the corresponding context of the corresponding previous interaction session for the given assistant query; and inputting the corresponding LLM output in memory accessible on the client device. The process includes: receiving a stream of audio data capturing a user's utterance as part of a current dialogue session between a user of a client device and an automated assistant performed by the client device, wherein the stream of audio data is generated by one or more microphones of the client device; determining, based on processing the stream of audio data, that the utterance contains a current assistant query corresponding to a given assistant query, and / or that the utterance is received in the current context of the current dialogue session corresponding to the dialogue context of a corresponding previous dialogue session for a given assistant query; and causing the automated assistant to utilize the corresponding LLM output when generating an assistant output to be provided for presentation to the user in response to the utterance.

[0127] These and other implementations of the technology disclosed herein may optionally include one or more of the following features:

[0128] In some implementations, multiple assistant queries directed to the automated assistant may have been previously issued by the user via the client device. In some implementations, multiple assistant queries directed to the automated assistant may have been previously issued by multiple additional users, in addition to the user of the client device, via each of the client devices.

[0129] In some implementations, indexing the corresponding LLM output in memory accessible on the client device may be based on the embedding of a given assistant query generated when processing a given assistant query. In some implementations, indexing the corresponding LLM output in memory accessible on the client device may be based on one or more terms or phrases contained in a given assistant query generated when processing a given assistant query. In some implementations, indexing the corresponding LLM output in memory accessible on the client device may be based on the embedding of the corresponding context of a corresponding previous dialogue session for a given assistant query. In some implementations, indexing the corresponding LLM output in memory accessible on the client device may be based on one or more context signals contained in the corresponding context of a corresponding previous dialogue session for a given assistant query.

[0130] In addition, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein one or more processors are operable to execute instructions stored in associated memory, wherein the instructions are configured to cause the execution of any of the methods described above. Some implementations also include one or more non-temporary computer-readable storage media that store computer instructions executable by one or more processors to perform any of the methods described above. Some implementations also include a computer program product that includes instructions executable by one or more processors to perform any of the methods described above. [Explanation of Symbols]

[0131] 110 Client Devices 111 User Input Engine 112 Rendering Engines 113 Presence Sensor 114 Automated Assistant Client 115 Automated Assistant 120 Natural Conversation Systems 130 ASR engine 140 NLU engine 150 LLM engine 160 TTS engine 170 Offline Output Correction Engine 171 Assistant Activity Engine 172 Indexing Engine 180 Online Output Correction Engine 181 Assistant Query Engine 182 Assistant Personality Engine 190 Ranking Engine 191 1P System 192 3P System 199 Network 201 Audio data stream 202 Context 203 ASR output 204 NLU output 205 Assistant Output 206 Modified Assistant Output 207 Given Assistant Output 710 Computing Devices 712 Bus Subsystem 714 Processors 716 Network Interfaces 720 User Interface Output Devices 722 User Interface Input Devices 724 Storage subsystems 725 Memory subsystem 726 File Storage Subsystem 730 RAM 732 ROM

Claims

1. A method carried out by one or more processors, As part of an interaction session between the user of the client device and the automated assistant performed by the client device, A step of receiving a stream of audio data that captures the user's utterance, wherein the stream of audio data is generated by one or more microphones of the client device, and the utterance includes an assistant query. A step of determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query contained in the utterance, A step of processing the set of assistant outputs and the context of the dialogue session to generate a set of modified assistant outputs and to generate additional assistant queries related to the utterances based on the context of the dialogue session and based on the assistant queries contained in the utterances, wherein each of the modified assistant outputs in the set of modified assistant outputs reflects a first personality from among a plurality of separate personalities; A step of providing a given modified assistant output from the set of modified assistant outputs for presentation to the user, wherein the given modified assistant output includes additional assistant outputs that respond to the additional assistant query. A method that includes [a certain feature].

2. The method according to claim 1, wherein the LLM processes the set of assistant outputs and the context of the dialogue session to generate the set of modified assistant outputs, each reflecting a first personality, thereby causing the LLM to adapt the set of assistant outputs to a first vocabulary associated with the first personality and generate the set of modified assistant outputs.

3. The method according to claim 2, wherein the first personality is separate from the second personality, and the first vocabulary associated with the first personality and used to generate the set of modified assistant outputs is separate from the second vocabulary associated with the second personality.

4. The step of providing the given modified assistant output for presentation to the user is: The steps include generating synthesized speech audio data that captures the given modified assistant output using a first set of prosodic properties associated with the first personality, The steps include:

1. Arranging the synthesized speech audio data capturing the given modified assistant output so that it is rendered audibly through one or more speakers of the client device. The method according to any one of claims 1 to 3, including the method described in any one of claims 1 to 3.

5. The method according to claim 4, wherein the first personality is separate from the second personality, and the first set of prosodic properties associated with the first personality and used to generate the synthesized speech audio data is separate from the second set of prosodic properties associated with the second personality.

6. The method according to any one of claims 1 to 5, further comprising the step of selecting a given modified assistant output from the set of modified assistant outputs based on a probability distribution over a series of words or phrases generated using the LLM.

7. The method according to any one of claims 1 to 6, wherein the LLM is a first LLM specific to the first personality among a plurality of separate LLMs.

8. The method according to any one of claims 1 to 7, wherein the first personality is defined by the user in the settings of the software application.

9. As part of a subsequent interaction session between the user and the automated assistant conducted by the client device, A step of receiving a subsequent stream of audio data that captures the user's subsequent utterance, wherein the subsequent stream of audio data is generated by one or more microphones of the client device, and the subsequent utterance includes a subsequent assistant query. A step of determining a subsequent set of assistant outputs based on processing the subsequent stream of audio data, wherein each assistant output in the subsequent set of assistant outputs responds to a subsequent assistant query contained in the subsequent utterance, The steps include processing the subsequent set of assistant outputs and the subsequent context of the subsequent dialogue session using the LLM or additional LLM to generate a subsequent set of modified assistant outputs, each reflecting a second personality distinct from the first personality, A step of providing a given later modified assistant output from the aforementioned later set of modified assistant outputs for presentation to the user. The method according to any one of claims 1 to 8, comprising:

10. The method according to claim 9, wherein the subsequent set of assistant outputs and the subsequent context of the subsequent dialogue session are processed using the additional LLM to generate the subsequent set of modified assistant outputs, each reflecting a second personality distinct from the first personality, wherein the LLM is related to the first personality and the additional LLM is related to the second personality.

11. A method carried out by one or more processors, As part of an interaction session between the user of the client device and the automated assistant performed by the client device, A step of receiving a stream of audio data that captures the user's utterance, wherein the stream of audio data is generated by one or more microphones of the client device, and the utterance includes an assistant query. A step of determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query contained in the utterance, A step of processing the set of assistant outputs and the context of the dialogue session in order to generate a set of assistant outputs modified using one or more LLM outputs generated using a Large-Scale Language Model (LLM), and to generate additional assistant queries related to the utterance based on the context of the dialogue session and based on the assistant queries contained in the utterance, wherein each of the one or more LLM outputs is determined based at least in part on the context of the dialogue session and one or more of the assistant outputs contained in the set of assistant outputs, and the set of assistant outputs modified using the one or more LLM outputs is generated. The steps include: (i) generating a set of first personality responses based on the set of assistant outputs, (ii) the context of the dialogue session, and (iii) one or more first LLM outputs from the one or more LLM outputs that reflect a first personality among a plurality of separate personalities; A step of providing a given modified assistant output from the set of modified assistant outputs for presentation to the user, wherein the given modified assistant output includes additional assistant outputs that respond to the additional assistant query. A method that includes [a certain feature].

12. A method carried out by one or more processors, As part of an interaction session between the user of the client device and the automated assistant performed by the client device, A step of receiving a stream of audio data that captures the user's utterance, wherein the stream of audio data is generated by one or more microphones of the client device, and the utterance includes an assistant query. A step of determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query contained in the utterance, A step of processing the set of assistant outputs and the context of the dialogue session in order to generate a set of modified assistant outputs using one or more LLM outputs generated using a Large-Scale Language Model (LLM), and to generate additional assistant queries related to the utterance based on the context of the dialogue session and based on the assistant queries contained in the utterance, wherein each of the one or more LLM outputs is determined based at least in part on the context of the dialogue session and one or more of the assistant outputs contained in the set of assistant outputs, and each of the modified assistant outputs in the set of modified assistant outputs reflects a first personality from among a plurality of separate personalities, A step of providing a given modified assistant output from the set of modified assistant outputs for presentation to the user, wherein the given modified assistant output includes additional assistant outputs that respond to the additional assistant query. A method that includes [a certain feature].

13. A method carried out by one or more processors, As part of an interaction session between the user of the client device and the automated assistant performed by the client device, A step of receiving a stream of audio data that captures the user's utterance, wherein the stream of audio data is generated by one or more microphones of the client device, and the utterance includes an assistant query. A step of determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query contained in the utterance, A step of determining whether to modify one or more of the assistant outputs included in the set of assistant outputs based on the processing of the utterance, In response to a decision to modify one or more of the assistant outputs included in the set of assistant outputs, A step of processing the set of assistant outputs and the context of the dialogue session in order to generate a set of modified assistant outputs using one or more LLM outputs generated using a Large-Scale Language Model (LLM), and to generate additional assistant queries related to the utterance based on the context of the dialogue session and based on the assistant queries contained in the utterance, wherein each of the one or more LLM outputs is determined based at least in part on the context of the dialogue session and one or more of the assistant outputs contained in the set of assistant outputs, and each of the modified assistant outputs in the set of modified assistant outputs reflects a first personality from among a plurality of separate personalities, A step of providing a given modified assistant output from the set of modified assistant outputs for presentation to the user, wherein the given modified assistant output includes additional assistant outputs that respond to the additional assistant query. A method that includes [a certain feature].

14. The step of determining whether to modify one or more of the assistant outputs included in the set of assistant outputs based on the processing of the utterance is: The steps include: processing the stream of audio data using an ASR model to generate an automatic speech recognition (ASR) output stream; The steps include: using an NLU model to process the stream of ASR output in order to generate a stream of natural language understanding (NLU) data; A step of identifying the user's intent when providing the utterance based on the stream of NLU data, The method according to claim 13, further comprising the step of determining whether to modify the assistant output based on the user's intent when providing the utterance.

15. The method according to claim 13 or 14, wherein the step of determining whether to modify one or more of the assistant outputs included in the set of assistant outputs is further based on one or more computational costs associated with modifying one or more of the assistant outputs.

16. The method according to claim 15, wherein the one or more computational costs associated with modifying one or more of the assistant outputs include one or more of battery consumption, processor consumption associated with modifying one or more of the assistant outputs, or latency associated with modifying one or more of the assistant outputs.

17. At least one processor, A system comprising: a memory that, when executed, stores instructions that enable at least one processor to operate to perform the method according to any one of claims 1 to 16.

18. A non-temporary computer-readable storage medium that stores instructions, when executed, causing at least one processor to perform an operation corresponding to the method described in any one of claims 1 to 16.

Images