Using large language model(s) in generating automated assistant response(s)

Abstract

To use large language models to generate automated assistant responses.SOLUTION: As part of an interaction session between a user and an automated assistant, an implementation: receives a stream of audio data capturing utterances containing assistant queries; determines a set of predicted assistant outputs each corresponding to a response to an assistant query, based on processing the stream of audio data; uses the large language model (LLM) output to process the assistant output and context of the interaction session, to generate a set of modified assistant outputs; and facilitates a given modified assistant output to be provided for presentation to the user in response to the utterance, from the set of modified assistant outputs. In some implementations, the LLM output may be generated offline for later use online. In additional or alternative implementations, the LLM output may be generated online when the utterance is received.SELECTED DRAWING: Figure 2

Description

[Background technology]

[0001] Humans may engage in human-to-computer interactions with interactive software applications referred to herein as “automated assistants” (also referred to as “chatbots,” “conversational personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” etc.). Automated assistants typically rely on a pipeline of components in interpreting and responding to utterances. For example, an automatic speech recognition (ASR) engine can process audio data corresponding to a user's utterance to generate an ASR output, such as an ASR hypothesis for the utterance (i.e., a series of terms and / or other tokens). Additionally, a natural language understanding (NLU) engine can process the ASR output (or touched / typed input) to generate an NLU output, such as a request (e.g., an intent) expressed by the user in providing the utterance (or touched / typed input), and, optionally, slot values ​​for parameters associated with that intent. Finally, the NLU output may be processed by various fulfillment components to generate fulfillment outputs, such as response content responsive to the utterance and / or one or more actions that may be performed in response to the utterance.

[0002] Generally, an interaction session with an automated assistant is initiated by a user providing an utterance, and the automated assistant can respond to the utterance using the pipeline of components described above. The user can continue the interaction session by providing additional utterances, and the automated assistant can again respond to the additional utterance using the pipeline of components described above. In other words, these interaction sessions are generally turn-based in that the user has a turn to provide an utterance in the interaction session, the automated assistant has a turn to respond to the utterance in the interaction session, the user has an additional turn to provide an additional utterance in the interaction session, the automated assistant has an additional turn to respond to the additional utterance in the interaction session, and so on. However, from the user's perspective, these turn-based interaction sessions may not be natural because they do not reflect the way humans actually converse with each other.

[0003] For example, if a first person provides an utterance during an interaction session to convey an initial thought to a second person (e.g., "I'm going to the beach today"), the second person can consider that utterance within the context of the interaction session when providing a response to the first person (e.g., "sounds fun, what are you going to do at the beach?", "nice, have you looked at the weather?", etc.). In particular, the second person, in responding to the first person, can provide utterances that keep the first person engaged in the interaction session in a natural manner. In other words, during an interaction session, rather than one person driving the interaction session, both the first person and the second person can provide utterances to encourage natural conversation.

[0004] However, if the second person in the above example is replaced by an automated assistant, the automated assistant may not provide a response that keeps the first person involved in the interaction session. For example, in response to the first person providing the utterance "I'm going to the beach today," the automated assistant may simply respond "sound fun" or "nice" without providing any additional response to prompt the interaction session, even though it can take some action and / or provide some response to prompt the interaction session, such as proactively asking the first person what they plan to do at the beach, proactively checking the weather forecast for a beach the first person often visits and including the weather forecast in the response, or proactively making some inference based on the weather forecast. As a result, the response provided by the automated assistant in response to the first person's utterance may not resonate with the first person because it may not reflect a natural conversation between multiple people. Furthermore, the first person may have to provide additional utterances to specifically request some information that the automated assistant can proactively provide (e.g., the weather forecast for the beach), increasing the volume of utterances directed to the automated assistant and wasting the computational resources of the client device utilized in processing these utterances. Summary of the Invention [Means for solving the problem]

[0005] Implementations described herein are directed to enabling an automated assistant to engage in natural conversation with a user during an interaction session. Some implementations can receive a stream of audio data capturing a user's speech. The stream of audio data can be generated by one or more microphones of a client device, and the speech can include an assistant query. Some implementations can further process a set of assistant outputs and the context of the interaction session to determine a set of automated assistants based on processing the stream of audio data and generate a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM). Each of the one or more LLM outputs can be determined based on at least a portion of the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs. Some implementations can further provide a given modified assistant output for presentation to the user from among the set of modified assistant outputs. Furthermore, each of the one or more LLM outputs can include, for example, a probability distribution over a set of one or more words and / or phrases spanning one or more vocabularies, and one or more of the set of words and / or phrases can be selected as the one or more LLM outputs based on the probability distribution. Additionally, the context of the interaction session may be determined based on one or more contextual signals including, for example, the time of day, the day of the week, the location of the client device, ambient noise detected in the client device's environment, user profile data, software application data, environmental data about the known environment of the user of the client device, the interaction history of the interaction session between the user and the automated assistant, and / or other contextual signals.

[0006] In some implementations, the set of assistant outputs can be determined based on processing a stream of audio data using a streaming ASR model to generate a stream of automatic speech recognition (ASR) outputs, such as one or more recognized terms or phrases predicted to correspond to the utterance, one or more phonemes predicted to correspond to the utterance, one or more predicted measures associated with each of the one or more recognized terms or phrases and / or one or more predicted phonemes, and / or other ASR outputs. Furthermore, the ASR outputs can be processed using an NLU model to generate a stream of natural language understanding (NLU) outputs, such as one or more predicted intents of the user in providing the utterance and one or more corresponding slot values ​​for one or more parameters associated with each of the one or more predicted intents. Furthermore, the stream of NLU data can be processed by one or more first-party (1P) and / or third-party (3P) systems to generate a set of assistant outputs. As used herein, one or more 1P systems include systems developed and / or maintained by the same entity (e.g., a common publisher) that develops and / or maintains the automated assistant described herein, while one or more 3P systems include systems developed and / or maintained by an entity separate from the entity that develops and / or maintains the automated assistant described herein. In particular, the set of assistant outputs described herein includes assistant outputs that are typically considered to respond to an utterance. However, by using the claimed technique, the set of assistant outputs generated in the manner described above can be further processed to generate a set of modified assistant outputs. Specifically, the set of assistant outputs can be modified using one or more LLM outputs, and a given modified assistant output can be selected from the set of modified assistant outputs to be provided for presentation to a user in response to receiving an utterance.

[0007] For example, assume that a user of a client device provides the utterance, "Hey Assistant, I'm thinking about going surfing today." In this example, the automated assistant can process the utterance in the manner described above to generate a set of assistant outputs and a set of modified assistant outputs. Assistant outputs included in the set of assistant outputs in this example may include, for example, "That sounds like fun!", "Sounds fun!", etc. Furthermore, assistant outputs included in the set of modified assistant outputs in this example may include, for example, "That sounds like fun, how long have you been surfing?", "Enjoy it, but if you're going to Example Beach again, be prepared for some light showers," etc. In particular, the assistant outputs included in the set of assistant outputs do not include any assistant output that drives the interaction session in a manner that further engages the user of the client device in the interaction session, while the assistant outputs included in the set of modified assistant outputs include assistant outputs that drive the interaction session in a manner that further engages the user of the client device in the interaction session by asking a contextually relevant question (e.g., "how long have you been surfing?"), assistant outputs that provide contextually relevant information (e.g., "but if you're going to Example Beach again, be prepared for some light showers"), and / or assistant outputs that resonate differently with the user of the client device within the context of the interaction session.

[0008] In some implementations, the set of modified assistant responses can be generated using one or more LLM outputs generated in an online manner. For example, in response to receiving an utterance, the automated assistant can cause a set of assistant outputs to be generated in the manner described above. Further, also in response to receiving an utterance, the automated assistant can cause the set of assistant outputs, the context of the interaction session, and / or the assistant query included in the utterance to be processed using one or more LLMs to generate a set of modified assistant outputs based on one or more LLM outputs generated using one or more LLMs.

[0009] In additional or alternative implementations, the set of modified assistant responses can be generated using one or more LLM outputs generated in an offline manner. For example, before receiving an utterance, the automated assistant can obtain multiple assistant queries and corresponding contexts of corresponding previous interaction sessions for each of the multiple assistant queries from an assistant activity database (which can be limited assistant activity of a user of a client device). Furthermore, the automated assistant can cause a set of assistant outputs to be generated for a given assistant query of the multiple assistant queries in the manner described above. Furthermore, the automated assistant can cause the set of assistant outputs, the corresponding context of the interaction session, and / or the given assistant query to be processed using one or more LLMs to generate a set of modified assistant outputs based on one or more LLM outputs generated using one or more LLMs. This process can be repeated for each of the multiple queries and the corresponding context of the previous interaction session obtained by the automated assistant.

[0010] Additionally, the automated assistant can index one or more LLM outputs in memory accessible by the user's client device. In some implementations, the automated assistant can cause one or more LLMs to be indexed in memory based on one or more terms included in multiple assistant queries. In additional or alternative implementations, the automated assistant can generate a corresponding embedding (e.g., a word2vec embedding or another lower-dimensional representation) for each of the multiple assistant queries and map each of the corresponding embeddings into the assistant query embedding space to index one or more LLM outputs. In additional or alternative implementations, the automated assistant can cause one or more LLMs to be indexed in memory based on one or more context signals included in corresponding previous contexts. In additional or alternative implementations, the automated assistant can generate a corresponding embedding for each of the corresponding contexts and map each of the corresponding embeddings into the context embedding space to index one or more LLM outputs. In additional or alternative implementations, the automated assistant can cause one or more LLMs to be indexed in memory based on one or more terms or phrases of the assistant outputs included in the set of assistant outputs for each of the multiple assistant queries. In additional or alternative implementations, the automated assistant can generate a corresponding embedding (e.g., a word2vec embedding or another lower-dimensional representation) for each of the assistant outputs included in the set of assistant outputs and map each of the corresponding embeddings to the assistant output embedding space to index the one or more LLM outputs.

[0011] Thus, when the utterance is subsequently received at the user's client device, the automated assistant can identify one or more LLM outputs previously generated based on the current assistant query corresponding to one or more of the assistant queries included in the multiple queries, the current context corresponding to one or more of the corresponding previous contexts, and / or one or more current assistant outputs corresponding to one or more of the previous assistant outputs. For example, in an implementation in which one or more LLM outputs are indexed based on corresponding embeddings for previous assistant queries, the automated assistant can cause an embedding for the current assistant query to be generated and mapped to the assistant query embedding space. Furthermore, the automated assistant can determine that the current assistant query corresponds to the previous assistant query based on the distance between the embedding for the current assistant query and the corresponding embedding for the previous assistant query in the query embedding space satisfying a threshold. The automated assistant can retrieve from memory the one or more LLM outputs generated based on processing the previous assistant query and utilize the one or more LLM outputs in generating a set of modified assistant outputs. Also, in implementations in which one or more LLMs are indexed, for example, based on one or more terms included in multiple assistant queries, the automated assistant can determine the edit distance between a current assistant query and multiple previous assistant queries, for example, to identify previous assistant queries that correspond to the current assistant query. Similarly, the automated assistant can retrieve from memory one or more LLM outputs generated based on processing previous assistant queries and utilize the one or more LLM outputs in generating a revised set of assistant outputs.

[0012] In some implementations, in addition to one or more LLM outputs, additional assistant queries may be generated based on processing the assistant query and / or the context of the interaction session. For example, when processing the assistant query and / or the context of the interaction session, the automated assistant can determine an intent related to the given assistant query based on a stream of NLU data. Furthermore, the automated assistant can identify at least one related intent related to the intent associated with the assistant query based on the intent related to the given assistant query (e.g., based on mapping the intent to at least one related intent in a database or memory accessible to the client device and / or based on processing the intent related to the given assistant query using one or more machine learning (ML) models or heuristically defined rules). Furthermore, the automated assistant can generate additional assistant queries based on the at least one related intent. For example, assume that the assistant query indicates that the user plans to go to the beach (e.g., "Hey assistant, I'm going to the beach today"). In this example, the additional assistant query may correspond to, for example, "what's the weather at Example Beach?" (e.g., to proactively determine weather information at a beach that a user named Example Beach regularly visits). In particular, the additional assistant queries may not be provided for presentation to the user of the client device.

[0013] Rather, in these implementations, the additional assistant output may be determined based on processing the additional assistant query. For example, the automated assistant may send a structured request to one or more 1P and / or 3P systems to obtain weather information as the additional assistant output. Further, assume that the weather information indicates that rain is expected at Example Beach. In some versions of these implementations, the automated assistant may further cause the additional assistant to be processed using one or more of the LLM output and / or one or more additional LLM outputs to generate an additional set of modified assistant outputs. Thus, in the initial example given above, a given modified assistant output provided to the user in response to receiving the utterance "Hey Assistant, I'm thinking about going surfing today" from the initial set of modified assistant outputs may be "Enjoy it," and a given additional modified assistant output from the additional set of modified assistant outputs may be "but if you're going to Example Beach again, be prepared for some light showers." In other words, the automated assistant

[0014] In various implementations, each of the one or more LLM outputs utilized in generating the set of modified assistant outputs can be generated using a corresponding set of parameters from a plurality of distinct sets of parameters. Each of the plurality of distinct sets of parameters can be associated with a distinct personality for the automated assistant. In some versions of these implementations, a single LLM can be utilized to generate one or more corresponding LLM outputs using a corresponding set of parameters for each of the distinct personalities, while in other versions of these implementations, multiple LLMs can be utilized to generate one or more corresponding LLM outputs using a corresponding set of parameters for each of the distinct personalities. Thus, when a given modified assistant output from the set of modified assistant outputs is provided for presentation to a user, it can reflect various changing contextual personalities through the prosodic properties of the different personalities (e.g., the intonation, inflection, tone, pauses, tempo, stress, rhythm, etc. of these different personalities).

[0015] In particular, the replies of these personalities described herein can reflect not only the prosodic qualities of the different personalities, but also the distinct vocabularies and / or speaking styles (e.g., verbose speaking, concise speaking, etc.) of the different personalities. For example, a given modified assistant output provided for presentation to a user can be generated using a first set of parameters that reflect a first personality of the automated assistant with respect to a first vocabulary to be utilized by the automated assistant and / or a first set of prosodic qualities to be utilized in providing the modified assistant output for audible presentation to the user. Alternatively, a modified assistant output provided for presentation to a user can be generated using a second set of parameters that reflect a second personality of the automated assistant with respect to a second vocabulary to be utilized by the automated assistant and / or a second set of prosodic qualities to be utilized in providing the modified assistant output for audible presentation to the user.

[0016] Thus, the automated assistant can dynamically adapt the personalities utilized in providing the modified assistant output for presentation to the user based on both the vocabulary utilized by the automated assistant and the prosodic characteristics utilized in rendering the modified assistant output for audible presentation to the user. In particular, the automated assistant can dynamically adapt the personalities utilized in providing the modified assistant output based on the context of the interaction session, including previous utterances received from the user, as well as previous assistant outputs provided by the automated assistant and / or any other contextual signals described herein. As a result, the modified assistant output provided by the automated assistant may resonate more with the user of the client device. Furthermore, it should be noted that the personalities used throughout a given interaction session can be dynamically adapted as the context of the given interaction session is updated.

[0017] In some implementations, the automated assistant may rank the assistant outputs included in the set of assistant outputs (i.e., not generated using one or more LLM outputs) and the set of modified assistant outputs (i.e., generated using one or more LLM outputs) according to one or more ranking criteria. Thus, when selecting a given assistant output to be provided for presentation to a user, the automated assistant can select from both the set of assistant outputs and the set of modified assistant outputs. The one or more ranking criteria may include, for example, one or more predicted measures indicating how responsive each of the assistant outputs included in the set of assistant outputs and the set of modified assistant outputs is predicted to be to the assistant query included in the utterance (e.g., an ASR measure generated when generating the stream of ASR outputs, an NLU measure generated when generating the stream of NLU outputs, a fulfillment measure generated when generating the set of assistant outputs), one or more intents included in the stream of NLU outputs, and / or other ranking criteria. For example, if the client device user's intent indicates that the user desires a factual answer (e.g., based on providing an utterance that includes the assistant query "why is the sky blue?"), the user likely desires a simple answer to the assistant query, and the automated assistant can favor one or more of the assistant outputs included in the set of one or more assistant outputs. However, if the client device user's intent indicates that the user provided an open-ended input (e.g., based on providing an utterance that includes the assistant query "what time is it?"), the user likely prefers a more conversational modality, and the automated assistant can favor one or more of the assistant outputs included in the set of modified assistant outputs.

[0018] In some implementations, before generating the set of modified assistant outputs, the automated assistant may even determine whether to generate the set of modified assistant outputs. In some versions of those implementations, the automated assistant may even determine whether to generate the set of modified assistant outputs based on one or more predicted intents of the user when providing the utterance as indicated by the stream of NLU data. For example, in an implementation in which the automated assistant determines that the utterance requests the automated assistant to perform a search (e.g., the assistant query, "Why is the sky blue?"), the automated assistant may determine not to generate the set of modified assistant outputs because the user is seeking a factual answer. In additional or alternative versions of those implementations, the automated assistant may even determine whether to generate the set of modified assistant outputs based on one or more computational costs associated with modifying one or more of the assistant outputs. The one or more computational costs associated with modifying one or more of the assistant outputs may include, for example, one or more of battery consumption, processor consumption associated with modifying one or more of the assistant outputs, or latency associated with modifying one or more of the assistant outputs. For example, if the client device is in a low power mode, the automated assistant may decide not to generate a modified set of assistant outputs in order to reduce battery consumption on the client device.

[0019] By using the techniques described herein, one or more technical advantages can be achieved. In one non-limiting example, the techniques described herein enable an automated assistant to engage in a natural conversation with a user during an interaction session. For example, the automated assistant can generate a modified assistant output using one or more LLM outputs of a more conversational nature. Thus, the automated assistant can proactively provide contextual information related to the interaction session that was not directly requested by the user (e.g., by generating additional assistant queries as described herein and by providing additional assistant outputs determined based on the additional assistant queries), thereby making the modified assistant output resonate with the user. Furthermore, the modified assistant output may be generated with different personalities, both in terms of contextually adapted vocabulary throughout the interaction session and prosodic characteristics utilized to audibly render the modified assistant output, thereby making the modified assistant output more resonate with the user. This can result in various technical advantages, such as saving computational resources on the client device, allowing the interaction session to conclude more quickly and efficiently, and / or reducing the volume of the interaction session. For example, information that is contextually relevant to an interaction session may be proactively provided for presentation to a user by an automated assistant, thereby reducing the amount of situations in which a user must request such information, thereby reducing the amount of user input received at a client device. Also, for example, in implementations in which one or more LLM outputs are generated in an offline manner and subsequently utilized in an online manner, latency may be reduced at runtime.

[0020] As used herein, an "interaction session" may include a logically self-contained exchange between a user and an automated assistant (and in some cases, other human participants). The automated assistant may distinguish multiple interaction sessions with a user based on various signals, such as the passage of time between sessions, changes in user status between sessions (e.g., location, before / during / after a scheduled meeting, etc.), detection of one or more interfering interactions between the user and the client device other than the interaction between the user and the automated assistant (e.g., the user switches applications for a while, the user moves away from a standalone voice-activated product and returns later), locking / sleeping the client device between sessions, changes in the client device used to interface with the automated assistant, etc. In particular, during a given interaction session, the user may interact with the automated assistant using various input modalities, including, but not limited to, spoken input, typed input, and / or touch input.

[0021] The above description is provided as a summary of only some implementations disclosed herein. Those implementations and other implementations are described in additional detail herein.

[0022] It should be understood that the techniques disclosed herein may be implemented locally on a client device, remotely by a server connected to the client device via one or more networks, and / or both. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 illustrates various aspects of the present disclosure and is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Figure 2] FIG. 1 illustrates an example process flow for utilizing a large-scale language model in generating assistant output, according to various implementations. [Figure 3] 1 is a flowchart illustrating an example method of utilizing a large language model in generating assistant output in an offline manner for later use in an online manner, according to various implementations. [Figure 4] 1 is a flowchart illustrating an example method of utilizing a large-scale language model in generating an assistant output based on generating an assistant query, according to various implementations. [Figure 5] 1 is a flowchart illustrating an example method of utilizing a large-scale language model in generating an Assistant output based on generated responses of an Assistant personality, according to various implementations. [Figure 6] FIG. 1 illustrates a non-limiting example of an interaction session between a user and an automated assistant, where the automated assistant utilizes a large-scale language model in generating assistant output, according to various implementations. [Figure 7] FIG. 1 illustrates an exemplary architecture of a computing device, according to various implementations. DETAILED DESCRIPTION OF THE INVENTION

[0024] Turning now to FIG. 1 , a block diagram of an example environment 100 is shown that illustrates various aspects of the present disclosure and in which implementations disclosed herein may be practiced. The example environment 100 includes a client device 110 and a natural conversation system 120. In some implementations, the natural conversation system 120 may be implemented locally at the client device 110. In additional or alternative implementations, the natural conversation system 120 may be implemented remotely (e.g., at a remote server) from the client device 110 as shown in FIG. 1 . In these implementations, the client device 110 and the natural conversation system 120 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks (“LANs,” including Wi-Fi LANs, mesh networks, Bluetooth, near-field communications, etc.) or wide area networks (“WANs,” including the Internet).

[0025] Client device 110 may be, for example, one or more of a desktop computer, a laptop computer, a tablet, a mobile phone, a vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (optionally having a display), a smart appliance such as a smart television, and / or a user wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0026] Client device 110 can execute automated assistant client 114. An instance of automated assistant client 114 may be an application separate from (e.g., installed “on”) the operating system of client device 110, or alternatively, may be implemented directly by the operating system of client device 110. Automated assistant client 114 can interact with natural conversation system 120, implemented locally at client device 110 or implemented and invoked remotely via one or more of networks 199 as shown in FIG. 1. Automated assistant client 114 (and optionally by interacting with other remote systems (e.g., servers)) can form what appears from a user's perspective to be a logical instance of automated assistant 115 with which the user may engage in human-to-computer interaction. An instance of automated assistant 115 is shown in FIG. 1 and is surrounded by a dashed line that includes automated assistant client 114 and natural conversation system 120 of client device 110. Thus, it should be understood that a user interacting with an automated assistant client 114 executing on a client device 110 may effectively interact with their own logical instance of the automated assistant 115 (or a logical instance of the automated assistant 115 shared among a household or group of other users). For brevity and simplicity, automated assistant 115 as used herein refers to an automated assistant client 114 executing locally on client device 110 and / or remotely on one or more remote servers that may implement natural conversation system 120.

[0027] In various implementations, client device 110 may include a user input engine 111 configured to detect user input provided by a user of client device 110 using one or more user interface input devices. For example, client device 110 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to the user's speech or other sounds in the client device's 110 environment. Additionally or alternatively, client device 110 may be equipped with one or more vision components configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected in one or more fields of view of the vision component. Additionally or alternatively, client device 110 may be equipped with one or more touch-sensing components (e.g., a keyboard and mouse, a stylus, a touchscreen, a touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to touch input directed at client device 110.

[0028] In various implementations, client device 110 may include a rendering engine 112 configured to provide content for audible and / or visual presentation to a user of client device 110 using one or more user interface output devices. For example, client device 110 may be equipped with one or more speakers that enable content to be provided for audible presentation to a user via client device 110. Additionally or alternatively, client device 110 may be equipped with a display or projector that enables content to be provided for visual presentation to a user via client device 110.

[0029] In various implementations, client device 110 may include one or more presence sensors 113 configured to provide a signal indicating a detected presence, particularly the presence of a person, with an acknowledgement from a corresponding user. In some of those implementations, automated assistant 115 may identify client device 110 (or another computing device associated with the user of client device 110) at which to satisfy an utterance based at least in part on the user's presence at client device 110 (or another computing device associated with the user of client device 110). The utterance may be satisfied by rendering response content at client device 110 and / or another computing device associated with the user of client device 110 (e.g., via rendering engine 112), by causing client device 110 and / or other computing devices associated with the user of client device 110 to be controlled, and / or by causing client device 110 and / or other computing devices associated with the user of client device 110 to perform any other action to satisfy the utterance. As described herein, the automated assistant 115 can utilize data determined based on the presence sensor 113 in determining which client device 110 (or other computing device) the user is near or has recently been near, and provide corresponding commands to only that client device 110 (or those other computing devices).In some additional or alternative implementations, the automated assistant 115 may utilize data determined based on the presence sensor 113 when determining whether any users (any user or specific users) are currently near the client device 110 (or other computing device), and may optionally suppress the provision of data to and / or from the client device 110 (or other computing device) based on users who are near the client device 110 (or other computing device).

[0030] The presence sensor 113 may take various forms. For example, the client device 110 may utilize one or more of the user interface input components described above with respect to the user input engine 111 to detect the presence of a user. Additionally or alternatively, the client device 110 may be equipped with other types of light-based presence sensors 113, such as a passive infrared ("PIR") sensor that measures infrared ("IR") light emanating from objects within its field of view.

[0031] Additionally or alternatively, in some implementations, presence sensor 113 may be configured to detect other phenomena related to the presence of a person or a device. For example, in some embodiments, client device 110 may be equipped with presence sensor 113 that detects, for example, other computing devices (e.g., mobile devices, wearable computing devices) carried / operated by a user and / or various types of wireless signals (e.g., waves such as radio waves, ultrasound, electromagnetic waves, etc.) emitted by other computing devices. For example, client device 110 may be configured to emit waves imperceptible to humans, such as ultrasound or infrared waves, that can be detected by other computing devices (e.g., via an ultrasound / infrared receiver such as an ultrasound-enabled microphone).

[0032] Additionally or alternatively, client device 110 may emit other types of imperceptible waves, such as radio waves (e.g., Wi-Fi, Bluetooth, cellular, etc.), that can be detected by other computing devices (e.g., mobile devices, wearable computing devices, etc.) carried / operated by the user and used to determine the user's specific location. In some implementations, GPS and / or Wi-Fi triangulation may be used to detect a person's location, for example, based on GPS and / or Wi-Fi signals to / from client device 110. In other implementations, other wireless signal characteristics, such as time of flight, signal strength, etc., may be used alone or collectively by client device 110 to determine the location of a particular person based on signals emitted by other computing devices carried / operated by the user. Additionally or alternatively, in some implementations, client device 110 may perform speaker identification (SID) to recognize a user from their voice and / or face identification (FID) to recognize a user from visual data capturing the user's face.

[0033] In some implementations, the speaker's movements may then be determined, for example, by the presence sensor 113 of the client device 110 (and optionally a GPS sensor, Soli chip, and / or accelerometer of the client device 110). In some implementations, based on such detected movements, the user's location may be predicted, and this location may be assumed to be the user's location when any content is caused to be rendered on the client device 110 and / or other computing devices, based at least in part on the proximity of the client device 110 and / or other computing devices to the user's location. In some implementations, the user may simply be assumed to be in the location where they last engaged with the automated assistant 115, especially if not much time has passed since that last engagement.

[0034] Additionally, client device 110 and / or natural conversation system 120 may include one or more memories for storage of data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components that facilitate communication over one or more of networks 199. In some implementations, one or more of the software applications may be installed locally at client device 110, while in other implementations, one or more of the software applications may be hosted remotely (e.g., by one or more servers) and may be accessible by client device 110 over one or more of networks 199.

[0035] In some implementations, the operations performed by automated assistant 115 may be implemented locally at client device 110 via automated assistant client 114. As shown in FIG. 1, automated assistant client 114 may include automatic speech recognition (ASR) engine 130A1, natural language understanding (NLU) engine 140A1, large-scale language model (LLM) engine 150A1, and text-to-speech (TTS) engine 160A1. In some implementations, the operations performed by automated assistant 115 may be distributed across multiple computer systems, such as when natural conversation system 120 is implemented remotely from client device 110 as shown in FIG. 1. In these implementations, automated assistant 115 may additionally or alternatively utilize ASR engine 130A2, NLU engine 140A2, LLM engine 150A2, and TTS engine 160A2 of natural conversation system 120.

[0036] Each of these engines may be configured to perform one or more functions. For example, ASR engine 130A1 and / or 130A2 may process a stream of audio data capturing speech and generated by a microphone of client device 110 using a streaming ASR model stored in machine learning (ML) model database 115A (e.g., a recurrent neural network (RNN) model, a transformer model, and / or any other type of ML model capable of performing ASR) to generate a stream of ASR output. In particular, the streaming ASR model may be utilized to generate a stream of ASR output as the stream of audio data is generated. Furthermore, NLU engine 140A1 and / or 140A2 may process the stream of ASR output using an NLU model (e.g., a long short-term memory (LSTM), a gated recurrent unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or grammar-based rules stored in ML model database 115A to generate a stream of NLU output. Moreover, the automated assistant 115 can cause the NLU output to be processed to generate a stream of fulfillment data. For example, the automated assistant 115 can send one or more structured requests to one or more first-party (1P) systems 191 via one or more networks 199 (or one or more application programming interfaces (APIs)) and / or to one or more third-party (3P) systems 192 via one or more networks, and receive fulfillment data from one or more of the 1P systems 191 and / or 3P systems 192 to generate the stream of fulfillment data. The one or more structured requests can include, for example, NLU data to be included in the stream of fulfillment data.The stream of fulfillment data may correspond, for example, to a set of Assistant outputs predicted to respond to an Assistant query contained in an utterance captured in the stream of audio data processed by ASR engine 130A1 and / or 130A2.

[0037] Additionally, LLM engines 150A1 and / or 150A2 can process a set of assistant outputs predicted to respond to an assistant query included in an utterance captured in the stream of audio data processed by ASR engines 130A1 and / or 130A2. As described herein (e.g., with respect to FIGS. 2-6), in some implementations, LLM engines 150A1 and / or 150A2 can use one or more LLM outputs to generate a modified set of assistant outputs, such that the set of assistant outputs is modified. In some versions of those implementations (e.g., as described with respect to FIG. 3), automated assistant 115 can cause one or more of the LLM outputs to be generated in an offline manner (e.g., without response to utterances being received during an interaction session) and subsequently utilized in an online manner (e.g., as utterances are received during an interaction session) to generate a modified set of assistant outputs. In additional or alternative implementations (e.g., as described with respect to FIGS. 4 and 5), automated assistant 115 can cause one or more of the LLM outputs to be generated in an online manner (e.g., as utterances are received during an interaction session). In these implementations, the one or more LLM outputs can be generated based on using one or more LLMs (e.g., one or more transformer models such as Meena, RNN, and / or any other LLM) stored in model database 115A to process a set of assistant outputs (e.g., a stream of fulfillment data), the context of the interaction session in which the utterance is received (e.g., based on one or more contextual signals stored in context database 110A), recognized text corresponding to an assistant query included in the utterance, and / or other information that automated assistant 115 can leverage in generating one or more of the LLM outputs.Each of the one or more LLM outputs can include, for example, a probability distribution over a set of one or more words and / or phrases across one or more vocabularies, where one or more of the set of words and / or phrases can be selected as the one or more LLM outputs based on the probability distribution. In various implementations, one or more of the LLM outputs can be stored in LLM output database 150A for later use in modifying one or more assistant outputs included in the set of assistant outputs.

[0038] Moreover, in some implementations, TTS engine 160A1 and / or 160A2 can process text data (e.g., text spoken by automated assistant 115) using TTS models stored in ML model database 115A to generate synthetic speech audio data, including computer-generated synthetic speech. The text data may correspond, for example, to one or more assistant outputs from a set of assistant outputs included in the stream of fulfillment data, one or more modified assistant outputs from a set of modified assistant outputs, and / or any other text data described herein. In particular, the ML models stored in ML model database 115A may be on-device ML models stored locally at client device 110, or may be shared ML models accessible to both client device 110 and / or a remote system when natural conversation system 120 is not implemented locally at client device 110. In additional or alternative implementations, audio data corresponding to one or more assistant outputs from the set of assistant outputs included in the stream of fulfillment data, one or more modified assistant outputs from the set of modified assistant outputs, and / or any other text data described herein may be stored in memory or one or more databases accessible by client device 110 so that the automated assistant does not need to use TTS engines 160A1 and / or 160A2 to generate any synthesized voice audio data when providing the audio data for audible presentation to the user.

[0039] In various implementations, the stream of ASR output may include, for example, a stream of ASR hypotheses (e.g., term hypotheses and / or transcript hypotheses) predicted to correspond to the user's utterance captured in the stream of audio data, one or more corresponding predicted values ​​(e.g., probabilities, log-likelihoods, and / or other values) for each of the ASR hypotheses, a plurality of phonemes predicted to correspond to the user's utterance captured in the stream of audio data, and / or other ASR outputs. In some versions of these implementations, ASR engine 130A1 and / or 130A2 can select one or more of the ASR hypotheses as recognized text corresponding to the utterance (e.g., based on the corresponding predicted values).

[0040] In various implementations, the stream of NLU output may include a stream of annotated recognized text, including, for example, one or more annotations of the recognized text for one or more (e.g., all) of the terms of the recognized text. For example, NLU engine 140A1 and / or 140A2 may include a portion of a speech tagger (not shown) configured to annotate terms with the grammatical roles of the terms. Additionally or alternatively, NLU engine 140A1 and / or 140A2 may include an entity tagger (not shown) configured to annotate entity references in one or more segments of the recognized text, such as references to people (e.g., including literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some implementations, data about entities may be stored in one or more databases, such as in a known graph (not shown). In some implementations, the known graph may include nodes representing known entities (and possibly entity attributes) and edges connecting the nodes to represent relationships between the entities. An entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and / or at a low level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on the content of the natural language input to resolve particular entities and / or may optionally communicate with a known graph or other entity database to resolve particular entities.

[0041] Additionally or alternatively, NLU engines 140A1 and / or 140A2 may include a coreference resolver (not shown) configured to group or “cluster” references to the same entity based on one or more contextual cues. For example, the coreference resolver may be used to resolve the term “them” in the input “buy them” to “buy theatre tickets” based on the mention of “theater tickets” in a client device notification rendered immediately before receiving the natural language input “buy them.” In some implementations, one or more components of NLU engines 140A1 and / or 140A2 may rely on annotations from one or more other components of NLU engines 140A1 and / or 140A2. For example, in some implementations, an entity tagger may rely on annotations from the coreference resolver when annotating all references to a particular entity. Also, for example, in some implementations, the coreference resolver may rely on annotations from the entity tagger when clustering references to the same entity.

[0042] 1 is described with respect to a single client device with a single user, it should be understood that this is for purposes of illustration and not intended to be limiting. For example, one or more additional client devices of the user may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user may form an ecosystem of devices that can utilize the techniques described herein. These additional client devices and / or computing devices may be in communication with client device 110 (e.g., via network 199). As another example, a given client device may be utilized by multiple users in a shared setting (e.g., a group of users, a household).

[0043] As described herein, the automated assistant 115 can determine whether to modify a set of assistant responses using one or more of the LLM outputs and / or determine one or more sets of modified assistant outputs based on one or more of the LLM outputs. The automated assistant 115 can utilize the natural conversation system 120 to make these decisions. In various implementations, as shown in FIG. 1 , the natural conversation system 120 can additionally or alternatively include an offline output modification engine 170, an online output modification engine 180, and / or a ranking engine 190. The offline output modification engine 170 can include, for example, an assistant activity engine 171 and an indexing engine 172. Furthermore, the online output modification engine 180 can include, for example, an assistant query engine 181 and an assistant personality engine 182. These various engines of the natural conversation system 120 are described in more detail with respect to FIGS. 2 through 5 .

[0044] 2, an example process flow 200 for utilizing an LLM in generating assistant output is shown. A stream of audio data 201 generated by one or more microphones of client device 110 of FIG. 1 may be processed by ASR engines 130A1 and / or 130A2 to generate a stream of ASR output 203. Further, ASR output 203 may be processed by NLU engines 140A1 and / or 140A2 to generate a stream of NLU output 204. In some implementations, NLU engines 140A1 and / or 140A2 may process the context 202 of an interaction session between a user of client device 110 and automated assistant 115, which is executed at least in part at the user's client device 110. In some versions of those implementations, the context 202 of the interaction session may be determined based on one or more contextual signals generated by the client device 110 (e.g., time of day, day of the week, location of the client device 110, ambient noise detected in the environment of the client device 110, and / or other contextual signals generated by the client device 110). In additional or alternative versions of those implementations, the context 202 of the interaction session may be determined based on one or more contextual signals stored in a context database 110A accessible at the client device 110 (e.g., user profile data, software application data, environmental data about the known environment of the user of the client device 110, interaction history of an ongoing interaction session between the user and the automated assistant 115 and / or past interaction history of one or more previous interaction sessions between the user and the automated assistant 115, and / or other contextual data stored in the context database 110A).Furthermore, the stream of NLU outputs 204 may be processed by one or more of the 1P system 191 and / or 3P system to generate a stream of fulfillment data including a set of one or more assistant outputs 205, each of which is predicted to respond to speech captured in the stream of audio data 201.

[0045] Typically, in a turn-based interaction session that does not utilize an LLM, ranking engine 190 may process the set of one or more assistant outputs 205 to rank each of the one or more assistant outputs 205 included in the set of one or more assistant outputs 205 according to one or more ranking criteria, and the automated assistant 115 may select one or more given assistant outputs 207 from the set of one or more assistant outputs 205 to provide for presentation to a user of client device 110 in response to receiving an utterance. In some implementations, the selected one or more given assistant outputs 207 may be processed by TTS engine 160A1 and / or 160A2 to generate synthesized speech audio data including synthesized speech corresponding to the selected one or more given assistant outputs 207, and rendering engine 112 can cause the synthesized speech audio data to be audibly rendered by a speaker of client device 110 for audible presentation to a user of client device 110. In additional or alternative implementations, the rendering engine 112 can cause text data corresponding to the selected one or more given assistant outputs 207 to be visually rendered by a display of the client device 110 for visual presentation to a user of the client device 110.

[0046] However, when using the claimed techniques, automated assistant 115 can further cause one or more sets of assistant outputs 205 to be processed by LLM engines 150A1 and / or 150A2 to generate one or more sets of modified assistant outputs 206. In some implementations, one or more LLM outputs may be generated in advance in an offline manner (e.g., before receiving a stream of audio data 201 generated by one or more of the microphones of client device 110) using offline output modification engine 170, and the one or more LLM outputs may be stored in LLM output database 150A. As described with respect to FIG. 3, one or more LLM outputs may be pre-indexed in LLM output database 150A based on a corresponding assistant query and / or a corresponding context of a corresponding interaction session in which the corresponding assistant query was received. Furthermore, LLM engine 150A1 and / or 150A2 can determine that an assistant query included in an utterance captured in the stream of audio data 201 matches the corresponding assistant query, and / or that the context 202 of the interaction session in which the assistant query is received matches the corresponding context of the corresponding interaction session in which the corresponding assistant query was received. LLM engine 150A1 and / or 150A2 can obtain one or more LLM outputs indexed by the corresponding assistant query and / or the corresponding context that matches the assistant query to modify a set of one or more assistant outputs 205. Moreover, the set of one or more assistant outputs 205 may be modified based on one or more of the LLM outputs, thereby obtaining a set of one or more modified assistant outputs 206.

[0047] In additional or alternative implementations, the one or more LLM outputs may be generated in an online manner (e.g., in response to receiving a stream of audio data 201 generated by one or more microphones of client device 110) using online output modification engine 180. As described with respect to FIGS. 4 and 5 , the one or more LLM outputs may be generated based on processing one or more sets of assistant outputs 205, recognized text corresponding to an assistant query included in an utterance captured in the stream of audio data 201 (e.g., included in the stream of ASR output 203), and / or the context 202 of an interaction session between a user of client device 110 and automated assistant 115 to generate the one or more LLM outputs using one or more LLMs stored in ML model database 115A. Furthermore, the set of one or more assistant outputs 205 may be modified based on one or more of the LLM outputs, thereby resulting in one or more sets of modified assistant outputs 206. In other words, in these implementations, LLM engines 150A1 and / or 150A2 can directly generate a set of one or more modified assistant outputs 206 using one or more of the LLMs stored in ML model database 115A.

[0048] In these implementations, in contrast to the typical turn-based interaction sessions described above that do not utilize an LLM, ranking engine 190 may process the set of one or more assistant outputs 205 and the set of one or more modified assistant outputs 206 to rank each of the one or more assistant outputs included in both the set of one or more assistant outputs 205 and the set of one or more modified assistant outputs 206 according to one or more ranking criteria. Thus, when selecting one or more given assistant outputs 207, automated assistant 207 may select from the set of one or more assistant outputs 205 and the set of one or more modified assistant outputs 206. In particular, the assistant outputs included in the set of one or more modified assistant outputs 206 are generated based on the set of one or more assistant outputs 205 and may convey the same or similar information, but with additional information that is relevant to the context 202 of the interaction (e.g., as described with respect to FIG. 4 ) and / or more natural, fluent, and / or more suited to the personality of the automated assistant, such that the one or more given assistant outputs 207 resonate more with the user of client device 110.

[0049] The one or more ranking criteria may include, for example, one or more predicted measures indicating how responsive each of the assistant outputs included in the set of one or more assistant outputs 205 and the set of one or more modified assistant outputs 206 is predicted to be to an assistant query included in an utterance captured in the stream of audio data 201 (e.g., ASR measures generated by ASR engines 130A1 and / or 130A2 when generating the stream of ASR outputs 203, NLU measures generated by NLU engines 140A1 and / or 140A2 when generating the stream of NLU outputs 204, fulfillment measures generated by one or more of 1P system 191 and / or 3P system 192), one or more intents included in the stream of NLU outputs 204, measures derived from a classifier processing each of the assistant outputs included in the set of one or more assistant outputs 205 and the set of one or more modified assistant outputs 206 to determine how natural, fluent, and / or consistent with the personality of the automated assistant each of the assistant outputs is when provided for presentation to a user, and / or other ranking criteria. For example, if the intent of the user of client device 110 indicates that the user desires a factual answer (e.g., based on providing an utterance that includes the assistant query "why is the sky blue?"), the user is likely to desire a simple answer to the assistant query, and ranking engine 190 can weight one or more of the assistant outputs included in the set of one or more assistant outputs 205. However, if the intent of the user of client device 110 indicates that the user provided an open-ended input (e.g., based on providing an utterance that includes the assistant query "what time is it?"), the user is likely to prefer a more conversational modality, and ranking engine 190 can weight one or more of the assistant outputs included in the set of one or more modified assistant outputs 206.

[0050] 1 and 2 are described herein with respect to a voice-based interaction session, it should be understood that this is for purposes of illustration and not intended to be limiting. Rather, it should be understood that the techniques described herein can be utilized regardless of the user's input modality. For example, in some implementations in which a user provides typed and / or touch input as an assistant query, automated assistant 115 can process the typed input using NLU engine 140A1 and / or 140A2 to generate a stream of NLU output 204 (e.g., skipping processing the stream of audio data 201), and LLM engine 150A1 and / or 150A2 can utilize text input (e.g., derived from the typed and / or touch input) corresponding to the assistant query in generating a set of one or more modified assistant outputs 206 in the same or similar manner as described above.

[0051] Turning now to FIG. 3 , a flowchart illustrating an example method 300 of utilizing a large-scale language model in generating assistant output in an offline manner for later use in an online manner is shown. For convenience, the operations of method 300 are described with reference to a system that performs the operations from process flow 200 of FIG. 2 . This system of method 300 includes one or more processors, memory, and / or other components of a computing device (e.g., client device 110 of FIG. 1 , client device 610 of FIG. 6 , and / or computing device 710 of FIG. 7 , one or more servers, and / or other computing devices). Moreover, while the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0052] In block 352, the system obtains multiple assistant queries directed to the automated assistant and corresponding contexts of corresponding previous interaction sessions for each of the multiple assistant queries. For example, the system can cause the assistant activity engine 171 of the offline output modification engine of FIGS. 1 and 2 to obtain the multiple assistant queries and corresponding contexts of previous interaction sessions in which the multiple assistant queries were received, for example, from the assistant activity database 170A shown in FIG. 1. In some implementations, the multiple assistant queries and corresponding contexts of previous interaction sessions in which the multiple assistant queries were received can be limited to those associated with a user of a client device (e.g., a user of client device 110 of FIG. 1). In other implementations, the multiple assistant queries and corresponding contexts of previous interaction sessions in which the multiple assistant queries were received can be limited to those associated with multiple users of the respective client devices (e.g., which may or may not include a user of client device 110 of FIG. 1).

[0053] At block 354, the system processes a given assistant query of the plurality of assistant queries using one or more LLMs to generate one or more corresponding LLM outputs, each of which is predicted to be responsive to the given assistant query. Each of the one or more corresponding LLM outputs may include, for example, a probability distribution over a set of one or more words and / or phrases across one or more vocabularies, where one or more of the sets of words and / or phrases may be selected as the one or more corresponding LLM outputs based on the probability distribution. In various implementations, in generating one or more corresponding LLM outputs for a given assistant query, the system can use one or more of the LLMs to process the assistant query along with the corresponding context of a corresponding previous interaction session in which the given assistant query was received, and / or a set of assistant outputs predicted to be responsive to the given assistant query (e.g., generated based on processing audio data corresponding to the given assistant query using one or more of ASR engines 130A1 and / or 130A2, NLU engines 140A1 and / or 140A2, and 1P system 191 and / or 3P system 192 as described with respect to FIG. 2). In some implementations, the system can process recognized text corresponding to the given assistant query, and in additional or alternative implementations, the system can process audio data capturing speech including the given assistant query. In some implementations, the system can have LLM engine 150A1 process a given assistant query using one or more of the LLMs locally at a client device of a user (e.g., a user of client device 110 in FIG. 1), while in other implementations, the system can have LLM engine 150A2 process a given assistant query using one or more of the LLMs remotely from the user's client device (e.g., at a remote server).As described herein, one or more corresponding LLM outputs can reflect a more natural conversational output than a typical assistant output that may be provided by an automated assistant, which allows the automated assistant to more fluently lead an interaction session, such that an assistant output that is modified based on one or more of the corresponding LLM outputs is likely to resonate more with a user who perceives the modified assistant output.

[0054] In some implementations, in addition to the one or more corresponding LLM outputs, additional assistant queries may be generated using one or more of the LLM models based on processing the given assistant query and / or the corresponding context of the corresponding previous interaction session in which the given assistant query was received. For example, when processing the given assistant query and / or the corresponding context of the corresponding previous interaction session in which the given assistant query was received, one or more of the LLMs can determine an intent related to the given assistant query (e.g., based on the stream of NLU output 204 generated using NLU engines 140A1 and / or 140A2 of FIG. 2). Furthermore, based on the intent related to the given assistant query, one or more of the LLMs can identify at least one related intent related to the intent associated with the given assistant query (e.g., based on mapping the intent to at least one related intent in a database or memory accessible to client device 110 and / or based on processing the intent related to the given assistant query using rules defined in one or more machine learning (ML) models or heuristics). Moreover, one or more of the LLMs can generate additional assistant queries based on at least one related intent. For example, assume that an assistant query indicates that the user has not yet eaten dinner (e.g., a given assistant query of "I'm feeling pretty hungry" received in the evening at the user's physical location as indicated by a corresponding context of a corresponding previous interaction session related to the user's intent indicating a desire to eat).In this example, the additional assistant queries may correspond to, for example, "what types of cuisine has the user indicated he / she prefers?" (e.g., reflecting a related cuisine type intent associated with the user's intent to indicate a desire to eat), "what restaurants nearby are open?" (e.g., reflecting a related restaurant search intent associated with the user's intent to indicate a desire to eat), and / or other additional assistant queries.

[0055] In these implementations, the additional assistant output may be determined based on processing the additional assistant query. In the above example where the additional assistant query is "What types of cuisine has the user indicated he / she prefers?", user profile data from one or more of the 1P systems 191 stored locally on the client device 110 may be utilized to determine that the user indicated a preference for Mediterranean and Indian cuisine. Based on the user profile data indicating that the user prefers Mediterranean and Indian cuisine, one or more corresponding LLM outputs may be modified to ask the user whether Mediterranean and / or Indian cuisine appeals to the user (e.g., "How does Mediterranean cuisine or Indian cuisine sound for dinner").

[0056] In the above example where the additional assistant query is "what restaurants nearby are open?", restaurant data from one or more of the 1P system 191 and / or the 3P system 192 may be utilized to determine which restaurants are open near the user's primary residence (and optionally limited to restaurants serving Mediterranean and Indian cuisine based on user profile data). Based on the results, one or more corresponding LLM outputs may be modified to provide the user with a list of one or more restaurants that are open near the user's primary residence (e.g., "Example Mediterranean Restaurant is open until 9:00 PM and Example Indian Restaurant is open until 10:00 PM"). Notably, the additional assistant queries initially generated using the LLM (e.g., "what types of cuisine has the user indicated he / she prefers?" and "what types of cuisine has the user indicated he / she prefers?" in the above example) may not be included in one or more corresponding LLM outputs and, as a result, may not be offered for presentation to the user. Rather, additional assistant outputs determined based on additional assistant queries (e.g., "how does Mediterranean cuisine or Indian cuisine sound for dinner" and "Example Mediterranean Restaurant is open until 9:00 PM and Example Indian Restaurant is open until 10:00 PM") may be included in one or more corresponding LLM outputs and, as a result, may be provided for presentation to the user.

[0057] In additional or alternative implementations, each of the one or more corresponding LLM outputs (and optionally additional assistant outputs determined based on additional assistant queries) may be generated using a corresponding set of parameters from multiple distinct sets of one or more parameters of the LLM. Each of the multiple distinct sets of parameters may be associated with a distinct personality for the automated assistant. In some versions of these implementations, a single LLM may be utilized to generate the one or more corresponding LLM outputs using a corresponding set of parameters for each of the distinct personalities, while other versions of these implementations may utilize multiple LLMs to generate the one or more corresponding LLM outputs using a corresponding set of parameters for each of the distinct personalities. For example, a single LLM may be utilized to generate a first LLM output using a first set of parameters reflecting a first personality (e.g., the personality of the chef in the above example where the given assistant query corresponds to "I'm feeling pretty hungry"), generate a second LLM output using a second set of parameters reflecting a second personality (e.g., the personality of the butler in the above example where the given assistant query corresponds to "I'm feeling pretty hungry"), and do the same for multiple other separate personalities. Also, for example, a first LLM may be utilized to generate a first LLM output using a first set of parameters reflecting a first personality (e.g., the personality of the chef in the above example where the given assistant query corresponds to "I'm feeling pretty hungry"), and a second LLM may be utilized to generate a second LLM output using a second set of parameters reflecting a second personality (e.g., the personality of the butler in the above example where the given assistant query corresponds to "I'm feeling pretty hungry"), and similarly for multiple other separate personalities.Thus, when the corresponding LLM output is provided for presentation to the user, it can reflect various changing contextual personalities via the different personalities' prosodic qualities (e.g., the intonation, inflection, tone, pauses, tempo, stress, rhythm, etc.) Additionally or alternatively, the user can define one or more personalities to be utilized by the automated assistant (e.g., via settings in an automated assistant application associated with the automated assistant described herein) in a consistent manner (e.g., always use the butler personality) and / or in a contextual manner (e.g., use the butler personality in the morning and evening, but a different personality during the day).

[0058] In particular, the replies of these personalities described herein may reflect not only the prosodic qualities of the different personalities, but also the vocabulary of the different personalities and / or the distinct speaking styles of the different personalities (e.g., verbose speaking style, terse speaking style, kind personality, sarcastic personality, etc.). For example, the chef personality described above may have a particular chef vocabulary, so that a probability distribution over a sequence of one or more words and / or phrases for one or more corresponding LLM outputs generated using a set of parameters for the chef personality may favor the sequence of words and / or phrases used by the chef over other sequences of words and / or phrases for other personalities (e.g., a scientist personality, a librarian personality). Thus, when one or more of the corresponding LLM outputs are provided for presentation to a user, it can reflect various changing contextual personalities not only in terms of the different personalities' prosodic qualities but also in terms of the different personalities' precise and realistic vocabulary, so that the one or more corresponding LLM outputs will better resonate with the user in various contextual scenarios. Moreover, it should be understood that the vocabulary and / or speaking styles of the different personalities can be defined with varying degrees of granularity. Continuing with the above example, the chef personality described above may have a specific Mediterranean chef vocabulary when asking about Mediterranean cuisine based on the additional assistant query being related to Mediterranean cuisine, a specific Indian chef vocabulary when asking about Indian cuisine based on the additional assistant query being related to Indian cuisine, and so on.

[0059] In block 356, the system indexes one or more of the corresponding LLM outputs in a memory accessible at the client device (e.g., LLM output database 150A of FIG. 1 ) based on the given assistant query and / or corresponding context of a corresponding previous interaction session for the given assistant query. For example, the system may cause indexing engine 172 of offline output modification engine 172 of FIGS. 1 and 2 to index one or more corresponding LLM outputs in LLM output database 150A. In some implementations, indexing engine 172 can index one or more corresponding LLM outputs based on one or more terms included in the given assistant query and / or one or more contextual signals included in the corresponding context of a corresponding previous interaction session in which the given assistant query was received. In additional or alternative implementations, indexing engine 172 can generate an embedding (e.g., a word2vec embedding or any other low-dimensional representation) of a given assistant query and / or an embedding of one or more contextual signals included in the corresponding context of a corresponding previous interaction session in which a given assistant query was received, and one or more of these embeddings can be mapped to an embedding space (e.g., a low-dimensional space). In these implementations, the one or more corresponding LLM outputs indexed in LLM output database 150A can later be utilized by online output modification engine 180 to modify the set of assistant outputs (e.g., as described below with respect to blocks 362, 364, and 366 of method 300 of FIG. 3).In various implementations, the system may additionally or alternatively generate one or more assistant outputs (e.g., one or more assistant outputs 205 as described with respect to FIG. 2) that are predicted to respond to a given assistant query (but are not generated using LLM engines 150A1 and / or 150A2), and one or more corresponding LLM outputs may additionally or alternatively be indexed by the one or more assistant outputs.

[0060] In some implementations, as indicated at block 358, the system may optionally receive user input for evaluating and / or modifying one or more of the corresponding LLM outputs. For example, a human evaluator may analyze one or more corresponding LLM outputs generated using one or more LLM models and modify one or more of the corresponding LLM outputs by changing one or more terms and / or phrases included in the one or more corresponding LLM outputs. Also, for example, the human evaluator may reindex, discard, and / or otherwise modify the index of one or more corresponding LLM outputs. Thus, in these implementations, one or more corresponding LLM outputs generated using one or more LLMs may be selected by the human evaluator to ensure the quality of the one or more corresponding LLM outputs. Moreover, any non-discarded, reindexed, and / or selected LLM outputs may be utilized to modify or retrain the LLM in an offline manner.

[0061] In block 360, the system determines whether there are additional assistant queries included in the plurality of assistant queries obtained in block 352 that have not been processed using one or more of the LLMs. If, in a repetition of block 360, the system determines that there are additional assistant queries included in the plurality of assistant queries obtained in block 352 that have not been processed using one or more of the LLMs, the system returns to block 354 and performs additional repetitions of blocks 354 and 356 with respect to the additional assistant queries rather than the given assistant query. These operations may be repeated for each assistant query included in the plurality of assistant queries obtained in block 352. In other words, the system may index one or more corresponding LLM outputs for each of the assistant queries and / or corresponding contexts of corresponding previous interaction sessions in which a corresponding one of the plurality of assistant queries is received before utilizing those one or more corresponding LLM outputs.

[0062] If, in a repetition of block 360, the system determines that there are no additional assistant queries included in the plurality of assistant queries obtained in block 352 that have not been processed using one or more of the LLMs, the system may proceed to block 362. In block 362, the system may monitor a stream of audio data generated by one or more microphones of the client device to determine whether the stream of audio data captures speech of a user of the client device directed to the automated assistant. For example, the system may monitor one or more specific words or phrases included in the stream of audio data (e.g., may monitor one or more specific words or phrases that invoke an automated assistant using a hotword detection model). Also, for example, the system may monitor speech directed to the client device, optionally in addition to one or more other signals (e.g., one or more gestures captured by a visual sensor of the client device, gaze directed toward the client device, etc.). In a repetition of block 362, if the system determines that the stream of audio data did not capture speech of a user of the client device directed to the automated assistant, the system may continue monitoring the stream of audio data in block 362. If, in an iteration of block 362, the system determines that the stream of audio data captures the speech of a user of a client device directed to an automated assistant, the system may proceed to block 364.

[0063] In block 364, the system determines, based on processing the stream of audio data, that the utterance includes a current assistant query corresponding to one of the plurality of assistant queries, and / or that the utterance is received in a current context of a current interaction session corresponding to a corresponding context of a corresponding previous interaction session for one of the plurality of assistant queries. For example, the system can process the stream of audio data (e.g., the stream of audio data 201 in FIG. 2) using ASR engines 130A1 and / or 130A2 to generate a stream of ASR output (e.g., the stream of ASR output 203 in FIG. 2). Further, the system can process the stream of ASR output using NLU engines 140A1 and / or 140A2 to generate a stream of NLU output (e.g., the stream of NLU output 204 in FIG. 2). Furthermore, based on the stream of ASR output and / or the stream of NLU output, the system can identify a current assistant query. In some implementations, the system can further determine a set of one or more assistant outputs (e.g., one or more assistant outputs 205 in FIG. 2 ) by having one or more of the 1P system 191 and / or 3P system process the stream of ASR outputs and / or the stream of NLU outputs.

[0064] In some implementations of method 300 of FIG. 3 , the system can utilize online output modification engine 180 to determine that one or more terms or phrases of the current assistant query correspond to one or more terms or phrases of one of the multiple assistant queries for which one or more corresponding LLM outputs are indexed (e.g., using any known technique for determining whether terms or phrases correspond to one another, such as exact matching, soft matching, edit distance, phonetic similarity, embedding, etc.). In response to determining that one or more terms or phrases of the current assistant query correspond to one or more terms or phrases of one of the multiple assistant queries, the system can retrieve (e.g., from LLM output database 150A) one or more of the corresponding LLM outputs that are indexed and associated with one of the multiple assistant queries in the iteration of block 356. For example, if the current query includes the phrase “I'm hungry,” the system can retrieve one or more of the corresponding LLM outputs generated in the above example described for the given assistant query. For example, the system can determine that both the current query and the given assistant query described above both include the phrase "I'm hungry" based on comparing the edit distance between the terms of the current query and the terms of the given assistant query. Also, for example, the system can generate an embedding of the current assistant query and map the embedding of the current query to the embedding space described above with respect to block 356. Furthermore, the system can determine that the current assistant query corresponds to the given assistant query based on the distance in the embedding space between the generated embedding of the current query and the previously generated embedding for the given assistant query satisfying a distance threshold.

[0065] 3 , the system can utilize online output modification engine 180 to determine that one or more contextual signals detected when the current assistant query is received correspond to one or more corresponding contextual signals when one of the multiple assistant queries is received (e.g., received on the same day of the week, received at the same time, received at the same location, with the same ambient noise present in the client device's environment, received in a particular sequence of utterances during an interaction session, etc.). In response to determining that one or more of the contextual signals associated with the current assistant query correspond to one or more contextual signals of one of the multiple assistant queries, the system can retrieve (e.g., from LLM output database 150A) one or more corresponding LLM outputs indexed in the repetition of block 356 and associated with one of the multiple assistant queries. For example, if the current query is received in the evening at the user's primary residence, the system can retrieve one or more of the corresponding LLM outputs generated in the above example described for the given assistant query. For example, the system can determine that both the described current query and the given assistant query are associated with a temporal context signal of "evening" and a location context signal of "primary location." Also, for example, the system can generate embeddings of one or more context signals related to the current assistant query and map the embeddings of the one or more context signals related to the current query to the embedding space described above with respect to block 356. Furthermore, the system can determine that one or more context signals related to the current assistant query correspond to one or more context signals related to the given assistant query based on the distance in the embedding space between the generated embeddings of the one or more context signals related to the current query and the previously generated embeddings for one or more context signals related to the given assistant query satisfying a threshold distance.

[0066] In particular, the system can utilize one or both of the current assistant query and the context of the interaction session in which the current assistant query is received (e.g., one or more detected contextual signals) in determining one or more corresponding LLM outputs to be utilized in generating one or more current assistant outputs to be provided for presentation to the user in response to the current assistant query. In various implementations, the system can additionally or alternatively utilize one or more assistant outputs generated for the current assistant query in determining one or more corresponding LLM outputs to be utilized in generating one or more current assistant outputs to be provided for presentation to the user in response to the current assistant query. For example, the system can additionally or alternatively generate one or more assistant outputs (e.g., one or more assistant outputs 205 as described with respect to FIG. 2 ) predicted to be responsive to the current assistant query (but not generated using LLM engines 150A1 and / or 150A2) and determine that one or more of the assistant outputs predicted to be responsive to the current assistant query correspond to one or more previously generated assistant outputs for one of the multiple assistant queries using the various techniques described above.

[0067] In block 366, the system causes the automated assistant to utilize one or more of the corresponding LLM outputs in generating one or more current assistant outputs to be provided for presentation to a user of the client device. For example, the system can rank one or more assistant outputs (e.g., one or more assistant outputs 205 as described with reference to FIG. 2) and one or more corresponding LLM outputs (e.g., one or more modified assistant outputs 206 as described with reference to FIG. 2) according to one or more ranking criteria. Further, the system can select one or more current assistant outputs from among the one or more assistant outputs and one or more corresponding LLM outputs. Moreover, the system can cause the one or more current assistant outputs to be visually and / or audibly rendered for presentation to a user of the client device.

[0068] 3 is described as generating one or more corresponding LLM outputs in an offline manner (e.g., by utilizing offline output modification engine 170 to generate corresponding LLM outputs using one or more of the LLMs and indexing the one or more corresponding LLM outputs in LLM output database 150A) and then utilizing the one or more corresponding LLM outputs in an online manner (e.g., by utilizing online output modification engine 180 to determine which LLM outputs to utilize from LLM output database 150A based on the current assistant query), it should be understood that this is for illustrative purposes and not intended to be limiting. For example, as described below with respect to FIGS. 4 and 5, online output modification engine 180 can additionally or alternatively utilize LLM engines 150A1 and / or 150A2 in a preferred implementation in an online manner.

[0069] Turning now to FIG. 4 , a flowchart illustrating an example method 400 of utilizing a large-scale language model in generating an assistant output based on generating an assistant query is shown. For convenience, the operations of method 400 are described with reference to a system that performs the operations from process flow 200 of FIG. 2 . This system of method 400 includes one or more processors, memory, and / or other components of a computing device (e.g., client device 110 of FIG. 1 , client device 610 of FIG. 6 , and / or computing device 710 of FIG. 7 , one or more servers, and / or other computing devices). Moreover, while the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0070] In block 452, the system receives a stream of audio data capturing a user's speech, the speech including an assistant query directed to the automated assistant, the speech received during an interaction session between the user and the automated assistant. In some implementations, the system may only process the stream of audio data to determine that the system captures an assistant query in response to determining that one or more conditions are met. For example, the system may monitor one or more specific words or phrases included in the stream of audio data (e.g., may monitor one or more specific words or phrases that invoke the automated assistant using a hotword detection model). Also, for example, the system may monitor speech directed toward the client device, optionally in addition to one or more other signals (e.g., one or more gestures captured by a visual sensor of the client device, gaze directed toward the client device, etc.).

[0071] In block 454, the system determines a set of Assistant outputs based on processing the stream of audio data, where each of the Assistant outputs included in the set responds to the Assistant query included in the utterance. For example, the system can process the stream of audio data (e.g., the stream of audio data 201) using ASR engines 130A1 and / or 130A2 to generate a stream of ASR output (e.g., ASR output 203). Further, the system can process the stream of ASR output (e.g., the stream of audio data 201) using NLU engines 140A1 and / or 140A2 to generate a stream of NLU output (e.g., the stream of NLU output 204). Furthermore, the system can have one or more 1P systems 191 and / or 3P systems 192 process the stream of NLU output (e.g., the stream of NLU output 204) to generate a set of Assistant outputs (e.g., the set of Assistant outputs) 205. In particular, the set of assistant outputs may correspond to one or more candidate assistant outputs that the automated assistant may consider using in responding to the utterance without the techniques described herein (i.e., techniques that do not utilize LLM engines 150A1 and / or 150A2 in modifying the assistant output as described herein).

[0072] In block 456, the system processes the set of assistant outputs and the context of the interaction session to (1) generate a set of modified assistant outputs using one or more LLM outputs, each of which is determined at least based on the context of the interaction session and / or one or more assistant outputs included in the set of assistant outputs, and (2) generate an additional assistant query associated with the utterance based at least in part on the context of the interaction session and at least in part on the assistant query included in the utterance. In various implementations, each of the LLM outputs may be further determined based on an assistant query included in the utterance captured in the stream of audio data. In some implementations, when generating the set of modified assistant outputs, one or more LLM outputs may have been generated in advance in an offline manner (e.g., using offline output modification engine 170 before receiving the utterance, as described above with respect to FIG. 3). In these implementations, the system can determine that an assistant query included in an utterance captured in the stream of audio data corresponds to a previous assistant query for which one or more LLM outputs have previously been generated, that the context of the interaction session corresponds to a previous context of a previous interaction session in which the previous assistant query was received, and / or that one or more of the assistant outputs included in the set of assistant outputs determined in block 454 correspond to one or more previous assistant outputs determined based on the previous query. Further, the system can obtain (e.g., using online output modification engine 180) one or more LLM outputs indexed based on the previous assistant query, the previous context, and / or one or more previous assistant outputs corresponding to the assistant query, the context, and / or one or more of the assistant outputs included in the set of assistant outputs, as described with respect to method 300 of FIG. 3, and use the one or more LLM outputs as the set of modified assistant outputs.

[0073] In additional or alternative implementations, when generating the set of modified assistant outputs, the system can at least process the context of the interaction session and / or one or more assistant outputs included in the set of assistant outputs in an online manner (e.g., using online output modification engine 180 in response to receiving an utterance) to generate one or more LLM outputs. For example, the system can cause LLM engine 150A1 and / or 150A2 to process the context of the interaction session, the assistant query, and / or one or more assistant outputs included in the set of assistant outputs using one or more LLMs to generate the set of modified assistant outputs. The one or more LLM outputs can be generated in an online manner in the same or similar manner as described above with respect to block 354 of method 300 of FIG. 3 with respect to generating one or more LLM outputs in an offline manner, but in response to receiving an utterance at the client device. For example, the system may process one or more assistant outputs included in the set of assistant outputs using one or more LLMs to generate one or more personality responses for each of the one or more assistant outputs included in the set of assistant outputs, as described above with respect to block 354 of method 300 of FIG. 3. In other words, each of the assistant outputs included in the set of assistant outputs may have a limited vocabulary and a consistent personality with respect to the prosodic properties associated with each of the assistant outputs. However, when processing each of the assistant outputs included in the set of assistant outputs to generate a set of modified assistant outputs, each of the modified assistant outputs may have a much larger vocabulary depending on the use of one or more LLMs in generating the one or more modified assistant outputs, and there may be much greater variation in the prosodic properties associated with each of the modified assistant outputs.As a result, each of the modified Assistant outputs may correspond to a contextually relevant Assistant output that is more resonant with a user engaged in an interaction session with the automated assistant.

[0074] Similarly, in some implementations, when generating the additional assistant query, the additional assistant may have been generated in advance in an offline manner (e.g., using offline output modification engine 170 before receiving the utterance, as described above with respect to FIG. 3). In these implementations, similar to what was described above with respect to obtaining one or more LLM outputs that were generated in advance in an offline manner, the system can obtain the additional assistant query that is indexed based on a previous assistant query, a previous context, and / or one or more previous assistant outputs that correspond to the assistant query, the context, and / or the assistant query, each corresponding to one or more of the assistant outputs included in the set of assistant outputs, as described with respect to method 300 of FIG. 3, and can utilize the previously generated additional assistant query as the additional assistant query.

[0075] Also similarly, in additional or alternative implementations, when generating the additional assistant query, the system can process at least the context of the interaction session and / or one or more assistant outputs included in the set of assistant outputs in an online manner (e.g., using online output modification engine 180 in response to receiving an utterance) to generate the additional assistant query. For example, the system can have LLM engine 150A1 and / or 150A2 process the context of the interaction session, the assistant query, and / or one or more assistant outputs included in the set of assistant outputs using one or more LLMs to generate the additional assistant query. The additional assistant query can be generated in an online manner in the same or similar manner as described above with respect to block 354 of method 300 of FIG. 3 with respect to generating the additional assistant query in an offline manner, but in response to receiving an utterance at the client device. In some implementations, one or more LLMs described herein can have multiple separate layers dedicated to performing several functions. For example, one or more first layers of one or more LLMs may be utilized in generating the personality answers described herein, and one or more second layers of one or more LLMs may be utilized in generating the additional assistant queries described herein. In additional or alternative implementations, one or more LLMs may communicate with one or more additional layers not included in one or more LLMs in generating the additional assistant queries described herein. For example, one or more layers of one or more LLMs may be utilized in generating the personality answers described herein, and one or more additional layers of another ML model in communication with one or more LLMs may be utilized in generating the additional assistant queries described herein. Non-limiting examples of personality answers and additional assistant queries are described in more detail below with respect to FIG. 6.

[0076] In block 458, the system determines, based on the additional assistant query, an additional assistant output responsive to the additional assistant query. In some implementations, the system can cause the additional assistant query to be processed by one or more of 1P system 191 and / or 3P system 192 in the same or similar manner as described with respect to processing the assistant query in FIG. 2 to generate the additional assistant output. In some implementations, the additional assistant output may be a single additional assistant output, while in other implementations, the additional assistant output may be included in a set of additional assistant outputs (e.g., similar to the set of additional assistant outputs 205 in FIG. 2). In additional or alternative implementations, the additional assistant query may be directly mapped to the additional assistant output based, for example, on user profile data of the user who provided the utterance and / or any other data accessible to the automated assistant.

[0077] At block 460, the system processes the set of modified assistant outputs based on the additional assistant outputs responsive to the additional assistant queries to generate an additional set of modified assistant outputs. In some implementations, the system may add the additional assistant output to the beginning or end of one or more of the modified assistant outputs to each of the one or more assistant outputs included in the set of modified assistant outputs generated at block 456. In additional or alternative implementations, as shown at block 460A, the system may process the additional assistant outputs and the context of the interaction session to generate an additional set of modified assistant outputs using one or more of the LLM outputs utilized at block 456 and / or one or more additional LLM outputs, in addition to the one or more LLM outputs utilized at block 456, where the additional assistant outputs are generated based at least in part on the context of the interaction session. The set of additional modified assistant outputs may be generated in the same or similar manner as described above with respect to generating the set of modified assistant outputs, but based on the additional assistant outputs rather than the set of assistant outputs (e.g., using one or more LLM outputs generated in an offline manner and / or using LLM engines 150A1 and / or 150A2 in an online manner).

[0078] In block 462, the system causes a given modified assistant output from the set of modified assistant outputs and / or a given additional modified assistant output from the set of additional modified assistant outputs to be provided for presentation to the user. In some implementations, the system may cause ranking engine 190 to rank each of the one or more modified assistant outputs included in the set of modified assistant outputs (and optionally each of the one or more assistant outputs included in the set of assistant outputs) according to one or more ranking criteria and select a given modified assistant output from the set of modified assistant outputs (or a given assistant output from the set of assistant outputs). Furthermore, the system may further cause ranking engine 190 to rank each of the one or more additional modified assistant outputs (and optionally an additional assistant output) included in the set of additional modified assistant outputs according to one or more ranking criteria and select a given additional modified assistant output (or an additional assistant output as a given additional assistant output) from the set of additional modified assistant outputs. In these implementations, the system can combine the given modified Assistant output and the given additional Assistant output such that the given modified Assistant output and the given additional Assistant output are provided for visual and / or audible presentation to a user of a client device involved in an interaction session with the automated assistant.

[0079] Turning now to FIG. 5 , a flowchart illustrating an example method 500 of utilizing a large-scale language model in generating an assistant output based on generated assistant personality responses is shown. For convenience, the operations of method 500 are described with reference to a system that performs the operations from process flow 200 of FIG. 2 . This system of method 500 includes one or more processors, memory, and / or other components of a computing device (e.g., client device 110 of FIG. 1 , client device 610 of FIG. 6 , and / or computing device 710 of FIG. 7 , one or more servers, and / or other computing devices). Moreover, while the operations of method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0080] In block 552, the system receives a stream of audio data capturing a user's speech, the utterance including an assistant query directed to the automated assistant, the utterance received during an interaction session between the user and the automated assistant. In block 554, the system determines a set of assistant outputs based on processing the stream of audio data, each of the assistant outputs included in the set responding to the assistant query included in the utterance. The operations of blocks 552 and 554 of method 500 of FIG. 5 may be performed in the same or similar manner as described with respect to blocks 452 and 454 of method 400 of FIG. 4, respectively.

[0081] In block 556, the system determines whether to modify one or more assistant outputs included in the set of assistant outputs. The system can determine whether to modify one or more assistant outputs based on, for example, the user's intent in providing the utterance (e.g., included in the stream of NLU output 204), the one or more assistant outputs included in the set of assistant outputs (e.g., the set of assistant outputs 205), one or more computational costs (e.g., battery consumption, processor consumption, latency, etc.) associated with modifying one or more of the assistant outputs included in the set of assistant outputs, the length of time interacting with the automated assistant, and / or other considerations. For example, if the user's intent indicates that the user who provided the utterance expects a quick and / or factual answer (e.g., "why is the sky blue?", "what's the weather?", "what time is it," etc.), in some cases, the system may decide not to modify one or more of the assistant outputs to reduce latency and consumption of computational resources in providing content responsive to the utterance. Also, for example, if a user's client device is in a power-saving mode, the system may decide not to modify one or more of the Assistant outputs in order to conserve battery power. Also, for example, if a user is engaged in an interaction with a session that exceeds a threshold time length (e.g., 30 seconds, 1 minute, etc.), the system may decide not to modify one or more of the Assistant outputs in an attempt to conclude the interaction session in a quicker and more efficient manner.

[0082] If, in the iteration of block 556, the system determines not to modify one or more of the assistant outputs included in the set of assistant outputs, the system may proceed to block 558. In block 558, the system causes a given assistant output from the set of assistant outputs to be provided for presentation to the user. For example, the system may cause ranking engine 190 to rank each of the assistant outputs included in the set of assistant outputs according to one or more ranking criteria, and select a given assistant output to be provided for visual and / or audible presentation to the user based on the ranking.

[0083] If, in the repetition of block 556, the system determines to modify one or more of the assistant outputs included in the set of assistant outputs, the system may proceed to block 560. In block 560, the system processes the set of assistant outputs and the context of the interaction session to generate a set of modified assistant outputs using one or more LLM outputs, each of the one or more LLM outputs being determined based on the context of the interaction session and / or the one or more assistant outputs included in the set of assistant outputs, each of the one or more LLM outputs reflecting a corresponding personality of the automated assistant from among a plurality of distinct personalities. As described with respect to block 354 of method 300 of FIG. 3, the one or more LLM outputs may be generated using various distinct parameters to reflect different personalities of the automated assistant. In various implementations, each of the LLM outputs may be further determined based on an assistant query included in an utterance captured in the stream of audio data. In some implementations, the system may process the set of assistant outputs and the context of the interaction session (and optionally the assistant query) to generate a modified set of assistant outputs using one or more of the pre-generated LLM outputs in an offline manner as described herein, while in additional or alternative implementations, the system may process the set of assistant outputs and the context of the interaction session (and optionally the assistant query) to generate a modified set of assistant outputs in an online manner as described herein. As discussed above with respect to method 400 of FIG. 4, non-limiting examples of personality replies and additional assistant queries are described in more detail below with respect to FIG. 6.

[0084] In block 562, the system causes a given assistant output from the set of assistant outputs to be provided for presentation to the user. In some implementations, the system may cause the ranking engine 190 to rank each of the one or more modified assistant outputs included in the set of modified assistant outputs (and optionally each of the one or more assistant outputs included in the set of assistant outputs) according to one or more ranking criteria, and select a given modified assistant output from the set of modified assistant outputs (or a given assistant output from the set of assistant outputs). Further, the system may cause the given modified assistant output to be provided for visual and / or audible presentation to the user.

[0085] While FIG. 5 is not described with respect to generating additional assistant queries, it should be understood that this is for illustrative purposes and is not intended to be limiting. Rather, it should be understood that generating additional assistant queries in method 400 of FIG. 4 is based on determining that there are contextually relevant additional assistant outputs that can be provided to facilitate an interaction session in order to provide a more natural conversational experience for the user, thereby making any assistant output provided for presentation to the user more reflective of a human interaction session and making the interaction session between the user and the automated assistant more resonant with the user. Moreover, while FIGS. 3 and 4 are not described with respect to determining whether to modify the set of assistant outputs, it should be understood that this is for illustrative purposes and is not intended to be limiting. Rather, it should be understood that determining whether to cause modification of the set of assistant responses using one or more LLMs can be performed in any of the exemplary methods of FIGS. 3, 4, and 5.

[0086] 6, a non-limiting example of an interaction session between a user and an automated assistant is shown, in which the automated assistant utilizes one or more LLMs when generating an assistant output. As described herein, in some implementations, the automated assistant can utilize one or more LLM outputs previously generated in an offline manner to generate a set of modified assistant outputs (e.g., as described above with respect to method 300 of FIG. 3). For example, the automated assistant can determine that one or more LLM outputs have been previously generated for it, that a previous assistant query corresponds to the assistant query included in the utterance, that a previous context of the previous interaction session corresponds to the context of the interaction session between the user and the automated assistant in which the utterance is received, and / or that one or more previous assistant outputs correspond to one or more assistant outputs included in the set of assistant outputs for the assistant query included in the utterance. Furthermore, the automated assistant can obtain one or more LLM outputs indexed (e.g., in LLM output database 150A) according to one or more of the previous assistant query, the previous context, and / or the previous assistant output included in the previous set of assistant outputs, and utilize the one or more LLM outputs as the set of modified assistant outputs. In additional or alternative implementations, the automated assistant can cause the assistant query, the context of the interaction session, and / or one or more assistant outputs included in the set of assistant outputs to be processed using one or more LLMs to generate one or more LLM outputs to be utilized as a set of modified assistant outputs in an online manner (e.g., as described above with respect to method 400 of FIG. 4 and method 500 of FIG. 5). Thus, the non-limiting example of FIG. 6 is provided to illustrate how utilization of LLMs in accordance with the techniques described herein can result in improved, natural conversation between a user and an automated assistant.

[0087] A client device 610 (e.g., an example of client device 110 in FIG. 1 ) may include various user interface components including, for example, a microphone for generating audio data based on speech and / or other audible input, a speaker for audibly rendering synthesized speech and / or other audible output, and / or a display 680 for visually rendering visual output. Additionally, the display 680 of the client device 610 may include various system interface elements 681, 682, and 683 (e.g., hardware and / or software interface elements) with which a user of the client device 610 may interact in order to cause the client device 610 to perform one or more actions. The display 680 of the client device 610 allows the user to interact with content rendered on the display 680 by touch input (e.g., by pointing user input at the display 680 or a portion thereof (e.g., a text entry box (not shown), a keyboard (not shown), or another portion of the display 680)) and / or by spoken input (e.g., by selecting the microphone interface element 684, or by simply speaking without necessarily selecting the microphone interface element 684 at the client device 610 (i.e., the automated assistant uses one or more terms or phrases, gestures, gaze, mouth movements, lip movements, and / or other conditions to enable spoken input)). While the client device 610 shown in FIG. 6 is a mobile phone, it should be understood that this is for purposes of illustration and not intended to be limiting. For example, client device 610 may be a standalone speaker with a display, a standalone speaker without a display, a home automation device, an in-vehicle system, a laptop, a desktop computer, and / or any other device capable of running an automated assistant to engage in a human-to-computer interaction session with a user of client device 610.

[0088] For example, assume that a user of client device 610 provides utterance 652, "Hey Assistant, what time is it?" In this example, the automated assistant can cause audio data capturing utterance 652 to be processed using ASR engines 130A1 and / or 130A2 to generate a stream of ASR output. Further, the automated assistant can cause the stream of ASR output to be processed using NLU engines 140A1 and / or 140A2 to generate a stream of NLU output. Moreover, the automated assistant can cause the stream of NLU output to be processed by one or more of 1P system 191 and / or 3P system 192 to generate a set of one or more assistant outputs. The set of assistant outputs can include, for example, "8:30 AM," "Good morning, it's 8:30 AM," and / or any other output that conveys the current time to the user of client device 610.

[0089] 6, the automated assistant determines to modify one or more of the assistant outputs included in the set of assistant outputs to generate a modified set of assistant outputs. For example, the automated assistant can determine that the assistant query included in utterance 652 requests the automated assistant to provide the current time for presentation to the user. The automated assistant can determine that one or more LLM outputs have been previously generated for requesting the automated assistant to provide the current time for presentation to the user, that a previous instance of an assistant query corresponds to the assistant query included in utterance 652 of FIG. 6, that a previous context of a previous interaction session corresponds to the context of an interaction session between the user and the automated assistant of FIG. 6 (e.g., the user requests the automated assistant to provide the current time in the morning (and optionally makes the request by starting the interaction session), that client device 610 is located in a particular location, and / or other contextual signals), and / or that one or more previous assistant outputs correspond to one or more of the assistant outputs included in the set of assistant outputs for the assistant query included in utterance 652 of FIG. 6. Additionally, the automated assistant may obtain one or more LLM outputs indexed (e.g., in LLM output database 150A) according to one or more of a previous assistant query, a previous context, and / or a previous assistant output included in a previous set of assistant outputs, and utilize the one or more LLM outputs as a set of modified assistant outputs. Also, for example, the automated assistant may cause the assistant query, the context of the interaction session, and / or one or more of the assistant outputs included in the set of assistant outputs to be processed using one or more LLMs to generate one or more LLM outputs to be utilized as a set of modified assistant outputs in an online manner.

[0090] 6 , assume further that the automated assistant determines to provide a modified assistant output 654 for presentation to the user: “Good morning [User]! It's 8:30 AM. Any fun plans today?” and that the modified assistant output 654 is determined based on one or more of the LLM outputs. The modified assistant output 654 provided for presentation to the user is personalized or adapted to the user of the client device 610 and the context of the interaction session in that the modified assistant output 654 greets the user with a contextually appropriate greeting (e.g., “Good Morning”) and addresses the user of the client device 610 by name (e.g., "[User]"). In particular, one or more of the LLM outputs may include one or more corresponding substitute terms (e.g., as indicated by "[User]" in the modified assistant output 654) that may be filled in with user profile data accessible to the automated assistant. It should be understood that while the example of FIG. 6 includes a corresponding substitute term for the name of the user of the client device 610, this is for illustrative purposes and is not intended to be limiting. For example, one or more of the corresponding proxy terms may be filled in with any data accessible to the automated assistant, such as a smart network-connected device identifier (e.g., smart lighting, smart TV, smart appliance, smart speaker, smart door lock, etc.), a known location associated with the user of the client device 610 (e.g., a city, state, county, region, area, country, place of work, the user's office, or the physical address of the primary residence of the user of the client device 610, etc.), entity criteria (e.g., reference people, places, things, etc.), software applications accessible on the user's client device 610, and / or any other data accessible to the automated assistant.

[0091] Moreover, modified assistant output 654 functions in response to the assistant query included in utterance 652 (e.g., "It's 8:30 AM"). However, modified assistant output 654 is not only personalized or adapted to the user and functions in response to the assistant query, modified assistant output 654 also helps lead the interaction session between the user and the automated assistant by further engaging with the user in the interaction session (e.g., "Any fun plans today?"). Without using the techniques described herein for using one or more LLM outputs in modifying the original generated set of assistant outputs based on processing utterance 652, the automated assistant might simply reply "It's 8:30 AM" without greeting the user of client device 610 (e.g., "Good morning"), without calling the user of client device 610 by name (e.g., "[User]"), and without further engaging with the user of client device 610 in the interaction session (e.g., "Any fun plans today?"). Thus, the modified assistant output 654 may resonate more with the user of the client device 610 than any of the assistant outputs included in the original generated set of assistant outputs that do not utilize one or more LLM outputs.

[0092] 6, further assume that the user of client device 610 provides utterance 656, "Yes, I'm thinking about going to the beach." In this example, the automated assistant can cause audio data capturing utterance 656 to be processed to generate a set of assistant outputs that are generated without using one or more LLM outputs. Additionally, the automated assistant can cause the assistant query included in utterance 656, the set of assistant outputs, and / or the context of the interaction session to be processed to generate (e.g., in an offline and / or online manner) a set of modified assistant outputs determined using one or more LLM outputs, and optionally, to generate additional assistant queries based on the assistant query.

[0093] In this example, the assistant outputs included in the set of assistant outputs (i.e., generated without using one or more LLM outputs) may be limited because the assistant query included in utterance 656 does not request the automated assistant to take any action. For example, the assistant outputs included in the set of assistant outputs may include "Sounds fun!", "Surf's up!", "That sounds like fun!", and / or other assistant outputs that respond to utterance 656 but do not further engage with the user of client device 610 in an interaction session. In other words, the assistant outputs included in the set of assistant outputs may have limited vocabulary diversity because they are not generated using one or more LLM outputs as described herein. Nevertheless, the automated assistant can leverage the assistant outputs included in the set of assistant outputs to determine how to modify one or more of the assistant outputs using one or more of the LLM outputs.

[0094] Moreover, as described with respect to FIGS. 3 and 4 , the automated assistant can generate additional assistant queries based on the assistant query and using one or more of the LLMs or a separate ML model in communication with one or more of the LLMs. For example, in the example of FIG. 6 , utterance 656 provided by a user of client device 610 indicates that the user plans to go to the beach. Based on identifying that the intent associated with utterance 656 indicates that the user plans to go to the beach, the automated assistant can determine a related intent associated with checking the weather at a beach frequently visited by the user of client device 610 (e.g., an exemplary beach named “Half Moon Bay”). Based on identifying the related intent, the automated assistant can generate an additional assistant query of “What's the weather?” with a location parameter of “Half Moon Bay” and query one or more of 1P system 191 and / or 3P system 192 to obtain additional assistant output including, for example, “weather for Half Moon Bay,” indicating that it will be raining and the temperature will be low all day at Half Moon Bay. In some implementations, the automated assistant can cause additional assistant outputs and / or the context of the interaction session to be processed to generate a set of additional modified assistant outputs determined using one or more of the LLM outputs and / or one or more additional LLM outputs.

[0095] 6 , the automated assistant can cause the assistant output included in the set of assistant outputs and the set of modified assistant outputs to be ranked according to one or more ranking criteria, and can select one or more of the assistant outputs based on the ranking (e.g., select the given assistant output, “Sounds fun!”). Further, the automated assistant can cause the assistant output included in the set of additional modified assistant outputs and the additional assistant output to be ranked according to one or more ranking criteria, and can select one or more of the assistant outputs based on the ranking (e.g., select the given additional assistant output, “But if you’re going to Half Moon Bay again, expect rain and chilly temps”). Moreover, the automated assistant can combine the selected given assistant output and the selected given additional assistant output to result in a modified assistant output 658, “Sounds fun! But if you’re going to Half Moon Bay again, expect rain and chilly temps,” and can cause the modified assistant output 658 to be provided for visual and / or audible presentation to the user of client device 610. Thus, in this example, the automated assistant can process the utterance 656 and provide additional contextual information related to the utterance (e.g., the weather at a beach that the user of the client device 610 is likely to visit) to further engage with the user of the client device 610 in the interaction session.Without the techniques described herein, a user of client device 610 may be required to proactively request weather information from an automated assistant, even though the automated assistant is capable of determining and providing the weather information, thereby increasing the amount of user input, wasting computational resources at client device 610 in processing the increased amount of user input, and increasing the cognitive load on the user of client device 610.

[0096] 6 , further assume that the user of client device 610 provides utterance 660, "Oh no ... thanks for the heads up, can you remind me to check the weather again in two hours?" In this example, the automated assistant can have audio data capturing utterance 660 processed to generate a set of assistant outputs that are generated without using one or more LLM outputs. Additionally, the automated assistant can have the assistant query included in utterance 660, the set of assistant outputs, and / or the context of the interaction session processed to generate (e.g., offline and / or online) a modified set of assistant outputs that are determined using one or more LLM outputs. Based on processing utterance 660, the automated assistant can decide to set a reminder to the user of client device 610 to check the weather in "Half Moon Bay" at 10:30 AM (e.g., two hours into the interaction session) or to proactively provide the weather in "Half Moon Bay" to the user of client device 610 at 10:30 AM. Further, the automated assistant can cause a modified assistant output 662 from the set of modified assistant outputs (i.e., generated using one or more LLM outputs) to be provided for visual and / or audible presentation to the user of client device 610: "Sure thing, I set the reminder and hope the weather clears up for you." Notably, modified assistant output 662 in the example of FIG. 6 is contextual with respect to the interaction session in that it indicates to the user that he or she hopes the weather will clear up. In contrast, an assistant output included in the set of assistant outputs (i.e., generated without using one or more of the LLM outputs) may simply provide an indication that a reminder has been set, without taking into account the context of the interaction session.

[0097] While FIG. 6 is described with respect to using one or more LLM outputs to generate a particular modified assistant output and select a particular modified assistant output to be provided for presentation to a user based on a particular utterance and the context of an interaction session, it should be understood that this is for illustrative purposes only and is not intended to be limiting. Rather, the techniques described herein can be utilized for any interaction session between any user and a corresponding instance of an automated assistant. Furthermore, while a transcript corresponding to an interaction session between a user and an automated assistant is shown on display 680 of client device 610, it should also be understood that this is for illustrative purposes only and is not intended to be limiting. For example, it should be understood that the interaction session can be performed on any device capable of running an automated assistant, regardless of whether the client device includes a display.

[0098] 6 may include various personality replies described herein. For example, modified assistant output 654 may be generated using a first set of parameters that reflect a first personality of the automated assistant with respect to a first vocabulary to be utilized by the automated assistant and / or a first set of prosodic characteristics to be utilized in providing modified assistant output 654 for audible presentation to the user. Furthermore, modified assistant output 658 may be generated using a second set of parameters that reflect a second personality of the automated assistant with respect to a second vocabulary to be utilized by the automated assistant and / or a second set of prosodic characteristics to be utilized in providing modified assistant output 658 for audible presentation to the user. In this example, the first personality may reflect the personality of a butler or maid that is utilized, for example, to provide a morning greeting, respond to a user requesting the current time, and ask the user if they have any appointments for the day. Additionally, the second personality can reflect the personality of a weatherman, surfer, or lifeguard (or a combination thereof), for example, utilized to indicate that going to the beach would be fun, but that the weather that day may not be ideal at the beach. For example, because the user indicated that they were going to the beach, a surfer personality may be utilized to provide the "Sounds fun!" portion of modified assistant output 658, and because the automated assistant is providing weather information to the user, a weatherman personality may be utilized to provide "But if you're going to Half Moon Bay again, expect rain and chilly temps."

[0099] Thus, the automated assistant can dynamically adapt the personalities utilized in providing the modified assistant output for presentation to the user based on both the vocabulary utilized by the automated assistant and the prosodic characteristics utilized in rendering the modified assistant output for audible presentation to the user. In particular, the automated assistant can dynamically adapt these personalities utilized in providing the modified assistant output based on the context of the interaction session, including previous utterances received from the user, as well as previous assistant outputs provided by the automated assistant and / or any other contextual signals described herein. As a result, the modified assistant output provided by the automated assistant may resonate better with the user of the client device.

[0100] 6 is described herein in connection with a user providing utterances throughout an interaction session, it should be understood that this is for purposes of illustration and not intended to be limiting. For example, a user can additionally or alternatively provide typed and / or touch input throughout an interaction session. In these implementations, the automated assistant can process the typed input (e.g., using NLU engines 140A1 and / or 140A2) to generate a stream of NLU output (e.g., and optionally skip any processing using ASR engines 130A1 and / or 130A2) and can process the stream of NLU data and text input corresponding to the assistant query derived from the typed and / or touch input (e.g., using LLM engines 150A1 and / or 150A2) in generating a set of one or more modified assistant outputs in the same or similar manner as described above.

[0101] 7, there is shown a block diagram of an exemplary computing device 710 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, the cloud-based automated assistant component, and / or other components may comprise one or more components of the exemplary computing device 710.

[0102] Computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, user interface output devices 720, user interface input devices 722, and a network interface subsystem 716. The input and output devices enable user interaction with computing device 710. Network interface subsystem 716 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0103] The user interface input devices 722 may include a keyboard and pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or into a communications network.

[0104] The user interface output devices 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or to another machine or computing device.

[0105] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic for performing selected aspects of the methods disclosed herein and for implementing the various components shown in Figures 1 and 2.

[0106] These software modules are generally executed by the processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 may include several memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution, and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of some implementations may be stored by the file storage subsystem 726, in the storage subsystem 724, or on other machines accessible by the processor 714.

[0107] The bus subsystem 712 provides a mechanism for allowing the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem 712 may use multiple buses.

[0108] Computing device 710 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 710 shown in Figure 7 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 710 are possible, having more or fewer components than the computing device shown in Figure 7.

[0109] In situations where the systems described herein may collect or otherwise monitor personal information about users or utilize personal and / or monitored information, users may be given the opportunity to control whether a program or feature collects user information (e.g., the user's social network, social behavior or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how content is received from content servers that may be relevant to the user. Also, some data may be handled in one or more ways before being stored or used so that personally identifiable information is removed. For example, a user's identification information may be handled such that personally identifiable information about the user cannot be determined, or the user's geographic location may be generalized, in which case geographic location information is obtained (such as to the city, zip code, or state level) such that the user's specific geographic location cannot be determined. Thus, users can control how information is collected and / or used about them.

[0110] In some implementations, a method implemented by one or more processors is provided as part of an interaction session between a user of a client device and an automated assistant implemented by the client device, comprising: receiving a stream of audio data capturing an utterance of the user, the stream of audio data being generated by one or more microphones of the client device, the utterance including an assistant query; determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs is responsive to the assistant query included in the utterance; and generating a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), wherein each of the one or more LLM outputs is determined based on at least a portion of the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs. The method includes processing the set of assistant outputs and the context of the interaction session to generate an utterance and generating an additional assistant query related to the utterance based at least in part on the context of the interaction session and based at least in part on the assistant query; determining an additional assistant output responsive to the additional assistant query based on the additional assistant query; processing the additional assistant outputs and the context of the interaction session to generate a set of additional modified assistant outputs using one or more of the LLM outputs or one or more additional LLM outputs generated using the LLM, wherein each of the additional LLM outputs is determined based at least in part on the context of the interaction session and the additional assistant output; and causing a given modified assistant output from the set of modified assistant outputs and a given additional modified assistant output from the set of additional modified assistant outputs to be provided for presentation to the user.

[0111] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0112] In some implementations, determining an assistant output responsive to an assistant query included in the utterance based on processing the stream of audio data may include processing the stream of audio data using an automatic speech recognition (ASR) model to generate a stream of ASR output; processing the stream of ASR output using a natural language understanding (NLU) model to generate a stream of NLU data; and determining a set of assistant outputs based on the NLU stream.

[0113] In some versions of these implementations, processing the set of Assistant outputs and the context of the interaction session to generate a modified set of Assistant outputs using one or more of the LLM outputs generated using the LLM may include processing the set of Assistant outputs and the context of the interaction session using the LLM to generate one or more of the LLM outputs and determining a modified set of Assistant outputs based on one or more of the LLM outputs. In some further versions of these implementations, processing the set of Assistant outputs and the context of the interaction session to generate one or more of the LLM outputs using the LLM may include processing the set of Assistant outputs and the context of the interaction session using a first set of LLM parameters from a plurality of separate sets of LLM parameters to determine one or more LLM outputs having a first personality from a plurality of separate personalities. The modified set of Assistant outputs may include one or more first personality Assistant outputs that reflect the first personality. In still further versions of these implementations, processing the set of Assistant outputs and the context of the interaction session to generate one or more of the LLM outputs using an LLM may include processing the set of Assistant outputs and the context of the interaction session using a second set of LLM parameters from the plurality of separate sets of LLM parameters to determine one or more of the LLM outputs having a second personality from the plurality of separate personalities. The modified set of Assistant outputs may include one or more second personality Assistant outputs that reflect the second personality, where the second personality may be distinct from the first personality.In still further versions of these implementations, one or more first personality assistant outputs included in the set of modified assistant outputs and reflecting a first personality may be determined using a first vocabulary associated with the first personality, and one or more second personality assistant outputs included in the set of modified assistant outputs and reflecting a second personality may be determined using a second vocabulary associated with the second personality, the second personality being distinct from the first personality based on the second vocabulary being distinct from the first vocabulary. In still further additional or alternative versions of these implementations, one or more first personality assistant outputs included in the set of modified assistant outputs and reflecting a first personality may be associated with a first set of prosodic properties to be utilized in providing the given modified assistant output for audible presentation to the user, and one or more second personality assistant outputs included in the set of modified assistant outputs and reflecting a second personality may be associated with a second set of prosodic properties to be utilized in providing the given modified assistant output for audible presentation to the user, and the second personality may be distinct from the first personality based on the second set of prosodic properties being distinct from the first set of prosodic properties.

[0114] In some versions of these implementations, processing the set of assistant outputs and the context of the interaction session to generate a modified set of assistant outputs using one or more of the LLM outputs generated using the LLM may include identifying one or more of the previously generated LLM outputs using the LLM model based on a previous assistant query of a previous interaction session that corresponds to the assistant query of the interaction session, and / or based on a previous context of a previous interaction session that corresponds to the context of the interaction session, and modifying the set of assistant outputs using one or more of the LLM outputs to determine a modified set of assistant outputs. In some further versions of these implementations, identifying one or more of the previously generated LLM outputs using the LLM model may include identifying one or more first LLM outputs of the one or more LLM outputs that reflect a first personality of a plurality of separate personalities. The modified set of assistant outputs may include one or more first personality assistant outputs that reflect the first personality. In still further versions of these implementations, identifying one or more of the previously generated LLM outputs using the LLM model may include identifying one or more second LLM outputs of the one or more LLM outputs that reflect a second personality of the plurality of separate personalities. The set of modified assistant outputs may include one or more second personality assistant outputs that reflect the second personality, where the second personality may be separate from the first personality.In still further or alternative versions of these implementations, the method may further include determining that a previous assistant query of a previous interaction session corresponds to the assistant query of the interaction session based on the ASR output including one or more terms of the assistant query that correspond to one or more terms of the previous assistant query of the previous interaction session. In still further or alternative versions of these implementations, the method may further include generating an embedding of the assistant query based on one or more terms in the ASR output that correspond to the assistant query, and determining that a previous assistant query of a previous interaction session corresponds to the assistant query of the interaction session based on comparing the embedding of the assistant query with a previously generated embedding of the previous assistant query of the previous interaction session. In still further or alternative versions of these implementations, the method may further include determining that a previous context of a previous interaction session corresponds to a context of the interaction session based on one or more context signals of the interaction session corresponding to one or more context signals of the previous interaction session. In still further versions of these implementations, the one or more context signals may include one or more of a time of day, a day of the week, a location of the client device, and ambient noise in an environment of the client device. In still further additional or alternative versions of those implementations, the method may further include generating an embedding of the context of the interaction session based on a context signal of the interaction session, and determining that the previous context of the previous interaction session corresponds to the context of the interaction session based on comparing the embedding of the one or more context signals with a previously generated embedding of a previous context of a previous interaction session.

[0115] In some versions of these implementations, processing the set of assistant outputs and the context of the interaction session to generate an additional assistant query associated with the utterance based at least in part on the assistant query based at least in part on the context of the interaction session may include: determining an intent associated with the assistant query included in the utterance based on the NLU output; identifying at least one associated intent related to the intent associated with the assistant query included in the utterance based on the intent associated with the assistant query included in the utterance; and generating the additional assistant query for the utterance based on the at least one associated intent. In some further versions of these implementations, determining an additional assistant output responsive to the additional assistant query based on the additional assistant query may include causing the additional assistant query to be sent to one or more first-party systems via an application programming interface (API) to generate an additional assistant output responsive to the additional assistant query. In some additional or alternative further versions of those implementations, determining an additional assistant output responsive to the additional assistant query based on the additional assistant query may include causing the additional assistant query to be transmitted over one or more networks to one or more third-party systems, and receiving an additional assistant output responsive to the additional assistant query in response to the additional assistant query being transmitted to one or more of the third-party systems.In some additional or alternative further versions of those implementations, processing the additional assistant outputs and the context of the interaction session to generate one or more of the additional LLM outputs determined using the LLM or a set of additional modified assistant outputs using one or more of the additional LLM outputs may include processing the set of additional assistant outputs and the context of the interaction session using the LLM to determine one or more of the additional LLM outputs, and determining the set of additional modified assistant outputs based on one or more of the additional LLM outputs. In some additional or alternative further versions of those implementations, processing the additional assistant outputs and the context of the interaction session to generate a set of additional modified assistant outputs using one or more of the LLM outputs or one or more of the additional LLM outputs determined using the LLM may include identifying one or more of the additional LLM outputs previously generated using the LLM model based on one or more of the additional LLM outputs having been previously generated based on a previous assistant query of a previous interaction session that corresponds to the additional assistant query of the interaction session, and / or based on one or more of the additional LLM outputs having been previously generated for a previous context of a previous interaction session that corresponds to the context of the interaction session, and causing the set of additional assistant outputs to be modified using one or more of the additional LLM outputs to determine a set of additional modified assistant outputs.

[0116] In some implementations, the method may further include ranking the superset of assistant outputs based on one or more ranking criteria, where the superset of assistant outputs includes at least the set of assistant outputs and the set of modified assistant outputs, and selecting a given modified assistant output from the set of modified assistant outputs based on the ranking. In some versions of these implementations, the method may further include ranking the superset of additional assistant outputs based on one or more of the ranking criteria, where the superset of assistant outputs includes at least the set of additional assistant outputs and the set of additional modified assistant outputs, and selecting a given additional modified assistant output from the set of additional modified assistant outputs based on the ranking. In some further versions of these implementations, causing the given modified assistant output and the given additional modified assistant output to be provided for presentation to the user may include combining the given modified assistant output and the given additional modified assistant output, processing the given modified assistant output and the given additional modified assistant output using a text-to-speech (TTS) model to generate synthetic speech audio data including synthetic speech that captures the given modified assistant output and the given additional modified assistant output, and causing the synthetic speech audio data to be audibly rendered for presentation to the user via a speaker of the client device.

[0117] In some implementations, the method may further include: ranking the superset of assistant outputs based on one or more ranking criteria, wherein the superset of assistant outputs includes a set of assistant outputs, a set of modified assistant outputs, an additional assistant output, and a set of additional modified assistant outputs; and selecting a given modified assistant output from the set of modified assistant outputs and a given additional modified assistant output from the set of additional modified assistant outputs based on the ranking. In some further versions of these implementations, providing the given modified assistant output and the given additional modified assistant output for presentation to the user may include processing the given modified assistant output and the given additional modified assistant output using a text-to-speech (TTS) model to generate synthetic speech audio data including synthetic speech capturing the given modified assistant output and the given additional modified assistant output; and audibly rendering the synthetic speech audio data for presentation to the user via a speaker of the client device.

[0118] In some implementations, generating a set of modified assistant outputs using one or more of the LLM outputs may be further based on processing at least a portion of the assistant query included in the utterance.

[0119] In some implementations, a method implemented by one or more processors is provided as part of an interaction session between a user of a client device and an automated assistant implemented by the client device, and includes steps of receiving a stream of audio data capturing an utterance of the user, the stream of audio data being generated by one or more microphones of the client device, the utterance including an assistant query; determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs is responsive to the assistant query included in the utterance; and generating a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), wherein each of the one or more LLM outputs is responsive to the assistant query included in the utterance. The method includes processing the set of assistant outputs and the context of the interaction session to generate an assistant output determined at least in part based on the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs, and generating an additional assistant query associated with the utterance based at least in part on the context of the interaction session and at least in part on the assistant query; determining an additional assistant output responsive to the additional assistant query based on the additional assistant query; processing the set of modified assistant outputs based on the additional assistant output responsive to the additional assistant query to generate a set of additional modified assistant outputs; and causing a given additional modified assistant output from the set of additional modified assistant outputs to be provided for presentation to the user.

[0120] In some implementations, a method implemented by one or more processors is provided as part of an interaction session between a user of a client device and an automated assistant implemented by the client device, and includes steps of receiving a stream of audio data capturing the user's speech, the stream of audio data being generated by one or more microphones of the client device, the utterance including an assistant query; and determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query included in the utterance; and processing the set of assistant outputs and the context of the interaction session to generate a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), each of the one or more LLM outputs being determined based at least in part on the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs. Generating the set of modified assistant outputs using one or more of the LLM outputs includes generating a set of first personality replies based on (i) the set of assistant outputs, (ii) a context of the interaction session, and (iii) one or more first LLM outputs of the one or more LLM outputs that reflect a first personality of the plurality of distinct personalities. The method further includes causing a given modified assistant output to be provided from among the set of modified assistant outputs for presentation to the user.

[0121] In some implementations, a method implemented by one or more processors is provided as part of an interaction session between a user of a client device and an automated assistant implemented by the client device, and includes receiving a stream of audio data capturing the user's speech, the stream of audio data being generated by one or more microphones of the client device, the utterance including an assistant query; determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query included in the utterance; processing the set of assistant outputs and the context of the interaction session to generate a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), wherein each of the one or more LLM outputs is determined based at least in part on the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs; and providing a given modified assistant output from the set of modified assistant outputs for presentation to the user.

[0122] In some implementations, a method implemented by one or more processors is provided as part of an interaction session between a user of a client device and an automated assistant implemented by the client device, and includes receiving a stream of audio data capturing an utterance of the user, the stream of audio data being generated by one or more microphones of the client device, the utterance including an assistant query; and determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query included in the utterance; and determining whether to modify one or more of the assistant outputs included in the set of assistant outputs based on processing the utterance; and in response to determining to modify one or more of the assistant outputs included in the set of assistant outputs, processing the set of assistant outputs and the context of the interaction session to generate a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), wherein each of the one or more LLM outputs is determined based at least in part on the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs; and providing a given modified assistant output from the set of modified assistant outputs for presentation to the user.

[0123] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0124] In some implementations, determining whether to modify one or more of the assistant outputs included in the set of assistant outputs based on processing the utterance may include processing the stream of audio data using an automatic speech recognition (ASR) model to generate a stream of ASR output; processing the stream of ASR output using a natural language understanding (NLU) model to generate a stream of NLU data; identifying a user's intent in providing the utterance based on the stream of NLU data; and determining whether to modify the assistant output based on the user's intent in providing the utterance.

[0125] In some implementations, determining whether to modify one or more of the Assistant outputs included in the set of Assistant outputs may further be based on one or more computational costs associated with modifying one or more of the Assistant outputs. In some versions of these implementations, the one or more computational costs associated with modifying one or more of the Assistant outputs may include one or more of battery consumption, processor consumption associated with modifying one or more of the Assistant outputs, or latency associated with modifying one or more of the Assistant outputs.

[0126] In some implementations, a method implemented by one or more processors is provided, comprising: obtaining a plurality of assistant queries directed to an automated assistant and a corresponding context of a corresponding previous interaction session for each of the plurality of assistant queries; for each of the plurality of assistant queries, processing a given assistant query of the plurality of assistant queries using one or more large-scale language models (LLMs) to generate a corresponding LLM output responsive to the given assistant query; indexing the corresponding LLM output in a memory accessible at the client device based on the given assistant query and / or the corresponding context of the corresponding previous interaction session for the given assistant query; and indexing the corresponding LLM output in a memory accessible at the client device. After dexterity, receiving a stream of audio data capturing the user's utterance as part of a current interaction session between a user of the client device and the automated assistant implemented by the client device, the stream of audio data being generated by one or more microphones of the client device; determining, based on processing the stream of audio data, that the utterance includes a current assistant query corresponding to the given assistant query and / or that the utterance is received in a current context of a current interaction session corresponding to an interaction context of a corresponding previous interaction session for the given assistant query; and causing the automated assistant to utilize the corresponding LLM output in generating an assistant output to be provided for presentation to the user in response to the utterance.

[0127] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0128] In some implementations, the multiple assistant queries directed to the automated assistant may have been previously submitted by a user via a client device. In some implementations, the multiple assistant queries directed to the automated assistant may have been previously submitted by multiple additional users in addition to the user of the client device via their respective client devices.

[0129] In some implementations, indexing the corresponding LLM output in memory accessible at the client device may be based on embedding of a given assistant query generated during processing of the given assistant query. In some implementations, indexing the corresponding LLM output in memory accessible at the client device may be based on one or more terms or phrases included in a given assistant query generated during processing of the given assistant query. In some implementations, indexing the corresponding LLM output in memory accessible at the client device may be based on embedding of a corresponding context of a corresponding previous interaction session for the given assistant query. In some implementations, indexing the corresponding LLM output in memory accessible at the client device may be based on one or more context signals included in a corresponding context of a corresponding previous interaction session for the given assistant query.

[0130] Additionally, some implementations include one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), and / or tensor processing unit (TPU)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the aforementioned methods. Some implementations also include computer program products that include instructions executable by the one or more processors to perform any of the aforementioned methods. [Explanation of symbols]

[0131] 110 client devices 111 User Input Engine 112 Rendering Engine 113 Presence Sensor 114 Automated Assistant Client 115 Automated Assistants 120 Natural Conversation System 130 ASR engine 140 NLU engine 150 LLM engine 160 TTS engine 170 Offline Output Correction Engine 171 Assistant Activity Engine 172 Indexing Engine 180 Online Output Correction Engine 181 Assistant Query Engine 182 Assistant Personality Engine 190 ranking engine 191 1P system 192 3P System 199 Network 201 Stream of audio data 202 Context 203 ASR output 204 NLU Output 205 Assistant Output 206 Corrected Assistant Output 207 Given Assistant Output 710 Computing Devices 712 Bus Subsystem 714 processor 716 Network Interface 720 User Interface Output Device 722 User Interface Input Devices 724 Storage Subsystem 725 Memory Subsystem 726 File Storage Subsystem 730 RAM 732 ROM

Claims

1. 1. A method implemented by one or more processors, comprising: As part of an interactive session between a user of a client device and an automated assistant implemented by said client device, receiving a stream of audio data capturing the user's speech, the stream of audio data being generated by one or more microphones on the client device, the utterance including an Assistant query; Determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query included in the utterance; Generating a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), each of the one or more LLM outputs being determined at least in part based on a context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs; generating an additional assistant query related to the utterance based at least in part on the context of the interaction session and based at least in part on the assistant query; Processing the set of assistant outputs and the context of the interaction session for determining an additional assistant output responsive to the additional assistant query based on the additional assistant query; processing the additional assistant outputs and the context of the interaction session to generate a set of additional modified assistant outputs using one or more of the LLM outputs or one or more additional LLM outputs generated using the LLM, each of the additional LLM outputs being determined at least in part based on the context of the interaction session and the additional assistant outputs; causing a given modified assistant output from the set of modified assistant outputs and a given additional modified assistant output from the set of additional modified assistant outputs to be provided for presentation to the user; A method comprising:

2. Determining the assistant output responsive to the assistant query included in the utterance based on processing the stream of audio data, processing the stream of audio data using an automatic speech recognition (ASR) model to generate a stream of ASR output; processing the stream of ASR output using a natural language understanding (NLU) model to generate a stream of NLU data; and causing the set of assistant outputs to be determined based on the stream of NLUs.

3. processing the set of assistant outputs and the context of the interaction session to generate the set of modified assistant outputs using one or more of the LLM outputs generated using the LLM, processing the set of assistant outputs and the context of the interaction session using the LLM to generate one or more of the LLM outputs; and determining the set of modified assistant outputs based on one or more of the LLM outputs.

4. processing the set of assistant outputs and the context of the interaction session to generate one or more of the LLM outputs using the LLM, processing the set of Assistant outputs and the context of the interaction session using a first set of LLM parameters among a plurality of distinct sets of LLM parameters to determine one or more of the LLM outputs having a first personality among a plurality of distinct personalities; 4. The method of claim 3, wherein the set of modified Assistant outputs includes one or more first personality Assistant outputs that reflect the first personality.

5. processing the set of assistant outputs and the context of the interaction session to generate one or more of the LLM outputs using the LLM, processing the set of assistant outputs and the context of the interaction session using a second set of LLM parameters among a plurality of distinct sets of LLM parameters to determine one or more of the LLM outputs having a second personality among a plurality of distinct personalities; the set of modified Assistant outputs includes one or more second personality Assistant outputs that reflect the second personality; The method of claim 4 , wherein the second personality is distinct from the first personality.

6. 6. The method of claim 5, wherein the one or more first personality assistant outputs included in the set of modified assistant outputs and reflecting the first personality are determined using a first vocabulary associated with the first personality, and the one or more second personality assistant outputs included in the set of modified assistant outputs and reflecting the second personality are determined using a second vocabulary associated with the second personality, and the second personality is distinct from the first personality based on the second vocabulary being distinct from the first vocabulary.

7. 6. The method of claim 5, wherein the one or more first personality assistant outputs included in the set of modified assistant outputs and reflecting the first personality are associated with a first set of prosodic properties to be utilized in providing the given modified assistant output for audible presentation to the user, and the one or more second personality assistant outputs included in the set of modified assistant outputs and reflecting the second personality are associated with a second set of prosodic properties to be utilized in providing the given modified assistant output for audible presentation to the user, and the second personality is distinct from the first personality based on the second set of prosodic properties being distinct from the first set of prosodic properties.

8. processing the set of assistant outputs and the context of the interaction session to generate the set of modified assistant outputs using one or more of the LLM outputs generated using the LLM, Identifying one or more of the LLM outputs previously generated using the LLM based on one or more of the LLM outputs having been previously generated based on a previous Assistant query of a previous interaction session that corresponds to the Assistant query of the interaction session, and / or based on one or more of the LLM outputs having been previously generated for a previous context of the previous interaction session that corresponds to the context of the interaction session; and causing the set of assistant outputs to be modified using one or more of the LLM outputs to determine the set of modified assistant outputs.

9. identifying one or more of the LLM outputs previously generated using the LLM model, identifying one or more first LLM outputs of the one or more LLM outputs that reflect a first personality of a plurality of distinct personalities; 9. The method of claim 8, wherein the set of modified Assistant outputs includes one or more first personality Assistant outputs that reflect the first personality.

10. identifying one or more of the LLM outputs previously generated using the LLM model, identifying one or more second LLM outputs of the one or more LLM outputs that reflect a second personality of a plurality of distinct personalities; the set of modified Assistant outputs includes one or more second personality Assistant outputs that reflect the second personality; The method of claim 9 , wherein the second personality is distinct from the first personality.

11. 11. The method of claim 8, further comprising: determining that the previous assistant query of the previous interaction session corresponds to the assistant query of the interaction session based on the ASR output including one or more terms of the assistant query that correspond to one or more terms of the previous assistant query of the previous interaction session.

12. generating an embedding of the assistant query based on one or more terms in the ASR output corresponding to the assistant query; 12. The method of claim 8, further comprising: determining that the previous assistant query of the previous interaction session corresponds to the assistant query of the interaction session based on comparing the embedding of the assistant query with a previously generated embedding of the previous assistant query of the previous interaction session.

13. 13. The method of claim 8, further comprising determining that the previous context of the previous interaction session corresponds to the context of the interaction session based on one or more context signals of the interaction session corresponding to one or more context signals of the previous interaction session.

14. The method of claim 13 , wherein the one or more contextual signals include one or more of the following: time of day, day of the week, location of the client device, and ambient noise in the client device's environment.

15. generating an embedding of the context of the interaction session based on a context signal of the interaction session; 15. The method of claim 8, further comprising determining that the previous context of the previous interaction session corresponds to the context of the interaction session based on comparing the embedding of the one or more context signals with a previously generated embedding of the previous context of the previous interaction session.

16. Processing the set of assistant outputs and the context of the interaction session to generate the additional assistant query associated with the utterance based at least in part on the context of the interaction session and based at least in part on the assistant query, Determining an intent associated with the assistant query included in the utterance based on the output of the NLU; Identifying at least one related intention related to the intention related to the assistant query included in the utterance based on the intention related to the assistant query included in the utterance; and generating the additional assistant query related to the utterance based on the at least one associated intent.

17. determining the additional assistant output responsive to the additional assistant query based on the additional assistant query, 17. The method of claim 16, comprising causing the additional assistant query to be sent via an application programming interface (API) to one or more first party systems to generate the additional assistant output responsive to the additional assistant query.

18. determining the additional assistant output responsive to the additional assistant query based on the additional assistant query, causing the additional assistant query to be transmitted over one or more networks to one or more third party systems; and receiving, in response to the additional assistant query being sent to one or more of the third-party systems, the additional assistant output responsive to the additional assistant query.

19. processing the additional assistant outputs and the context of the interaction session to generate the set of additional modified assistant outputs using one or more of the LLM outputs or one or more of the additional LLM outputs determined using the LLM; processing the set of additional assistant outputs and the context of the interaction session using the LLM to determine one or more of the additional LLM outputs; and determining the set of additional modified assistant outputs based on one or more of the additional LLM outputs.

20. processing the additional assistant outputs and the context of the interaction session to generate the set of additional modified assistant outputs using one or more of the LLM outputs or one or more of the additional LLM outputs determined using the LLM; Identifying one or more of the additional LLM outputs previously generated using the LLM model based on the one or more of the additional LLM outputs having been previously generated based on a previous assistant query of a previous interaction session that corresponds to the additional assistant query of the interaction session, and / or based on the one or more of the additional LLM outputs having been previously generated for a previous context of the previous interaction session that corresponds to the context of the interaction session; and causing the set of additional assistant outputs to be modified with one or more of the additional LLM outputs to generate the set of additional modified assistant outputs.

21. Ranking a superset of assistant outputs based on one or more ranking criteria, wherein the superset of assistant outputs includes at least the set of assistant outputs and the set of modified assistant outputs; and selecting the given modified assistant output from the set of modified assistant outputs based on a ranking.

22. Ranking a superset of additional assistant outputs based on one or more of the ranking criteria, wherein the superset of assistant outputs includes at least the additional assistant outputs and the set of additional modified assistant outputs; and selecting the given additional modified assistant output from the set of additional modified assistant outputs based on the ranking.

23. causing the given modified assistant output and the given additional modified assistant output to be provided for presentation to the user, combining the given modified assistant output with the given additional modified assistant output; Processing the given modified Assistant output and the given additional modified Assistant output using a text-to-speech (TTS) model to generate synthetic speech audio data including synthetic speech that captures the given modified Assistant output and the given additional modified Assistant output; and causing the synthesized voice audio data to be audibly rendered for presentation to the user via a speaker of the client device.

24. Ranking a superset of Assistant outputs based on one or more ranking criteria, wherein the superset of Assistant outputs includes the set of Assistant outputs, the set of modified Assistant outputs, the additional Assistant outputs, and the set of additional modified Assistant outputs; 24. The method of claim 1, further comprising the steps of: selecting the given modified assistant output from the set of modified assistant outputs and selecting the given additional modified assistant output from the set of additional modified assistant outputs based on a ranking.

25. causing the given modified assistant output and the given additional modified assistant output to be provided for presentation to the user, Processing the given modified Assistant output and the given additional modified Assistant output using a text-to-speech (TTS) model to generate synthetic speech audio data including synthetic speech that captures the given modified Assistant output and the given additional modified Assistant output; and causing the synthesized voice audio data to be audibly rendered for presentation to the user via a speaker of the client device.

26. 26. The method of any one of claims 1 to 25, wherein generating a set of modified Assistant outputs using one or more of the LLM outputs is further based on processing at least a portion of the Assistant query included in the utterance.

27. 1. A method implemented by one or more processors, comprising: As part of an interactive session between a user of a client device and an automated assistant implemented by said client device, receiving a stream of audio data capturing the user's speech, the stream of audio data being generated by one or more microphones on the client device, the utterance including an Assistant query; Determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query included in the utterance; Generating a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), each of the one or more LLM outputs being determined at least in part based on a context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs; generating an additional assistant query related to the utterance based at least in part on the context of the interaction session and based at least in part on the assistant query; Processing the set of assistant outputs and the context of the interaction session for determining an additional assistant output responsive to the additional assistant query based on the additional assistant query; processing the set of modified Assistant outputs based on the additional Assistant outputs responsive to the additional Assistant queries to generate an additional set of modified Assistant outputs; providing a given additional modified assistant output from among the set of additional modified assistant outputs for presentation to the user; A method comprising:

28. 1. A method implemented by one or more processors, comprising: As part of an interactive session between a user of a client device and an automated assistant implemented by said client device, receiving a stream of audio data capturing the user's speech, the stream of audio data being generated by one or more microphones on the client device, the utterance including an Assistant query; Determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query included in the utterance; processing the set of assistant outputs and the context of the interaction session to generate a set of modified assistant outputs using one or more LLM outputs generated using a large language model (LLM), wherein each of the one or more LLM outputs is determined based at least in part on the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs, and generating the set of modified assistant outputs using one or more of the LLM outputs; generating a set of first personality replies based on (i) the set of Assistant outputs, (ii) the context of the interaction session, and (iii) one or more first LLM outputs of the one or more LLM outputs that reflect a first personality of a plurality of distinct personalities; providing a given modified assistant output from the set of modified assistant outputs for presentation to the user; A method comprising:

29. 1. A method implemented by one or more processors, comprising: As part of an interactive session between a user of a client device and an automated assistant implemented by said client device, receiving a stream of audio data capturing the user's speech, the stream of audio data being generated by one or more microphones on the client device, the utterance including an Assistant query; Determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query included in the utterance; Processing the set of assistant outputs and the context of the interaction session to generate a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), wherein each of the one or more LLM outputs is determined based at least in part on the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs; providing a given modified assistant output from the set of modified assistant outputs for presentation to the user; A method comprising:

30. 1. A method implemented by one or more processors, comprising: As part of an interactive session between a user of a client device and an automated assistant implemented by said client device, receiving a stream of audio data capturing the user's speech, the stream of audio data being generated by one or more microphones on the client device, the utterance including an Assistant query; Determining a set of assistant outputs based on processing the stream of audio data, wherein each assistant output in the set of assistant outputs responds to the assistant query included in the utterance; Determining whether to modify one or more of the assistant outputs included in the set of assistant outputs based on the processing of the utterance; In response to determining to modify one or more of the assistant outputs included in the set of assistant outputs, Processing the set of assistant outputs and the context of the interaction session to generate a set of modified assistant outputs using one or more LLM outputs generated using a large-scale language model (LLM), wherein each of the one or more LLM outputs is determined based at least in part on the context of the interaction session and one or more of the assistant outputs included in the set of assistant outputs; providing a given modified assistant output from the set of modified assistant outputs for presentation to the user; A method comprising:

31. determining whether to modify one or more of the assistant outputs included in the set of assistant outputs based on the processing of the utterance, processing the stream of audio data using an automatic speech recognition (ASR) model to generate a stream of ASR output; processing the stream of ASR output using a natural language understanding (NLU) model to generate a stream of NLU data; determining an intent of the user in providing the utterance based on the stream of NLU data; and determining whether to modify the assistant output based on the intent of the user in providing the utterance.

32. 32. The method of claim 30 or 31, wherein determining whether to modify one or more of the assistant outputs included in the set of assistant outputs is further based on one or more computational costs associated with modifying one or more of the assistant outputs.

33. 33. The method of claim 32, wherein the one or more computational costs associated with modifying one or more of the assistant outputs include one or more of battery consumption, processor consumption associated with modifying one or more of the assistant outputs, or latency associated with modifying one or more of the assistant outputs.

34. 1. A method implemented by one or more processors, comprising: Obtaining a plurality of assistant queries directed to an automated assistant and corresponding contexts of corresponding previous interaction sessions for each of the plurality of assistant queries; for each of the plurality of assistant queries, processing a given assistant query of the plurality of assistant queries using one or more large language models (LLMs) to generate a corresponding LLM output responsive to the given assistant query; indexing the corresponding LLM output in a memory accessible at a client device based on the corresponding context of the given assistant query and / or the corresponding previous interaction session for the given assistant query; After indexing the corresponding LLM output in the memory accessible at the client device, as part of a current interaction session between a user of the client device and an automated assistant implemented by the client device, receiving a stream of audio data capturing the user's speech, the stream of audio data being generated by one or more microphones of the client device; Based on processing the stream of audio data, determining that the utterance includes a current assistant query corresponding to the given assistant query, and / or that the utterance is received in a current context of the current interaction session corresponding to the corresponding context of the corresponding previous interaction session for the given assistant query; causing the automated assistant to utilize the corresponding LLM output in generating an assistant output to be provided for presentation to the user in response to the utterance; A method comprising:

35. 35. The method of claim 34, wherein the plurality of assistant queries directed to the automated assistant were previously issued by the user via the client device.

36. 35. The method of claim 34, wherein the plurality of assistant queries directed to the automated assistant were previously issued by a plurality of additional users, in addition to the user of the client device, via respective client devices.

37. 37. The method of claim 34, wherein indexing the corresponding LLM output in the memory accessible at the client device is based on an embedding of the given assistant query generated when processing the given assistant query.

38. 37. The method of claim 34, wherein indexing the corresponding LLM output in the memory accessible at the client device is based on one or more terms or phrases included in the given assistant query generated when processing the given assistant query.

39. 37. The method of claim 34, wherein indexing the corresponding LLM output in the memory accessible at the client device is based on embedding the corresponding context of the corresponding previous interaction session for the given assistant query.

40. 37. The method of claim 34, wherein indexing the corresponding LLM output in the memory accessible at the client device is based on one or more context signals included in the corresponding context of the corresponding previous interaction session for the given assistant query.

41. at least one processor; and a memory storing instructions that, when executed, cause the at least one processor to perform operations corresponding to any one of claims 1 to 40.

42. 41. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform operations corresponding to any one of claims 1 to 40.

Images