LLM latency reduction via bridging multiple LLMs of different sizes

Abstract

One implementation utilizes a smaller LLM to generate content responsive to a user query, causing a portion of the generated content to be rendered as an immediate response to the user query. Another implementation utilizes a larger LLM to generate a refined portion of the content that begins with and includes the portion following that portion of the generated content. This refined portion can then be rendered after the portion of the generated content. In some implementations, instead of using a smaller LLM, the portion of the generated content rendered as an immediate response can be generated based on a default text string or template, where a template can be determined / selected from multiple predefined templates based on natural language understanding of the user query.

Description

Background Technology

[0001] Various generative models have been proposed that can be used to process natural language (NL) content and / or other inputs to generate outputs that reflect generative content in response to the inputs. For example, large language models (LLMs) have been developed that can be used to process NL content and / or other inputs to generate LLM outputs that reflect generative NL content and / or other generative content in response to the inputs. For example, LLM can be used to process NL content such as "how to change DNS settings on Acme router" to generate LLM output that reflects several responsive NL sentences like: "First, type the router's IP address in a browser; the default IP address is 192.168.1.1. Then enter the username and password; the defaults are admin and admin. Finally, select the advanced settings tab and find the DNS settings section." However, current utilization of generative models has one or more drawbacks.

[0002] As an example, many generative models can be extremely large, typically comprising billions of parameters (e.g., over 100 billion, over 250 billion, or over 500 billion parameters). Due to the sheer size of such generative models, processing inputs using them to generate corresponding generative outputs can require substantial amounts of memory, processors, power, and / or other computational resources. This resource utilization can be significant on a per-input basis and extremely significant when hundreds or thousands of inputs are being processed per minute, per second, or other intervals. Furthermore, the large size of such generative models can lead to significant latency in generating the corresponding generative outputs and, consequently, in rendering the corresponding generative content. Such latency can result in prolonged user interaction with the computer. Further, due to the large size of such generative models, many or all client devices may be unable to utilize them on their devices. For example, memory constraints on client devices may prevent such generative models from being loaded into memory.

[0003] Smaller-sized counterparts of such generative models do exist, such as separately trained counterparts with fewer parameters, or pruned and / or quantized counterparts generated by applying one or more pruning and / or quantization techniques to a larger counterpart. For example, smaller counterparts of a larger model may have 25%, 33%, 50%, 66%, or other percentage fewer parameters than the larger model. However, such smaller-sized counterparts may be less robust and / or less accurate than their larger-sized counterparts. Therefore, while processing inputs using such smaller-sized counterparts may be computationally more efficient and / or can be performed with less latency, there is a greater risk that the corresponding generative outputs generated by processing the inputs may be inaccurate and / or inadequately specified. Summary of the Invention

[0004] The implementations disclosed herein involve using first and second generative models with different computational efficiencies to generate content for rendering in response to user utterances, typed input, and / or other types of requests or queries from users (e.g., human users). In various implementations, the first generative model may be computationally more efficient than the second generative model (e.g., by having fewer parameters). In some of these implementations, the first generative model may be utilized on a client device, the second generative model may be utilized on a remote server far from the client device, and the second generative model may not be usable on the client device (i.e., due to computational constraints of the client device). In some implementations, in response to receiving a user query, the first generative model may be utilized to process the user query to generate a first response that includes natural language (sometimes referred to as the "first natural language response") and responds to the user query.

[0005] In various implementations, the first response may include (e.g., begin) a first part (e.g., a sentence), such as a part that does not make a factual statement, and may include one or more additional parts that can make factual statements. For example, a user query may be a query seeking information about one or more entities, and a first generative model may be trained (e.g., fine-tuned) to determine / generate a first response that begins with one or more sentences or other parts that do not make factual statements about one or more entities. In some implementations, a text-to-speech (TTS) engine may be used to process one or more parts of the first response to generate corresponding audio data. In response to a user query, the generated audio data corresponding to that part may be rendered to the user (e.g., audibly and immediately).

[0006] In various implementations, the user query may also be provided to the second generative model along with instructions that cause the second generative model to generate a refined response in response to the user query, beginning with (or following) the portion of the first response designated as to be rendered. For example, a prompt may be generated that includes the user request, the portion from the first response, and / or instructions that cause the second generative model to generate a refined response in response to the user request, beginning with (or following) the portion from the first response. This prompt may be provided to the second generative model, causing the second generative model to process the prompt to generate a second response (as the "refined response") in response to the user query.

[0007] Depending on the specific content of the prompts given to the second generative model, the second response may include, but must not include (e.g., begin with) a portion from the first response. For example, when the instruction to the second generative model is to generate a refined response that begins with that portion, the second (or refined) response may begin with that portion and may also include a refined portion with one or more additional sentences following that portion. As an additional example, when the instruction to the second generative model is to generate a refined response that follows that portion, the refined response may include one or more sentences to follow that portion, but the refined response itself may include or begin with that portion. In other words, in this additional example, the refined response may not include a portion from the first response.

[0008] By generating a prompt that includes a portion from the first response and designated for rendering, the processing performed using a second generative model in generating the refined response is controlled. That is, by including this portion (optionally along with additional instructions associated with it) in the prompt, the processing performed using the second generative model is controlled such that the refined response generated by such processing is consistent with the portion designated for rendering. This control ensures that the latency and duration benefits gained by immediately rendering the portion are not negated by the refined response. For example, this ensures that the refined response is not a repetition of the portion. Moreover, this can be further ensured in implementations that generate a prompt that includes the portion and also includes instructions for generating output that follows but does not include the portion.

[0009] In various implementations, while a portion of the first response from the first generative model is being rendered to the user (e.g., while the audio data for the first sentence is being audibly rendered to the user), a second generative model can be used to process the cue. For example, the model output of the second generative model generated based on the cue can be generated before the audio data for that portion is fully rendered. The model output of the second generative model corresponding to the cue can be used to determine a second response that begins (or follows) that portion, and this second response, beginning with that portion, can be determined and / or transmitted to the TTS engine before the first sentence is fully rendered. This can be optionally achieved, for example, by controlling / reducing the rendering speed of the audio data corresponding to the first sentence (when the first sentence is audibly rendered).

[0010] In various implementations, the second response may include a refined portion in natural language following the first portion, wherein the refined portion may, but does not necessarily, provide a statement of fact. After the audio data for the first portion (which is determined using the first generative model) has been fully rendered, the audio data for the refined portion (which is determined using the second generative model) may be rendered. For example, the audio data for the refined portion may be rendered after the audio data for the first portion, without any intermediate audio data (e.g., audio data for any additional portions determined using the first generative model).

[0011] It should be noted that in some implementations, in response to a user query, the portion determined using the first generative model can be visually rendered, rather than audibly rendered or as a supplement to audibly rendered. Following the visual rendering of these portions, the refined portions can be visually rendered, rather than audibly rendered or as a supplement to audibly rendered. In these implementations, the user query can optionally be typed input or touch input. For example, in response to receiving input from a user via a user input device, the first sentence of the first response generated using the first generative model can be visually rendered at the user interface of the device receiving the user query, and the refined portion of the second response generated using the second generative model can be visually rendered at the user interface, for example, 0.6 seconds (or other time interval) after the first sentence is initially visually rendered. The first sentence of the first response can optionally be rendered together with a representation of the user input. For example, both typed input and the first sentence can be rendered at the user interface, where the first sentence is rendered as an immediate response to the typed input. The refined portion can be visually rendered after the initial rendering of the first sentence, with no sentences existing between the first sentence and the refined portion. Optionally, the first sentence may not contain factual statements. For example, the first sentence could be an opening line greeting the user typing the input (e.g., "That is a really good question to ask what is AI?"), and / or may include non-factual information about the topics or entities in the typed input (e.g., "AI is atrending technology", "AI has been popular for several reasons"). Optionally, in response to the typed input, the refined portion can be rendered to provide more complex and accurate content. For example, the refined portion may include one or more factual statements about one or more entities referenced in the typed input (e.g., the year the concept or theory was introduced, the name of the author of the book or painting, the location of the event, the size of the company, the total amount of time spent constructing the structure, etc.).

[0012] In various implementations, the first generative model can be a smaller Large Language Model (LLM) with fewer than 100 billion parameters, while the second generative model can be an even larger LLM with over 200 billion parameters. Because the second generative model is a larger LLM, the refined portion can provide more accurate, complex, and / or user-expected content in response to a user query compared to any non-rendered portion of the response from the first generative model. In other words, the second generative model can objectively be more robust and / or more accurate than the smaller LLM. Furthermore, because the first generative model is a smaller LLM, the first sentence can be rendered to the user (e.g., audibly and / or visually) with reduced latency. This is likely due to, for example, the smaller LLM comprising fewer parameters and / or the smaller LLM being utilized (and capable of being utilized) on client devices. In other words, collaboratively rendering content in response to a user query using first and second generative models with different computational efficiencies not only reduces latency when providing such content but also improves the accuracy and / or quality of the content itself. Furthermore, by rendering the first sentence with reduced latency, the total duration of human-computer interaction can be reduced because the initial rendering is based on the output of a first generative model with lower latency and can begin without waiting on a second generative model with higher latency. Therefore, the implementation seeks to reduce the total duration of human-computer interaction by leveraging the lower-latency first generative model to render the initial content faster, while also utilizing the objectively more robust and accurate second generative model to generate subsequent content to resolve user requests.

[0013] As a non-limiting example of work, a user can provide a spoken utterance to a computing device, such as “why is my basement leaking?” or “Assistant, why is my basement leaking?”. The spoken utterance can be parsed / identified to determine a user query in natural language (i.e., “why is my basement leaking” in natural language). This user query in natural language can be provided to a first LLM (which is an example of the aforementioned first generative model) local to the computing device for processing. For example, the first LLM local to the computing device can be used to process the user query in natural language to generate a first model output, from which a first response (in response to the user query) can be derived.

[0014] For example, the first response includes a first sentence that does not make a factual statement, and further includes a second part (containing one or more sentences) in addition to the first sentence. The second part may include, for example, a second sentence following the first sentence, a third sentence following the second sentence, and so on. The first sentence of a spoken response to “Assistant, why is my basement leaking?” could be, for example, “Sorry your basement is leaking.” Optionally, the first sentence may include content determined based on the tone of the spoken utterance (e.g., worried, excited, etc.) (e.g., Oh no, I’m sorry…). Alternatively or additionally, the audio data used for the first sentence may have a tone determined based on the tone of the spoken utterance (e.g., worried, excited, etc.). Optionally, the first and / or second sentences may provide, for example, a summary of the first response. As an example, the first sentence could be, “Oh no you may want to check out the following common reasons for basement leakage.” As another example, the first sentence could be "Oh no, I am sosorry your basement is leaking," and the second sentence could be "There are some common reasons for basement leakage."

[0015] Continuing with the non-limiting example above, in some implementations, the user query can be provided to a second LLM (which is an example of the previously mentioned second generative model) located remotely to a computing device, whereby the second LLM can be used to process the user query in natural language to generate a second model output from which a second response is derived. The second LLM can be accessed via a server device that communicates with the computing device. In some other implementations, instead of the user query, a text prompt can be provided to the second LLM to generate the second model output. The text prompt can include, for example, the user query in natural language, a first sentence, and a request / instruction to generate content in response to the user query that begins with the first sentence. It should be noted that if the first sentence is already included / identified in the request to generate content in response to the user query, the text prompt may not need to include the first sentence again. In other words, the text prompt can include (1) the user query in natural language and (2) a request to generate content in response to the user query that begins with the first sentence, without separately or repeatedly including the first sentence from the first response.

[0016] As described above, a second LLM can be used to process text prompts to generate a second model output that produces a second response. The second response can begin with a first sentence (from the first response) and can continue with a refined portion following the first sentence. A TTS engine can be used to process the first sentence, rendering it audibly in response to spoken utterances containing the user's query. Immediately following the rendering of the first sentence, the refined portion generated using the second LLM can be rendered audibly instead of the second portion generated using the first LLM, thereby improving the accuracy or quality of the content rendered in response to the user query.

[0017] In the non-limiting examples above, in some implementations, instead of rendering the first sentence, multiple sentences or a predetermined number of sentences can be audibly rendered in response to a user request, while the computing device waits at the server device to generate a refined portion using a second LLM. For example, the refined portion can be received by the computing device (e.g., a TTS engine) before the audible rendering of the first sentence is complete. In this case, the TTS engine can immediately audibly render the refined portion once the audible rendering of the first sentence is complete.

[0018] In various implementations, the refined portion can be received by the TTS engine of the computing device after the audible rendering of the first sentence is complete. In some implementations, the refined portion can be received by the TTS engine of the computing device within a predefined time period (e.g., 0.5 seconds) after the completion of the audible rendering of the first sentence. In those implementations, the TTS engine can audibly render the refined portion once it has been processed into its audible counterpart using natural language. In some implementations, the refined portion can be received by the TTS engine of the computing device after a predefined time period since the completion of the audible rendering of the first sentence. In these implementations, the TTS engine can audibly render the second sentence from the first response at the end of the predefined time period, and then audibly render the refined portion immediately following the rendering of the second sentence.

[0019] In various implementations, the first LLM can be a smaller LLM, and the second LLM can be a larger LLM, where the smaller LLM is a quantized and / or pruned version of the larger LLM. In some other implementations, the smaller LLM is not a quantized and / or pruned version of the larger LLM, but is completely independent of it. For example, the smaller LLM may have a different architecture relative to the larger LLM, and / or may be trained on a unique training dataset relative to the larger LLM. For example, the smaller LLM may have a smaller input dimension than the larger LLM, the smaller LLM may have a smaller output dimension than the larger LLM, and / or the smaller LLM may include various intermediate layers of different sizes and / or types relative to the larger LLM.

[0020] Smaller LLMs can be computationally more efficient than larger LLMs. For example, processing a request using a smaller LLM can occur with less latency compared to using a larger LLM. As another example, processing a request using a smaller LLM can utilize less memory, processor, and / or power resources compared to using a larger LLM. In some implementations, the smaller LLM can reside on a device at the client device, while the larger LLM can be located remotely from the client device. For example, the larger LLM can reside on a server device communicating with the client device. Utilizing a smaller LLM (rather than a larger LLM) to generate initial content in response to a user request, and causing that initial content to be visually rendered, can more quickly satisfy the information needs of users who have provided verbal statements or user requests.

[0021] In some implementations, user requests / queries can be processed to determine one or more query features, contextual features, and / or attribute features associated with the client device and / or the user providing the query / request. For example, when a user request includes a natural language query (e.g., automatically generated or generated based on user interface input), one or more query features may include: terms of the query; embeddings of the terms of the query (e.g., generated using separate encoders); the topic or domain reflected by the query; and / or other features that can be derived from the query. As another example, when a user request includes a query with an image, query features may include: automatically generated captions for the image; descriptors of objects automatically detected in the image; and / or other features that can be derived from the image. Contextual features may be, or may include, a first feature related to, for example, the tone of the user request (if the user request is audible) determined based on audio data captured from the user request. For example, attribute features may be determined based on the user's profile.

[0022] In some implementations, smaller or larger LLMs are sequence-to-sequence models, which are transformer-based and / or may include encoders and / or decoders. A non-limiting example of an LLM is Google's Pathways Language Model (PaLM). Another non-limiting example of an LLM is Google's Language Model for Dialogue Applications (LaMDA).

[0023] In some implementations, a method implemented using one or more processors is provided. The method includes receiving a user query in natural language. In response to receiving the user query, the method includes: processing the user query using a first generative model to generate a natural language response in response to the user query; causing a first portion of the natural language response to be audibly rendered; generating a text prompt as a natural language request including the user query and including the generation of a refined natural language response in response to the user query, beginning with the first portion of the natural language response; providing the generated text prompt to a second generative model, wherein providing the generated text prompt to the second generative model causes the generated text prompt to be processed using the second generative model to generate a refined natural language response beginning with the first portion and including the refined portion; and causing the refined portion of the refined natural language response to be audibly rendered after the first portion of the natural language response, while a second portion of the natural language response is not rendered between them.

[0024] The foregoing is presented as an overview of only some of the implementations disclosed in this paper. This paper discloses these and other implementations in more detail. For example, additional and / or alternative implementations are disclosed herein, such as those for using templates to generate the first response instead of leveraging a first generative model.

[0025] As another example, instead of using a first LLM to process the user query (which is determined from the user utterance), the first LLM can be used to process the portion of the user utterance that is determined to contain the complete user query, before or simultaneously with additional parts of the user utterance being processed (e.g., to determine the entire transcription of the user utterance). The user utterance could be, for example, “okay, how to cook a turkey? This is my first time cooking turkey.” In this case, the first part of the transcription of the user utterance corresponding to “how to cook a turkey” can be processed and determined to include the complete user query “how to cook a turkey” in natural language. Before or simultaneously with determining / generating additional portions of the transcription of the user utterance corresponding to “This is my first time cooking a turkey,” a first LLM (e.g., a smaller LLM) can be used to process the first portion of the transcription (“okay, how to cook a turkey?”) as input to generate an immediate response to the user utterance. The immediate response can be rendered in response to the user utterance. Alternatively, instead of rendering the entire immediate response, a portion of the immediate response, such as the first sentence of the immediate response, can be rendered in response to the user utterance.

[0026] In the example above, after identifying the additional portion of the transcription, a second LLM (e.g., a larger LLM) can be used to process the entire transcription of the user's statement ("okay, how to cook a turkey? This is my first time cooking a turkey") as input to generate a refined response. The refined response can include a refined portion that follows or will follow the first sentence of the immediate response. The refined portion can be rendered immediately after the first sentence of the immediate response. Regarding the immediate response, the refined portion can be more user-specific / desired (as specified in the user's statement—"This is my first time cooking a turkey") by including a recipe friendly to those with little or no cooking experience or who have never cooked turkey before. By partially fulfilling the user intent corresponding to the action of searching for a turkey recipe (i.e., not further determining / modifying the user intent based on the user statement "This is my first time cooking a turkey"), the latency in rendering the immediate response (or a portion thereof) in response to the user query "how to cook a turkey" can be reduced. By providing a more refined portion of the response immediately following the immediate response, a more accurate response can be provided in response to the user's words.

[0027] Various implementations may include a non-transitory computer-readable storage medium storing instructions executable by a processor to perform one or more methods such as those described herein. Other various implementations may include a system comprising a memory and one or more hardware processors operable to execute instructions stored in the memory to perform one or more methods such as those described herein. Attached Figure Description

[0028] Figure 1A A block diagram depicts an example environment that illustrates various aspects of this disclosure and in which some of the implementations disclosed herein can be implemented.

[0029] Figure 1B An example of a rendering response is shown, which illustrates various aspects of this disclosure and allows for the implementation of some of the methods disclosed herein.

[0030] Figure 1C Another example of a rendering response is shown, which illustrates various aspects of this disclosure and in which some of the implementations disclosed herein can be implemented.

[0031] Figure 2 A flowchart illustrating an example method for generating content in response to a user request, according to various aspects of this disclosure, is provided.

[0032] Figure 3 A flowchart illustrating another example method for generating content in response to a user request to generate content, according to various aspects of this disclosure, is depicted.

[0033] Figure 4 A flowchart illustrating further example methods for generating content in response to a user request for content generation, according to various aspects of this disclosure, is depicted.

[0034] Figure 5 Example architectures of computing devices based on various implementations are described. Detailed Implementation

[0035] Figure 1A This is a block diagram illustrating various aspects of this disclosure and in which the implementations disclosed herein can be carried out. For example... Figure 1A As shown, environment 100 may include client computing device 10 (“client device”) and server computing device 12 (“server device”) communicating with client computing device 10 via one or more networks 13. One or more networks 15 may include, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, and / or any other suitable network.

[0036] The client computing device 10 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a vehicle computing device (e.g., an in-vehicle entertainment system), an interactive speaker, a smart appliance such as a smart TV, and / or a wearable device that includes a computing device (e.g., glasses with a computing device, a virtual or augmented reality computing device), and this disclosure is not limited thereto.

[0037] In various implementations, the client computing device 10 may include a user input engine 101 configured to detect user input provided by a user of the client computing device 10 using one or more user interface input devices. For example, the client computing device 10 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to a user's spoken words (e.g., user utterance T) or other sounds in the environment of the client computing device 10. Alternatively, the client computing device 10 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected in the field of view of one or more of the visual components.

[0038] Alternatively, the client computing device 10 may be equipped with one or more touch-sensitive components (e.g., keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to touch input directed at the client computing device 10. Some examples of queries that can be included in a request as described herein may be queries based on user input provided by the user of the client computing device 10 and detected by the user input engine 101. For example, a query may be a typed query entered via a physical or virtual keyboard, a suggested query selected via a touchscreen or mouse, a spoken voice query detected by the microphone of the client device, or an image query based on an image captured by the visual components of the client device.

[0039] In various implementations, the client computing device 10 may include a rendering engine 110 and / or storage 115. In various implementations, the rendering engine 110 may be configured to provide audible and / or visual content (e.g., a natural language-based response generated by an LLM) for presentation to a user of the client computing device 10 using one or more user interface output devices. For example, the client computing device 10 may be equipped with one or more speakers that enable the provision of audible content to a user via the client computing device 10. Alternatively, the client computing device 10 may be equipped with a display or projector that enables the provision of visual content to a user via the client computing device 10.

[0040] In various implementations, the client computing device 10 may further include multiple local components. These local components may include an Automatic Speech Recognition (ASR) engine 111, a Natural Language Understanding (NLU) engine 112, an execution engine 113, and / or a Text-to-Speech (TTS) engine 114. In some implementations, the ASR engine 111, NLU engine 112, execution engine 113, and / or TTS engine 114 may be included in an automated assistant (also referred to as a “chatbot,” “interactive assistant,” etc.), which may be installed or accessed via the client computing device 10. In some implementations, the user R of the client computing device 10 may have a registered account associated with the automated assistant and / or other third-party applications. Third-party applications may include, for example, note-taking applications, shopping applications, messaging applications, and / or any other suitable applications (or services) installed on (or accessible via) the client computing device 10.

[0041] Server computing device 12 may be, for example, a web server, a proxy server, a VPN server, or any other type of server as required. In various implementations, server computing device 12 may include cloud-based components that are the same as or similar to the multiple local components installed on client computing device 1. For example, server computing device 12 may include a cloud-based ASR engine 121, a cloud-based NLU engine 122, a cloud-based fulfillment engine 123, and / or a cloud-based TTS engine 124. Server computing device 12 may optionally include data storage 126.

[0042] ASR engine 111 (and / or cloud-based ASR engine 121) can use one or more streaming ASR models (e.g., recurrent neural network (RNN) models, transformer models, and / or any other type of ML model capable of performing ASR) to process the audio data stream captured from spoken utterances and generated by the microphone of the client computing device 10 to generate a corresponding ASR output stream. It is noteworthy that streaming ASR models can be used to generate the corresponding ASR output stream while generating the audio data stream.

[0043] NLU engine 112 and / or cloud-based NLU engine 122 can use one or more NLU models (e.g., Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or syntax-based rules to process the corresponding ASR output stream to generate a corresponding NLU output stream. Performance engine 113 and / or cloud-based performance engine 123 can cause the corresponding NLU output stream to be processed to generate a corresponding performance data stream. The corresponding performance data stream can correspond, for example, to a corresponding given assistant output predicted in response to spoken utterances captured in the corresponding audio data stream processed by ASR engine 111 (and / or cloud-based ASR engine 121).

[0044] The TTS engine (e.g., 114 and / or 124) can use a TTS model to process a corresponding text content stream (e.g., text defined by the LLM-based assistant 11) to generate synthesized speech audio data including computer-generated synthesized speech. The corresponding text content stream may correspond to, for example, one or more given assistant outputs, one or more modified given assistant outputs, and / or any other text content described herein. The aforementioned ML model may be an on-device ML model stored locally on the client computing device 10, a remote ML model executed remotely from a server computing device (e.g., at a remote server device 12), or a shared ML model accessible to both the client computing device 10 and / or the remote system (e.g., the remote server computing device 12). In additional or alternative implementations, the corresponding synthesized speech audio data stream corresponding to one or more given assistant outputs, one or more modified given assistant outputs, and / or any other text content described herein may be pre-cached in memory or one or more databases accessible to the client computing device 10, so that the LLM-based assistant does not need to use the TTS engine 114 (or 124) to generate the corresponding synthesized speech audio data.

[0045] In various implementations, the corresponding ASR output stream may include, for example, a stream of ASR hypotheses (e.g., term hypotheses and / or transcription hypotheses) predicted to correspond to spoken utterances captured in the corresponding audio data stream by the user, one or more corresponding prediction metrics (e.g., probability, log-likelihood, and / or other values) for each ASR hypothesis included in the ASR hypothesis stream, multiple phonemes predicted to correspond to spoken utterances captured in the corresponding audio data stream by the user, and / or other ASR outputs. In some versions of those implementations, ASR engines 111 and / or 121 may select one or more ASR hypotheses from the ASR hypotheses as the corresponding identified text corresponding to the spoken utterances (e.g., selected based on corresponding prediction metrics).

[0046] In various implementations, the corresponding NLU output stream may include, for example, an annotated stream of identified text, which includes one or more annotations to the identified text for one or more (e.g., all) terms of the identified text, one or more corresponding prediction metrics (e.g., probability, log-likelihood, and / or other values) included in the NLU output stream, and / or other NLU outputs. For example, NLU engines 112 and / or 122 may include part-of-speech taggers (not depicted) configured to annotate terms using syntactic roles. Alternatively, NLU engines 112 and / or 122 may include entity taggers (not depicted) configured to annotate entity references in one or more segments of the identified text, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, (real and imagined) locations, etc. In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some implementations, a knowledge graph may include nodes representing known entities (and in some cases, entity attributes), and edges connecting the nodes and representing relationships between entities. Entity annotators may annotate references to entities at a high-granularity level (e.g., to enable the identification of all references to entity categories (such as people)) and / or at a lower-granularity level (e.g., to enable the identification of all references to a specific entity (such as a specific person)). Entity annotators may rely on the content of natural language input to resolve specific entities and / or may optionally communicate with the knowledge graph or other entity databases to resolve specific entities.

[0047] Alternatively or concurrently, NLU engines 112 and / or 122 may include a coreference resolver (not depicted) configured to group or “cluster” references to the same entity based on one or more contextual clues. For example, the coreference resolver may be used to resolve the term “them” in the input “buy them” to “buy theater tickets” based on a mention of “theater tickets” in a client device notification rendered immediately before the natural language input “buy them” is received. In some implementations, one or more components of NLU engines 112 and / or 122 may rely on annotations from one or more other components of NLU engines 112 and / or 122. For example, in some implementations, an entity annotator may rely on annotations from the coreference resolver to annotate all references to a particular entity. Furthermore, for example, in some implementations, the coreference resolver may rely on annotations from the entity annotator to cluster references to the same entity.

[0048] Alternatively or concurrently, NLU engines 112 and / or 122 may include a query determination engine configured to determine, based on the corresponding NLU output stream, whether speech recognition of spoken utterance T (e.g., from user R) includes a user query. In some implementations, fulfillment engine 113 may include a first LLM engine 1131 and / or a second LLM engine 125. Alternatively, the second LLM engine 125 may be located at server device 12 instead of client computing device 10. The first LLM engine 1131 may communicate with a first generative model 190A (e.g., a first LLM), and the second LLM engine 1132 may communicate with a second generative model 190B (e.g., a second LLM). In some implementations, client computing device 11 or server device 12 may include a prompt generation engine 116 (see...). Figure 1B The prompt generation engine is configured to generate prompts to be processed using either a first generative model 190A or a second generative model 190B as input. In response to a query determination engine determining that the speech recognition of spoken utterance (e.g., from user R) includes a user query, the prompt generation engine 116 can generate a first text (also referred to as a "first prompt") based on the user query (e.g., determined from the speech recognition of the user utterance).

[0049] The first text may be or may include a user query, wherein a first generative model 190A may be used to process the first text as input to generate a first model output (“first output”) from which a first response is determined. The first response may include a first part and a second part, wherein the first part may include one or more sentences that do not make factual statements. The first part of the first response may be processed using a TTS engine 114 to generate first audio data for the first part of the first response. The first audio data may be audibly rendered as an immediate response to the user query.

[0050] In some implementations, in response to decoding / determining a first portion from a first output of a first generative model corresponding to a user query, a prompt generation engine 116 may generate second text (“second prompt”, “second text prompt”) based on the user query (determined from speech recognition of the user’s utterance) and / or based on a first portion of a first response generated using a first generative model 190A. For example, the prompt generation engine 116 may generate second text to include the user query and instructions to generate a response beginning with the first portion of the first response. A second generative model 190B may be used to process the second text as input to generate a second model output (“second output”) from which a second response is determined. The second response may include the first portion and a refined portion following the first portion. TTS 114 may be used to process the refined portion to generate second audio data including the refined portion, wherein the second audio data may be audibly rendered to provide a refined response to the user query following the aforementioned first audio data. In some implementations, the rate at which the first audio data is rendered may be configured such that the latency between the first and second audio data is minimized or reduced. For example, the second audio data can be rendered audibly within a predetermined time period after the first audio data has been rendered.

[0051] In various implementations, the first generative model can be a smaller Large Language Model (LLM) with fewer than 100 billion parameters, while the second generative model can be a larger LLM with over 200 billion parameters. Because the second generative model is a larger LLM, the refined portion can provide more accurate or complex content in response to user requests compared to the second part. Because the first generative model is a smaller LLM, the first part can be audibly rendered to the user with reduced latency. In other words, utilizing first and second generative models with different computational efficiencies to collaboratively and selectively render content in response to user requests (e.g., the first part and the refined portion) not only reduces the latency of providing content in response to user queries but also improves the accuracy or quality of the content in response to user queries.

[0052] In various implementations, the first LLM can be a smaller LLM, and the second LLM can be a larger LLM, where the smaller LLM is a quantized and / or pruned version of the larger LLM. In some other implementations, the smaller LLM is not a quantized and / or pruned version of the larger LLM, but is completely independent of the larger LLM. For example, the smaller LLM may have a different architecture relative to the larger LLM, and / or may be trained on a unique training dataset relative to the larger LLM. For example, the first LLM may be trained to not make factual statements for the first few sentences of the first response, while the second LLM may be trained to make factual statements for the first sentence of the refined portion. Optionally, the input dimension of the smaller LLM may be smaller than the input dimension of the larger LLM, the output dimension of the smaller LLM may be smaller than the output dimension of the larger LLM, and / or the smaller LLM may include various intermediate layers of different sizes and / or types relative to the larger LLM.

[0053] Smaller LLMs can be computationally more efficient than larger LLMs. For example, processing a request using a smaller LLM can occur with less latency compared to using a larger LLM. As another example, processing a request using a smaller LLM can utilize less memory, processor, and / or power resources compared to using a larger LLM. In some implementations, the smaller LLM can reside on a device at the client device, while the larger LLM can be located remotely from the client device. For example, the larger LLM can reside on a server device communicating with the client device. However, this is not necessary. For example, both the smaller and larger LLMs can reside locally on / at the client device, or both can be located remotely from the client device. Utilizing a smaller LLM (rather than a larger LLM) to generate initial content in response to a user request, and causing that initial content to be visually rendered, can more quickly satisfy the information needs of users providing verbal statements or user requests.

[0054] Figure 1B A non-limiting example of a rendering response is shown, illustrating various aspects of this disclosure and allowing for the implementation of some of the methods disclosed herein. Figure 1BAs shown, user R can provide user utterance 14A to client device 10, such as “Okay, explain general relativity”. In this example, ASR engine 111 (and / or 121) can be used to process user utterance 14A to determine the speech recognition of user utterance 14A. In some implementations, the speech recognition of user utterance 14A can be processed to determine a user query for “general relativity”. In some other implementations, NLU engine 112 (and / or 122) can be used to process the speech recognition of user utterance 14A to determine the user intent (e.g., explain) and / or parameters associated with the user intent (e.g., “general relativity”) for the action (e.g., explain general relativity). The user intent and / or parameters associated with the user intent or the user query can be received by execution engine 113 (and / or 123).

[0055] In some implementations, execution engine 113 (local to client device 10) may include a first LLM engine 1131 and a second LLM engine 125, or may communicate with the first LLM engine and the second LLM engine. In some other implementations, execution engine 123 (remote to client device 10) may include a first LLM engine 1131 and a second LLM engine 125, or may communicate with the first LLM engine and the second LLM engine. In some other implementations, execution engine 113 (local to client device 10) may include a first LLM engine 1131 (or may communicate with the first LLM engine), and execution engine 123 (remote to client device 10) may include a second LLM engine 125 (or may communicate with the second LLM engine). For example, in response to receiving a user query for "general relativity", fulfillment engine 113 (and / or 123) may initiate prompting engineering engine 116 (which may be included in or communicate with fulfillment engine 113 / 123) to generate a first text prompt 17A (also referred to as "first text" or "first prompt"). Prompt engineering engine 116 may, for example, generate first text prompt 17A to include the user query for "general relativity". For example, first LLM 190A may be used to process first text prompt 17A as input.

[0056] In some implementations, the prompting engineering engine 116 can generate a second text prompt 17B (also referred to as "second text" or "second prompt"), which can be processed using a second LLM 190B. The second text prompt 17B can be the same as or different from the first text prompt 17A. For example, given the user utterance 14A of "okay, please explain general relativity," both the first text prompt 17A and the second text prompt 17B can be, for example, "describe general relativity" or "describe the concept of general relativity." In this example, processing the first text prompt 17A using the first LLM 190A can be done in parallel with processing the second text prompt 17B using the second LLM 190B.

[0057] It should be noted that even when the first text prompt 17A and the second text prompt 17B are the same or similar, the output of the first LLM 190A (generated based on processing the first text prompt 17A) can differ from the output of the second LLM 190B (generated based on processing the second text prompt 17B). This is because the first LLM 190A and the second LLM 190B can be of different types, can have different numbers of parameters, and / or can be trained differently (e.g., using different training datasets). For example, in some implementations, as described above, the first LLM 190A can be a distilled, quantized, and / or pruned version of the second LLM 190B.

[0058] In some other implementations, the second LLM 190B can be trained on multiple training instances to predict the first sentence (or first pair of sentences) of the response generated / determined from the output of the first LLM 190A, or the second LLM 190B can be fine-tuned on multiple training instances to predict the first sentence (or first pair of sentences) of the response generated / determined from the output of the first LLM 190A. In these implementations, each of the multiple training instances can include training instance inputs and ground truth responses. Training instance inputs can include (and sometimes only include) user queries (e.g., "explain general relativity" in natural language), and ground truth responses corresponding to the training instance inputs can include (and sometimes only include) natural language responses (e.g., Figure 1B or Figure 1CThe refined portion 19B of the code will follow the first sentence of the response generated based on the user query processed using a first trained LLM 190A (or template). Figure 1B or Figure 1C (Part 19A in the first part). In other words, the true value response can exclude the first sentence of the response generated based on the first LLM190A (or template) trained to process the user query.

[0059] For example, the first LLM 190A can be trained to not make factual statements in the first sentence (or the first two sentences, or any other applicable number of sentences at the very beginning of the first response 18 in response to user utterance 14A). This is achieved, for example, by training the first LLM 190A using (and sometimes only using) training instances such that each training instance has training instance inputs including a user query and a ground truth response that does not make factual statements in the first sentence (or any other applicable number of sentences at the beginning of the response to the user utterance). Alternatively or additionally, the first LLM 190A can be trained to provide the first sentence (or the first few sentences) of the first response 18 in a manner that includes specific content (e.g., “Oh well, that is a complex subject”, etc.). Alternatively, the first LLM 190A can be trained to provide the first sentence (or the first few sentences) of the first response 18 in a manner that the first sentence (or the first few sentences) has a specific length or is within a specific length range (e.g., 20 words, 30 words, or between 20 and 50 words, etc.).

[0060] However, the description of training the first LLM 190A is not limited thereto. For example, the first LLM 190A can be trained to provide a first response 18 with a specific total length (e.g., no more than 80 words, 100 words, etc., not exceeding a predefined length threshold) in response to user utterance 14A. By training the first LLM 190A to provide a first portion 18A of the first response 18 with a specific length and / or by training the first LLM 190A to provide a first response 18 with a specific total length not exceeding a predefined length threshold, the efficiency of using the first LLM 190A to provide the first portion 18A in an instantaneous response to user utterance 14A can be ensured or improved. For example, in response to determining the decoding of the first portion 18 from the first model output of the first LLM 190A based on processing the first text prompt 17A, the first portion 18A of the first response 18 can be processed using the TTS engine 114 to generate first audio data corresponding to the first portion 18A. In this scenario, the first audio data can be rendered to the user R as an immediate response to the user's utterance 14A, while the client device 10 (or another device) awaits a refined response 19B from the second LLM 190B with higher quality (e.g., better accuracy).

[0061] The second LLM 190B can, for example, be trained to make factual statements starting from the first sentence or the second sentence (or another applicable sentence). The second LLM 190B can be trained using a second training dataset that is larger than the first training dataset used to train the first LLM 190A. The second training dataset used to train the second LLM 190B can include training instances that are different from the training instances in the first training dataset. For example, the training instances (or a portion thereof) in the second training dataset can each include (1) training instance inputs containing a user query, and (2) ground truth responses that respond to the user query and make factual statements starting from a sentence (e.g., the first or second sentence). In this example, the training instances (or a portion thereof) in the first training dataset can each include training instance inputs containing a user query and a ground truth response that responds to the user query and does not make factual statements in the first sentence of the ground truth response (or in the first few sentences of the ground truth response). The second LLM 190B can be trained to provide a second response 19 with an approximate total length, such as 200 words, 250 words, etc.

[0062] It should be noted that the prompt generation engine 116 can generate both a first text prompt and a second text prompt in response to receiving speech recognition (and / or user query) of user utterance 14A. However, in some other implementations, the prompt generation engine 116 can wait to generate a second text prompt 17B based on partial or complete output of the first LLM 190A, wherein the partial or complete output is generated based on speech recognition of user utterance 14A (or user query determined from speech recognition).

[0063] In some implementations, the prompt generation engine 116 can be configured to receive the first sentence (or first part 18A) of the first response 18 once the first sentence or first part 18A is decoded from the first output of the first LLM 190A. In these implementations, the prompt generation engine 116 can generate a second text prompt 17B based on the user query and / or based on the first part 18A (which may include the first sentence or the first few sentences of the first response 18). As a concrete example, given that the user utterance 14A is “okay, explain general relativity” and the first part 18A of the first response 18 is a single sentence “The theory of general relativity is complex,” the second text prompt 17B could be in the format of “explain general relativity starting with the sentence of 'The theory of general relativity is complex'” or “explain general relativity to follow the sentence of 'The theory of general relativity is complex'”. Optionally, in this example, the second text prompt 17B could further include a word count limit and could be, for example, “explain general relativity in about 225 words, starting with the sentence of 'The theory of general relativity is complex'”, etc.

[0064] In response to the prompt generation engine 116 generating a second text prompt 17B, the second text prompt 17B can be transmitted to the device hosting the second LLM 190B. For example, if the second LLM 190B is at the client device 10, the second text prompt 17B can be forwarded to the second LLM 190B for processing using the second LLM 190B as input. If the second LLM 190B is at a server located remotely from the client device 10, the second text prompt 17B can be forwarded to the server for processing using the second LLM 190B as input.

[0065] In the above practical examples, such as Figure 1B As shown, a second LLM 190B can be used to process the second text prompt 17B to generate a second model output from which a second response 19 is determined. The second response 19 responds to user utterance 14A and may include a first portion 19A and a refined portion 19B, wherein the first portion 19A of the second response 19 may be identical to the first portion 18A of the first response 18 (e.g., when the second prompt includes an instruction to begin the response with the first portion 18A). The refined portion 19B can be processed using a TTS engine 114 to generate second audio data corresponding to the refined portion 19B. In this practical example, in response to determining that the first audio data corresponding to the first portion 18A of the first response 18 has been rendered via the client device, rendering of the second audio data corresponding to the refined portion 19B of the second response 19 can be initiated. For example, as... Figure 1B As shown, the refined section 19B can make factual statements, such as "It's a theory developed by Albert Einstein," or "It's a theory developed by Albert Einstein in about 1915," etc.

[0066] It should be noted that, although Figure 1B The second response 19 is shown as including both the first portion 19A and the refined portion 19B, but in some other implementations, the first portion 19A may not be included in the second response. For example, when the second prompt includes an instruction to generate content (or a refined response) that follows but does not include the first portion 18A, the second response 19 generated using the second LLM 190B may include the refined portion 19B but not the first portion 19A.

[0067] Figure 1CAnother non-limiting example of a rendering response is shown, which illustrates various aspects of this disclosure and in which some of the implementations disclosed herein can be implemented. Figure 1C As shown, user R can provide user utterance 14B, such as “Why is my basement leaking?”, to client device 10 (e.g., a smartphone). User utterance 14B can be processed by ASR engine 111 (and / or 121) to determine the speech recognition of user utterance 14B. The speech recognition of utterance 14B can be processed to determine a user query (e.g., a user query regarding the cause of the basement leak). In response to receiving the determined user query, prompt generation engine 116 can be activated to determine a first prompt 17A and / or a second prompt 17B based on the determined user query.

[0068] In some implementations, the first prompt 17A and the second prompt 17B can be determined approximately simultaneously. The second prompt 17B may be the same as or different from the first prompt 17A. For example, given “Why my basement is leaking” as a non-limiting specific example of user utterance 14B, the first prompt 17A may be or may include speech recognition of the user utterance “Why my basement is leaking?” or text such as “Generate a response for 'Why my basement is leaking' and limit the first sentence of the response to 20 words”. In this specific example, the second prompt 17B may be or may include speech recognition of, for example, the user utterance “Why is my basement leaking?” or text such as “Generate a response with about 260 words for a query of 'Why my basement is leaking'”.

[0069] In some other implementations, the second prompt 17B can be determined after the first prompt 17A. For example, a first LLM 190A can be used to process the first prompt 17A as input to generate a first model output from which a first response 18 (which includes a first portion 18A) is determined. The second prompt 17B can depend on the first model output (or a portion thereof). For example, the second prompt 17B can include instructions for generating a response to the user utterance 14B that begins with the first portion 18A of the first response 18. In some implementations, the first portion 18A can be a single sentence S1. In some other implementations, the first portion 18A can include multiple sentences S1, ..., Si, ..., Sn (where "n" is a positive integer greater than 1, and 1 ≤ i ≤ n). Given the specific example above, if the first part 18A of the first response 18 includes the single sentence "So sorry to hear that," then the second prompt 17B could be, for example, "Generate a response for 'Why my basement is leaking' starting with the content of 'sosorry to hear that.'" Continuing with this specific example, if the first part 18A of the first response 18 includes the first sentence "So sorry to hear that" and the second sentence "There could be several reasons," then the second prompt 17B could be, for example, "Generate a response for 'Why my basement is leaking' starting with the content of 'So sorry to hear that. There could be several reasons.'"

[0070] In some implementations, the first LLM 190A can be trained to provide a first portion 18A that does not make factual statements (e.g., about one or more entities in the speech recognition of user utterance 14B). In some implementations, optionally, the first response 18 may further include a second portion 18B (e.g., the second portion following the first portion 18A). The first LLM 190A can be trained to generate the second portion 18B, which makes at least one factual statement about one or more entities in the speech recognition of user utterance 14B. For example, the second portion 18B may begin with a sentence that makes a factual statement about an entity or event in the speech recognition of user utterance 14B.

[0071] In some implementations, in response to the determination of the first portion 18A (e.g., processing or decoding of the first model output based on the first LLM 190A), the first portion 18A may be provided to a second response monitoring engine 117, which monitors the presence of a second response 19 generated using the second LLM 190B based on the second cue 17B (or, in some cases, a refined portion 19B of the second response 19, rather than the second response 19 itself). A second portion 18B of the first response 18 may be provided to the second response monitoring engine 117, but this may not be necessary or required.

[0072] In some implementations, the first sentence S1 may be provided directly to the TTS engine 114 immediately upon determination (or upon determination of the first part 18A, or upon determination of the first response 18). In some other implementations, the first sentence S1 may be provided to the TTS engine 114 by a second response monitoring engine 117. The TTS engine 114 is used to process the first sentence S1 to generate audio data for the first sentence S1. The audio data for the first sentence S1 may be audibly rendered to the user R as an immediate response to the user's utterance 14B. It should be noted that when auditory rendering is not desired (e.g., based on the silence mode of the computing device 10, or due to other circumstances), the first sentence S1 may also be visually rendered to the user immediately upon determination.

[0073] In some implementations, while the audio data for the first sentence S1 is being rendered, the second response monitoring engine 117 may monitor the second response 19 (or, if the second prompt 17B includes an instruction to cause the second LLM 190B to generate a second response 19 beginning with the first portion 18A of the first response 18, monitor a refined portion 19B of the second response 19 such that the repeated first portion 18A does not need to be transmitted back to the second response monitoring engine 117) to determine whether the second response 19 has been received. In some implementations, in response to determining that the second response 19 (or the refined portion 19B of the second response 19) is received before the audio data for the first sentence S1 has been fully rendered and based on the fact that the second prompt 17B does not include an instruction to generate a second response 19 beginning with the first portion 18A (such that the first portion 19A of the second response 19 is different from the first portion 18A of the first response 18), the second response monitoring engine 117 may provide the second response 19 to the TTS engine 114 to generate audio data for the second response 19. The audio data for the second response 19 can be rendered after the audio data for the first sentence S1 is rendered, for example, no intermediate audio data is rendered between the audio data for the second response 19 and the audio data for the first sentence S1.

[0074] In some implementations, in response to determining that the audio data for the first sentence S1 has been fully rendered and based on an instruction in the second cue 17B to generate a second response 19 beginning with a first portion 18A (such that the first portion 19A of the second response 19 is the same as the first portion 18A of the first response 18), the second response monitoring engine 117 may provide the refined portion 19B of the second response 19 to the TTS engine 114 without providing the first portion 19A. The TTS engine 114 may be used to process the refined portion 19B to generate audio data for the refined portion 19B, wherein the audio data for the refined portion 19B may be rendered after the audio data for the first sentence S1. For example, the audio data for the refined portion 19B may be rendered after the audio data for the first sentence S1, without rendering any intermediate audio data between the audio data for the refined portion 19B and the audio data for the first sentence S1.

[0075] In some implementations, in response to determining that a second response 19 (or a refined portion 19B of the second response 19) is received within a predefined time period (e.g., 0.5 seconds) since the audio data for the first sentence S1 has been fully rendered, the second response monitoring engine 117 may provide the second response 19 (or the refined portion 19B) to the TTS engine 114 to generate corresponding audio data to be rendered after the audio data for the first sentence S1. The corresponding audio data for the second response 19 (or the refined portion 19B) may be rendered after the audio data for the first sentence S1 without rendering any intermediate audio data between the audio data for the second response 19 (or the refined portion 19B) and the audio data for the first sentence S1.

[0076] In some implementations, in response to determining that the second response 19 (or the refined portion 19B of the second response 19) has been received after a predefined time period mentioned above, the second response monitoring engine 117 may provide the TTS engine with the second sentence S2 of the first response 18, and continue to monitor the presence of the second response 19 (or its refined portion 19B) while audio data for the second sentence S2 is being generated or rendered. In these implementations, the TTS engine 114 may generate audio data for the second sentence S1, and the audio data for the second sentence S2 may be rendered after the audio data for the first sentence S1, without any intermediate audio data. If the second response monitoring engine 117 determines that the second response 19 (or its refined portion 19B) has been received during the rendering of the audio data for the second sentence S2 (or within a predefined time period since the rendering of the audio data for the second sentence S2 has been completed), then the audio data for the second response 19 (or its refined portion 19B) may be rendered after the audio data for the second sentence S2, without any intermediate audio data.

[0077] Continuing with the specific example above, the second response monitoring engine 117 can receive or detect the refined portion 19B (e.g., "the most common reason for residential basement leaking is…") before the full rendering of the audio data for the first sentence "So sorry to hearthat," or within the aforementioned predefined time period. In this case, the audio data for the refined portion 19B (e.g., "the most common reason for residential basement leaking is…") can be rendered after the audio data for the first sentence. The audio data for the refined portion 19B can be rendered without any intermediate audio data between the audio data for the first sentence and the audio data for the refined portion 19B.

[0078] Continuing with the specific example above, the refined portion 19B (e.g., "the most common reason for residential basement leaking is...") can be received or detected by the second response monitoring engine 117 after a predefined time period since the complete rendering of the audio data used for the first sentence. However, the refined portion 19B can be received / detected before the complete rendering of the audio data used for the second sentence ("There could be several reasons"), or within a predefined time period since the complete rendering of the audio data used for the second sentence. In these cases, the audio data for the refined portion 19B (e.g., "the most common reason for residential basement leaking is...") can be rendered after the audio data used for the second sentence. The audio data for the refined portion 19B can be rendered without any intermediate audio data between the audio data used for the second sentence and the audio data used for the refined portion 19B. For clarity, this document will not repeat the description of a similar case in which the refined portion 19B (or the second response 19) is received after a predefined time period since the full rendering of the audio data used for the second sentence.

[0079] In various implementations, once the second response 19 (or the refined portion 19B) is received, the second response monitoring engine 117 can pause or stop monitoring the presence of the second response 19 (or the refined portion 19B).

[0080] In some implementations, the first response 18 can be divided into a first part 18A and a second part 18B based on the position of the first sentence making a factual statement about one or more entities in user utterance 14B. For example, in response to the fact that the fourth sentence determining the first response 18 is the first sentence making a factual statement, the first response 18 can be divided into a first part 18A including the first, second, and third sentences and a second part 18A including the fourth sentence and any sentences following the fourth sentence. In some other implementations, the length of the first part 18A can be determined based on the type of the first LLM 190A. For example, the first LLM 190A can be trained to not make factual statements for the first few pairs of sentences of the first response 18. In this case, the first pair of sentences (e.g., a predetermined number of sentences at the very beginning of the first response 18) can be included in the first part 18A, while any sentences following the first pair of sentences are included in the second part 18B. It should be noted that in some implementations, the first response 18 can include and only include the first part 18A, where the entire first part 18A does not make any factual statements.

[0081] In various implementations, the first portion 18A of the first response 18 can be rendered at a predetermined speed. For example, the speed at which the first portion 18A of the first response 18 is rendered can be adjusted depending on the estimated network connection conditions between the client device 10 hosting the first LLM 190A and the server device hosting the second LLM 190B. Alternatively, the first portion 18A (or in particular, the first sentence S1) can be rendered with a specific tone (e.g., encouragement, concern, etc.) based on the type and / or content of the user utterance 14B.

[0082] In various implementations, the second portion 18B of the first response is not rendered to the user R via the client device 10. In some other implementations, the second portion 18B may be rendered to the user R under certain conditions (e.g., poor network connectivity between the client device 10 and the server device). For example, in response to determining that the second response 19 (or the refined portion 19B of the second response 19) has not been received within an additional predefined time period since the complete rendering of the last sentence (i.e., Sn) in the first portion 18A of the first response 18, the second portion 18B may be provided to the TTS engine 114 to generate audio data for the second portion 18B. The audio data for the second portion 18B may be rendered after the rendering of the sentence Sn. The additional predefined time period may be the same as or different from the predefined time period. For example, the additional predefined time period may be longer than the predefined time period. In this case, once the second response 19 or the refined portion 19B is received, the audio data generated for the received second response 19 (or refined portion 19B) may still be rendered after the entire rendering of the first response 18. For example, custom audio data, such as “Let me provide you with more accurate information here,” can be rendered before rendering the audio data generated for the received second response 19 (or the refined portion 19B) and after rendering the audio data for the first response 18.

[0083] It should be noted that, although Figure 1B and Figure 1C The instructions specify audible rendering of the first portion 18A and / or the refined portion 19B, but in various other implementations, these portions (e.g., 18A or 19B, or other aspects) may be visually rendered rather than audibly rendered. For example, in response to receiving typed input from a user via a user input device (e.g., a keyboard), the first portion (e.g., the first sentence of the first response 18 generated using the first LLM 190A) may be visually rendered to the user at the display, and the refined portion 19B generated using the second LLM 190B may also be visually rendered at the display, although the refined portion 19B may be rendered after, for example, 0.5 seconds (or other short time intervals) since the first sentence was visually rendered.

[0084] For example, given the typed input "how to fix a blank thermostat," the first sentence of the first response 18 (e.g., "There can be several reasons for a blank thermostat.") can be visually rendered as an immediate response to the typed input. The refined portion 19b can be visually rendered after the first sentence, suggesting—"The first thing you may want to do is to check or replace the battery for a battery-operated thermostat." The refined portion 19B can be rendered to provide more complex and accurate content than the second portion 18B (and it should be noted that the second portion 18B may be partially rendered or not rendered at all, if applicable). For example, in some implementations, the first sentence and / or the second part 18B will not make factual statements, but the refined part may include one or more factual statements about one or more entities referenced in the typed input (e.g., the year the concept or theory was introduced, the name of the author of the book or painting, the location of the event, the size of the company, the total amount of time spent in building the structure, etc.).

[0085] Now go to Figure 2 The diagram depicts a flowchart illustrating an example method for generating content in response to a user request for content generation, according to various aspects of this disclosure. For convenience, the operation of method 200 is described with reference to a system performing these operations. The system of method 300 includes one or more processors, memories, and / or other components of a computing device (e.g., client computing device 10 of FIG. 1, one or more servers, and / or other computing devices). Furthermore, although the operations of method 200 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0086] At box 201, the system receives a user query in natural language. In some implementations, the user query in natural language can be determined, for example, from user utterances received at a client device. For example, one or more components (e.g., the aforementioned ASR engine and / or NLU engine) can be used to process the user utterances to determine if the user utterances include a user query. In some implementations, the user query can be a query seeking information about a specific entity (e.g., an object, event, theory, etc.). As a non-limiting example, the user query could be “how to cook a turkey,” which can be determined from user utterances such as “okay, how to cook a turkey?”

[0087] As another example, the user utterance could be, for instance, “okay, how to cook a turkey? This is my first time cooking a turkey.” In this example, the first part of the user utterance corresponding to the user query “how to cook a turkey” can be processed, and the user intent to perform the action of searching for a turkey recipe can be performed, either by processing and performing or not by processing and performing the second part of the user utterance corresponding to the user statement “This is my first time cooking a turkey.” By partially performing the user intent corresponding to performing the action of searching for a turkey recipe (i.e., not further determining the user intent based on the user statement “This is my first time cooking a turkey”), the latency of rendering a response in response to the user query “how to cook turkey” can be reduced, while the computational resources for fully performing the user intent corresponding to the entire user utterance can be preserved.

[0088] In some implementations, user queries in natural language can be text input received from the user. For example, text input can be received via a touchpad, keyboard, or any other suitable input unit / device on the client device.

[0089] At box 203, in response to receiving a user query, the system performs one or more actions. For example, in response to receiving a user query, the system may use a first generative model to process the user query to generate a first natural language response in response to the user query (203A). The system may further cause a first portion of the first natural language response to be rendered (audibly or visually) (203B). In some implementations, the first generative model is a first large language model (LLM). In some implementations, the first portion of the first natural language response may be a single sentence (e.g., “cooking turkey can be a lot of fun”), which is generated as the first sentence of the natural language response. In some other implementations, the first portion of the first natural language response may include a predetermined number of sentences beginning with the first natural language response.

[0090] In some implementations, a first generative model can be used to generate a first model output based on a user query, wherein a first natural language response is determined based on the first model output. In some implementations, the first generative model can be trained to not make factual statements in a first part of the first natural language response, while allowing a second part of the first natural language response, or configuring it to include sentences that make factual statements (e.g., the time or time range typically required to cook a turkey) (or begin with a sentence that makes a factual statement). Therefore, the first part of the natural language response does not include factual statements about one or more entities identified from the user query.

[0091] In some other implementations, the first generative model can be trained or fine-tuned to avoid making any factual statements. In this case, the natural language response (including the first part and any additional parts, such as the second part) may not include factual statements at all.

[0092] It should be noted that while a first generative model is used to generate the first natural language response in various implementations, in some other implementations, a first generative model may not be used to determine / generate the first natural language response. For example, the first natural language response may include at least one default text string (e.g., a default sentence such as “Good question!”) ​​as the first part to be rendered by default and immediately (e.g., within 0.1 s) in response to the user query. This may be applicable, for example, as long as the user utterance is determined to include the user query (e.g., “Is burning bush invasive?”). As another example, the first natural language response may be generated based on a template (e.g., “So you'd like to know more about [entity]?”), where a template can be selected from multiple predefined templates, each having one or more slots to be filled based on the content of the user query. For example, the content of the user query (e.g., “What is Paris known for?”) may be an entity identified by the user query (e.g., “Paris”).

[0093] At box 205, the system generates a text prompt that includes the user query and a natural language request that generates a refined natural language response that responds to the user query and begins with the first part of the natural language response. For example, given the user query is "how to cook a turkey" and the first part is "cooking a turkey can be a lot of fun", the text prompt could be "draft a response that starts with 'cooking a turkey can be a lot of fun', responsive to the user query of 'how to cook turkey'". In some implementations, the text prompt may optionally include a word count limit for the refined natural language response.

[0094] In some implementations, instead of generating a refined natural language response that responds to the user query and begins with the first part of the natural language response, the natural language request may be a request that generates a refined natural language response that responds to the user query and follows (but does not include) the first part of the natural language response. For example, the text suggestion could be "draft a response responsive to the user query of 'how to cook turkey', following content of 'cooking a turkey can be a lot of fun'", instead of "draft a response that starts with 'cooking a turkey can be a lot of fun', responsive to the user query of 'how to cook a turkey'".

[0095] At box 207, the system provides the generated text prompt to the second generative model, thereby causing the second generative model to process the generated text prompt. The second generative model can be used to process the generated text prompt to generate a refined natural language response that begins with (or follows) the first part and includes the refined part that follows the first part.

[0096] The second generative model can be, for example, a second LLM. The second LLM can be larger than the first LLM by having a larger number of parameters. In various implementations, the first LLM can be a smaller LLM, and the second LLM can be a larger LLM, where the smaller LLM is a quantized and / or pruned version of the larger LLM. In some other implementations, the smaller LLM is not a quantized and / or pruned version of the larger LLM, but is completely independent of it. For example, the smaller LLM can have a different architecture relative to the larger LLM, and / or can be trained on a unique training dataset relative to the larger LLM. For example, the smaller LLM can have a smaller input dimension than the larger LLM, the smaller LLM can have a smaller output dimension than the larger LLM, and / or the smaller LLM can include various intermediate layers of different sizes and / or types relative to the larger LLM.

[0097] Smaller LLMs can be computationally more efficient than larger LLMs. For example, processing a request using a smaller LLM can occur with less latency compared to using a larger LLM. As another example, processing a request using a smaller LLM can utilize less memory, processor, and / or power resources compared to using a larger LLM. In some implementations, the smaller LLM can reside on a device at the client device, while the larger LLM can be located remotely from the client device. For example, the larger LLM can reside on a server device communicating with the client device. Using a smaller LLM (rather than a larger LLM) to generate the first natural language response and visually rendering a portion of the first natural language response in response to a user request can more quickly satisfy the information needs of the user providing the user's utterance.

[0098] It should be noted that although in various implementations the text hint at box 205 depends on the first part of the natural language response (indicating that the processing of the text hint follows the processing of the user query using the first generative model), in some implementations the text hint generated at box 205 may alternatively be based on the user query, but not include the natural language request to generate a refined natural language response that begins with or follows the first part of the natural language response. For example, the text hint at box 205 may be the same as the user query, or may include the user query. In this example, processing the text hint using the second generative model can be done in parallel with processing the user query using the first generative model (or generating the first natural language response using the aforementioned default text string or template). This can help reduce the latency of rendering the refined natural language response to the user who provided the user query.

[0099] At box 209, the system causes a refined portion of the refined natural language response to be rendered (e.g., audibly or visually) after rendering a first portion of the first natural language response. In some implementations, the refined portion of the refined natural language response may be rendered immediately after the first portion of the first natural language response, without rendering a second portion of the first natural language response (if present). For example, the refined portion may be audibly rendered after the first portion of the first natural language response, without any intermediate audio data (e.g., audio data for the second portion of the first natural language response) between the audio data for the first portion and the audio data for the refined portion.

[0100] In some implementations, the first part of the first natural language response may include inaccurate information (e.g., this could be due to inaccurate speech recognition, reflecting an inaccurate understanding of the user's query, or even including inaccurate factual statements). In these implementations, the refined part of the refined natural language response may include statements that correct the inaccurate information in the first part of the first natural language response. For example, a user might provide the statement "How about visiting Wales for the spring vacation," and the first part of the first natural language response could be the sentence "Whales are one of the most beautiful animals on earth." For example, such a sentence could be generated based on inaccurate speech recognition and / or natural language understanding that confuses the animal "whales" with the country "whales." In this example, the output of a second generative model based on the processing of text prompts (which includes, for example, generating a refined response to the user query “How about visiting Wales for the spring vacation”, which will be followed by the sentence “Whales are one of the most beautiful animals on earth”) (this second generative model can be fully trained using a large dataset including content such as “the best whale-watching season in the US is summer to fall” and “Wales has some of the most beautiful beaches in the world to visit”) can indicate that the sentence “Whales are one of the most beautiful animals on earth” contains inaccurate information for the user query.

[0101] Continuing with the example above, the output of the second generative model can be processed to determine a refined portion that includes statements used to correct inaccuracies in the first part of the natural language response (in this example, the sentence "Whales are one of the most beautiful animals on earth"). For example, the refined portion might include statements such as "Orperhaps you mean Wales the country?" to correct inaccuracies in the first part (e.g., by mentioning the animal "whales"), followed by additional information such as "In which case, I'd like to introduce some nice beaches to visit in Wales…". In other words, in this example, after the audible rendering of “Whales are one of the most beautiful animals on earth” (the first part of the natural language response), the refined part can be audibly rendered with the statement “Or perhaps you mean Wales the country?” to correct inaccurate information, and further rendered with the additional information “In which case, I'd like to introduce some nice beaches to visit in Wales…” to provide more accurate and user-expected information.

[0102] It should be noted that a user query can be determined from text input rather than audible input (e.g., user speech). In this case, if the refined portion of the refined natural language response indicates that the first part of the natural language response includes inaccurate information, the first part of the natural language response (including inaccurate information) visually rendered within the user interface of the client device can be erased by the system and replaced with the refined portion of the refined natural language response. For example, a user query could be used to explain "general relativity" based on text input "explain general relativity," and the first part of the natural language response generated by the first generative model could be "General relativity has become the foundation for today's understanding of the cosmos. It is a theory developed by Einstein in 1905," and thus include inaccurate information (i.e., general relativity was developed by Einstein in 1915, while special relativity was developed by Einstein in 1905). In this scenario, the refined portion of the refined natural language response may begin with or include the content “It is a theory developed by Einstein in 1915, and is the current description of gravitation in modern physics…”, indicating the presence of inaccurate information in the first portion of the natural language response rendered to the user within the previously mentioned user interface on the client device. In response to the determination that the refined portion indicates the first portion includes inaccurate information, the inaccurate information rendered at the user interface (e.g., “It is a theory developed by Einstein in 1905”) may be erased (e.g., word-by-word or all at once) and replaced with the refined portion.In this scenario, after replacing the inaccurate information with the refined portion, the content displayed at the user interface could be, for example, "General relativity has become the foundation for today's understanding of the cosmos. It is a theory developed by Einstein in 1915, and is the current description of gravitation in modern physics…". In other words, when the first part and the refined portion contradict each other, the system described in this disclosure can provide a reverse editing process that removes the inaccurate information in the first part without rendering it, resulting in a more accurate visual rendering of the refined portion.

[0103] In some implementations, the system can further estimate the latency between the rendering of the first part of the natural language response and the reception of the refined part of the natural language response. In some implementations, the system can dynamically adjust the rate at which the first part of the natural language response is audibly rendered based on the estimated latency. This reduces the time interval between the rendering of the first part and the rendering of the refined part.

[0104] Now go to Figure 3 The diagram depicts a flowchart illustrating an example method for generating content in response to a user request for content generation, according to various aspects of this disclosure. For convenience, the operation of method 300 is described with reference to a system performing the operation. This system of method 300 includes one or more processors, memories, and / or other components of a computing device (e.g., client computing device 10 of FIG. 1, one or more servers, and / or other computing devices). Furthermore, although the operations of method 300 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0105] At box 301, the system receives a user query in natural language. In some implementations, the user query in natural language can be text input received from the user, or it can be speech recognition from the user's spoken input.

[0106] At box 303, the system performs one or more actions in response to receiving a user query. These actions may include generating a natural language response to the user query (303A), and / or causing a first portion of the natural language response to be audibly rendered (303B). In some implementations, the first portion of the natural language response is the first sentence of the natural language response. In some implementations, alternatively or additionally, the first portion of the natural language response does not include factual statements. For example, if the user query includes one or more entities, the first portion of the natural language response may not include factual statements about the one or more entities identified from the user query.

[0107] In some implementations, the system may generate a text prompt (box 305) based on a first portion of the user query and / or the natural language response, and provide the generated text prompt to a generative model (box 307). In some implementations, the system provides the generated text prompt to a generative model (e.g., an LLM), causing the generative model to process the generated text prompt as input to generate a generative model output from which a refined natural language response is determined. The system may generate the text prompt before or simultaneously with the audible rendering of the first portion of the natural language response. The text prompt may include, for example, a user query and a first portion of the natural language model. In this case, the system may generate the text prompt in response to the determination of the first portion of the natural language response.

[0108] In some implementations, the text prompt may include a user query and a natural language request / instruction that generates a refined natural language response to the user query, beginning with a first part of the natural language response. In some implementations, a generative model may be used to process the generated text prompt to generate a model output from which a refined natural language response is derived, wherein the refined natural language response begins with a first part and includes a refined portion following the first part.

[0109] In some implementations, templates are used to generate the natural language response (at box 303A), where one or more fields of the template are populated with information parsed from the user query. In some implementations, an additional generative model can be used to generate the natural language response, where the additional generative model includes fewer parameters than the generative model. In some implementations, the additional generative model is trained or fine-tuned to avoid making factual statements (e.g., at least for the first part of the natural language response).

[0110] In some implementations, at box 309, the system may cause the refined portion of the refined natural language response to be audibly rendered after the first portion of the natural language response, without rendering the second portion of the natural language response (e.g., in between).

[0111] Now go to Figure 4 The diagram depicts a flowchart illustrating another example method for generating content in response to a user request for content generation, according to various aspects of this disclosure. The system of method 400 includes one or more processors, memories, and / or other components of a computing device (e.g., client computing device 10 of FIG. 1, one or more servers, and / or other computing devices). Furthermore, although the operations of method 400 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0112] At box 401, the system receives user queries in natural language. User queries in natural language can be determined from the user's spoken words (or other types of user input, such as touch input or typed input). For example, as mentioned above, a user query could be a query about the cause of a basement leak (or other event). In other examples, a user query could be a query used to describe or introduce a theory or other object.

[0113] At box 403, in response to receiving a user query, the system generates a natural language response to the user query, wherein the natural language response begins with one or more sentences that do not make a factual statement. In some implementations, optionally, the natural language response may include, and only includes, one or more sentences that do not make a factual statement. In some other implementations, optionally, the natural language response may further include one or more additional sentences following the one or more sentences that do not make a factual statement, wherein the one or more additional sentences may each include a sentence that makes a factual statement.

[0114] In some implementations, a natural language response can be generated by populating one or more slots in a template with content determined from a user query presented in natural language. In other implementations, the natural language response can be generated based on processing the user query using a generative model (e.g., the smaller LLM mentioned earlier) (or hints generated based on the user query).

[0115] At box 405, the system performs one or more actions. For example, the system may cause the first sentence of the natural language response to be rendered via a computing device (405A) and may generate a text prompt to be processed using a generative model (e.g., the larger LLM mentioned earlier) (405B). The text prompt may be generated based at least on the user query. For example, the text prompt may be generated to include only the user query, or to include both the user query and the first sentence of the natural language response. The text prompt may be processed using a generative model (e.g., the larger LLM) to generate a model output from which a refined natural language response is determined. The system may further monitor the receipt of the refined natural language response (405C).

[0116] In some implementations, the text prompt may include a user query. In some implementations, the text prompt may further include an instruction to configure a refined natural language response to begin with a first sentence of the natural language response (which does not make a factual statement). In some implementations, alternatively, the text prompt may further include an instruction to configure a refined natural language response to begin with one or more sentences of the natural language response that do not make factual statements.

[0117] At box 407, in response to determining that a refined natural language response (or a refined portion of a refined natural language response) has been received, the system may cause the refined portion of the refined natural language response to be rendered. The refined portion of the refined natural language response may be content of the refined natural language response that is not included in the natural language response itself. For example, the refined natural language response may include the first sentence of the natural language response (because the instructions for generating the refined natural language response require the refined natural language response to begin with the first sentence from the natural language response). In this example, the refined portion of the refined natural language response may be a refined natural language response that excludes the first sentence (e.g., the first sentence is from the natural language response and the refined natural language response begins with that first sentence).

[0118] In some implementations, in response to determining that a refined natural language response (or a refined portion of a refined natural language response) has not been received (e.g., when the first sentence has been rendered or within a predefined time period after the rendering of the first sentence), the system may cause a second sentence of the natural language response to be rendered while continuing to monitor the presence of the refined natural language response (or a refined portion thereof).

[0119] Now go to Figure 5The diagram depicts a block diagram of an example computing device 510 that can be optionally used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based LLM-based assistant component, and / or other components may include one or more components of the example computing device 510.

[0120] Computing device 510 typically includes at least one processor 514 that communicates with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow users to interact with computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0121] User interface input device 522 may include a keyboard, pointing devices (such as a mouse, trackball, touchpad or graphics tablet, scanner, touchscreen integrated into the display), audio input devices (such as a voice recognition system, microphone), and / or other types of input devices. Generally, the term "input device" is intended to include all possible types of means and methods for inputting information into computing device 510 or into a communication network.

[0122] User interface output device 520 may include a display subsystem, printer, fax machine, or non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is intended to include all possible types of means and methods for outputting information from computing device 510 to a user or another machine or computing device.

[0123] Storage subsystem 524 stores the functional programming and data constructs of some or all of the modules described herein. For example, storage subsystem 524 may include selected aspects for performing the methods disclosed herein, as well as logic for implementing the various components depicted in Figure 1.

[0124] These software modules are typically executed by processor 514 alone or in conjunction with other processors. The memory 525 used in storage subsystem 524 may include multiple memories, including main random access memory (RAM) 530 for storing instructions and data during program execution and read-only memory (ROM) 532 for storing fixed instructions. File storage subsystem 526 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing certain functionalities of the implementation may be stored in file storage subsystem 526 within storage subsystem 524, or in other machines accessible to processor 514.

[0125] Bus subsystem 512 provides mechanisms for enabling various components and subsystems of computing device 510 to communicate with each other as intended. Although bus subsystem 512 is schematically shown as a single bus, alternative implementations of bus subsystem 512 may use multiple buses.

[0126] The computing device 510 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly changing nature of computers and networks, therefore... Figure 5 The description of the computing device 510 is intended only as a specific example for illustrating some implementation methods. Many other configurations of the computing device 510 are possible, and these configurations are related to… Figure 5 The computing device depicted has more or fewer components compared to the one described.

[0127] In situations where the systems described herein collect or otherwise monitor personal information about users, or may utilize personal information and / or monitored information, users may be provided with the opportunity to control whether programs or features collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from content servers. Additionally, certain data may be processed in one or more ways before it is stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed to the point that the user's personally identifiable information cannot be determined, or, in the case of obtaining geographic location information, the user's geographic location may be generalized (e.g., to the city, zip code, or state level) to the point that the user's specific geographic location cannot be determined. Therefore, users can control how information about themselves is collected and / or used.

[0128] Some other implementations disclosed in this paper recognize that training a generative model may require a large number (e.g., millions) of training instances. Due to the need for such a large number of training instances, many of these instances will lack the input and / or output properties expected when deploying the generative model for use. For example, some training instance outputs of an LLM may be undesirably grammatically incorrect, undesirably overly concise, undesirably overly robust, etc. Furthermore, for example, some training instance inputs of an LLM may lack expected contextual data, such as user attributes associated with the input, conversation history associated with the input, etc. Because of the many missing expected input and / or output properties in LLM training instances, LLM will generate many output instances that similarly lack expected output properties after training and upon deployment.

[0129] Additionally, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein these instructions are configured to cause the execution of any of the methods described above. Some implementations also include one or more transient or non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the methods described above. Some implementations also include a computer program product comprising instructions executable by one or more processors to perform any of the methods described above.

[0130] While several implementations have been described and illustrated herein, a variety of other means and / or structures may be utilized to perform functions and / or obtain results and / or one or more of the advantages described herein, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize, or can determine, many equivalents of the particular implementations described herein using routine experimentation alone. Therefore, it should be understood that the foregoing implementations are presented by way of example only, and it should be understood that implementations may be practiced in ways other than those specifically described and claimed within the scope of the appended claims and their equivalents. Implementations of this disclosure relate to each individual feature, system, and / or method described herein. Furthermore, any combination of two or more such features, systems, and / or methods, provided that such features, systems, and / or methods are not inconsistent with each other, is included within the scope of this disclosure.

[0131] Among various implementations, one method is provided that uses one or more processors, and the method includes: receiving a user query comprising natural language, the user query being generated based on user interface input at a client device. The user interface input that generates the user query in natural language can be typed input or spoken input (e.g., from a human user).

[0132] In various implementations, the method further includes: in response to receiving a user query, processing the user query (e.g., using a template or a first generative model) to generate a natural language response to the user query. In some implementations, a template may be selected from a plurality of pre-configured templates based on the type (and / or content) of the user query. In some implementations, the template may include one or more pre-configured sentences having one or more slots to be filled based on the content (and / or context, such as location, time, etc.) of the user query. For example, given the user query is “Why my thermostat is blank,” a template containing content such as “Oh no, sorry to hear that. There are several reasons why __…” can be selected (e.g., based on a user utterance including keywords such as “why”), and the slots of the selected template can be filled with the content of the user utterance (e.g., “my thermostat is blank”) to complete the template. This can generate a complete template, such as "Oh no, sorry to hear that. There are several reasons why a thermostat is blank." The complete template can then be used as a natural language response to a user's query.

[0133] In some implementations, instead of using templates, a first generative model (e.g., the smaller LLM mentioned earlier) can be used to process the user query as input to generate a first model output from which a natural language response in response to the user query is determined / generated. The first generative model can be trained (or fine-tuned) to generate natural language responses of a specific length (e.g., less than 40 words, 25 to 45 words, etc.), specific content reflecting a particular tone (e.g., concern, etc.) in response to the user query (e.g., “Oh no…”), and / or without factual statements (e.g., for the entire natural language response, for the natural language response starting with a predetermined number of sentences, etc.).

[0134] In some implementations, the natural language response (e.g., generated using a template or a first generative model) may include a first part and / or a second part following the first part. The first part of the natural language response may include (and sometimes only include) the first sentence of the natural language response that begins with it. In some implementations, the first part of the natural language response may, for example, exclude factual statements about one or more entities queried by the user. In some implementations, the second part of the natural language response may (but must) include factual statements about one or more entities queried by the user.

[0135] In various implementations, the method further includes: in response to receiving a user query, causing at least a first portion of a natural language response to be audibly rendered. In some implementations, the first portion of the natural language response may be audibly rendered via a client device receiving the user query. In some implementations, the first portion of the natural language response may be audibly rendered via a device different from (but communicating with) the client device receiving the user query. In some implementations, optionally, the first portion of the natural language response may be audibly rendered at a speed determined based on the length of the first portion. For example, if the first portion of the natural language response has a short length (e.g., less than a first predetermined number of words), the speed may be slow; or if the first portion of the natural language response has a long length (e.g., greater than a second predetermined number of words), the speed may be fast; or if the first portion of the natural language response has a length between the first and second predetermined numbers of words, the speed may be medium.

[0136] In various implementations, the method further includes: in response to receiving a user query, generating a prompt that includes at least the user query (or a portion thereof) and a natural language request that generates a response to the user query (e.g., a refined response). For example, the prompt could be “the user query is 'why is my thermostat blank', generate a response to this user query”. In some different versions of these implementations, the prompt may include the user query, a first part of the natural language response, and a natural language request that responds to the user query and will follow the first part of the natural language response.

[0137] For example, a natural language response could include the first part of "Oh no, sorry to hear that." In this example, the prompt could be: "The user query is 'why is mythermostat blank', generate a refined response to this user query starting with the sentence(s) of 'Oh no, sorry to hear that'." Alternatively, the prompt could be: "The user query is 'why is my thermostat blank', generate a refined response to this user query to follow with the sentence(s) of 'Oh no, sorry to hear that'." In the latter case, the sentence "Oh no, sorry to hear that." does not need to be repeatedly included in both the natural language response and the refined response in order to save computational resources associated with generating the refined response and / or network resources associated with transmitting the refined response. As another example, the natural language response may include the first part of "Oh no, sorry to hear that. There are several reasons why a thermostat is blank". In this example, the prompt could be, for example, "the user query is 'why is my thermostat blank', generate a refined response to this user query that follows sentence(s) of 'Oh no, sorry to hear that. There are several reasons why a thermostat is blank'".These examples are provided here for illustrative purposes only and are not intended to be limiting.

[0138] In various implementations, the method further includes: in response to receiving a user query, causing a second generative model to process the generated prompt, thereby producing a refined natural language response that includes a refined portion following the first portion. In some implementations, the refined natural language response may include the first portion, wherein the refined portion follows the first portion. In some other implementations, the refined natural language response does not include the first portion, but includes only the refined portion, wherein the refined portion is rendered after the rendering of the first portion.

[0139] In some implementations, the second generative model includes a significantly larger number of parameters than the first generative model. For example, the first generative model could be a smaller Large Language Model (LLM) with fewer than 100 billion parameters, while the second generative model could be a larger LLM with over 200 billion parameters. Because the second generative model is a larger LLM, the refined portion can provide more accurate, complex, and / or user-expected content in response to user queries compared to the second portion (especially when the second portion contains one or more factual statements (e.g., "leaking") of one or more entities mentioned in the user query).

[0140] In some implementations, the first generative model (e.g., a smaller LLM) may be a quantized and / or pruned version of the second generative model (e.g., a larger LLM). In other implementations, the first generative model (e.g., a smaller LLM) is not a quantized and / or pruned version of the second generative model (e.g., a larger LLM), but is completely independent of the second generative model. For example, the first generative model may have a different architecture than the second generative model, and / or may be trained on a unique training dataset relative to the second generative model. For example, the input size of the first generative model may be smaller than the input size of the second generative model, the output size of the first generative model may be smaller than the output size of the second generative model, and / or the first generative model may include various intermediate layers that differ in size and / or type from those layers of the second generative model.

[0141] In some implementations, the first generative model can be computationally more efficient than the second generative model. For example, processing a request using the first generative model can be done with less latency compared to using the second generative model. As another example, processing a request using the first generative model can utilize less memory, processor, and / or power resources compared to using the second generative model. In some implementations, the first generative model can be on a device at the client device, while the second generative model can be located remotely from the client device. For example, the second generative model can be on a server device communicating with the client device. Using the first generative model (rather than the second generative model) to generate initial content and visually rendering the generated initial content in response to a user request can more quickly satisfy the information needs of users providing spoken utterances or user requests.

[0142] In some implementations, the first generative model is stored at the client device, and the second generative model is stored at a server device located remotely from the client device. For example, memory constraints on the client device may prevent the second generative model from being used or stored at the client device. In some other implementations, both the first and second generative models are stored at the client device (e.g., if sufficient memory exists at the client device). In some still other implementations, both the first and second generative models are stored at the server device.

[0143] In some implementations, at least a portion of the first part of the natural language response is audibly rendered before generating the entire refined response. In some implementations, causing at least the first part of the natural language response to be audibly rendered includes causing only the first part of the natural language response to be rendered.

[0144] In various implementations, the method further includes: in response to receiving a user query, causing a refined portion of the refined natural language response to be audibly rendered. In some implementations, the refined portion of the refined natural language response is audibly rendered after the rendering of the first portion. Optionally, the refined portion of the refined natural language response is audibly rendered after the rendering of the first portion, while a second portion of the natural language response is not audibly rendered in between (i.e., between the first portion and the refined portion).

[0145] In some implementations, the refined portion of the refined natural language response is audibly rendered immediately after the first portion. Here, "immediately after" can mean rendering the refined portion within 1 second, 0.5 seconds, 0.1 seconds, or other sufficiently short duration after the first portion is rendered, with no intermediate audible output between the first and refined portions. By rendering the first portion immediately in response to a user query (e.g., within 1 second, 0.5 seconds, etc.) and then immediately afterward rendering the refined portion (instead of the first portion), not only is user engagement in the human-computer dialogue rapid, but responses with enhanced accuracy (e.g., more accurate factual statements) are also provided.

[0146] In various implementations, the method further includes: determining an estimated latency for receiving the refined response before causing the generated prompt to be processed by the second generative model; and determining the length of a first portion from the natural language response based on the estimated latency. In various implementations, determining the estimated latency may be based on a measurement or anticipated current server load associated with one or more servers hosting the second generative model. As a non-limiting example, the natural language response may include a first sentence and a second sentence totaling 20 words. In this non-limiting example, if the current server load associated with one or more servers hosting the second generative model is determined to be heavy, the system can determine the estimated latency and determine the number of sentences to be rendered based on the estimated latency. For example, given a heavy current server load, all 20 words of the natural language response (including the first and second sentences) may be rendered as an immediate response to a user query. If the current server load is determined to be light, the first sentence (which may have 10 words or 9 words, etc.) may be rendered as an immediate response to a user query, and the second sentence is not rendered after the first sentence.

[0147] In some implementations, alternatively or additionally, the estimated latency may be determined based on user queries (e.g., based on the length and / or type of user queries). In some implementations, the first part may be rendered audibly at a specific speed, wherein the specific speed may depend on the estimated latency determined based on the current server load (which is associated with one or more servers hosting the second generative model).

[0148] Among various implementations, another method implemented using one or more processors is provided, and the method includes: receiving a user query comprising natural language, the user query being generated based on user interface input at a client device; in response to receiving the user query: generating a natural language response in response to the user query, causing at least a first portion of the natural language response to be audibly rendered; and generating a text prompt for a natural language request comprising the user query, the first portion of the natural language response, and generating a refined response in response to the user query and to follow the first portion of the natural language response. The method may further include: in response to receiving the user query and while audibly rendering the first portion of the natural language response: causing a generative model to process the generated text prompt, thereby producing a refined natural language response comprising the refined portion to follow the first portion. The method may further include: after the audible rendering of the first portion of the natural language response, causing the refined portion of the refined natural language response to be audibly rendered. The refined portion of the refined natural language response can be audibly rendered after the first part of the natural language response has been audibly rendered, without any intermediate audible rendering (e.g., excluding the audible rendering of the second part of the natural language response).

[0149] Among various implementations, another method is provided that uses one or more processors, and the method includes: receiving a user query comprising natural language, the user query being generated based on user interface input at a client device. In response to receiving the user query, the method further includes: generating a natural language response to the user query; causing at least a first portion of the natural language response to be rendered via the client device; while the first portion of the natural language response is being rendered via the client device: generating a prompt comprising the user query, the first portion of the natural language response, and generating a refined response in response to the user query and to follow the first portion of the natural language response. The method further includes: causing a generative model to process the generated text prompt, thereby producing a refined natural language response that includes the refined portion following the first portion; determining that no refined portion of the refined natural language response was received when the rendering of the first portion was completed; and in response to determining that no refined portion of the refined natural language response was received when the rendering of the first portion was completed, causing sentences in a second portion of the natural language response following the first portion to be rendered, and causing the refined portion of the refined natural language response to be rendered after sentences in the second portion.

[0150] In various implementations, instead of using a first LLM to process the user query (which is determined from the user utterance), the first LLM can be used to process the portion of the user utterance determined to contain the complete user query, before or simultaneously with the processing of additional parts of the user utterance (e.g., to determine the entire transcription of the user utterance). The user utterance could be, for example, “okay, how to cook a turkey? This is my first time cooking a turkey.” In this case, the first part of the transcription of the user utterance corresponding to “how to cook a turkey” can be processed and determined to include the complete user query “how to cook a turkey” in natural language. Before or simultaneously with the determination / generation of additional parts of the transcription of the user utterance corresponding to “This is my first time cooking a turkey”, the first LLM (e.g., a smaller LLM) can be used to process the first part of the transcription as input to generate an immediate response to the user utterance. The immediate response can be rendered in response to the user utterance. Instead of rendering the entire instant response, a portion of the instant response, such as the first sentence of the instant response, can be rendered in response to the user's words.

[0151] In the example above, after identifying the additional portion of the transcription, a second LLM (e.g., a larger LLM) can be used to process the entire transcription of the user's statement ("okay, how to cook a turkey? This is my first time cooking a turkey") as input to generate a refined response. The refined response can include a refined portion that follows or will follow the first sentence of the immediate response. The refined portion can be rendered immediately after the first sentence of the immediate response. Regarding the immediate response, the refined portion can be more user-specific / desired (as specified in the user's statement—"This is my first time cooking a turkey") by including a recipe friendly to those with little or no cooking experience or who have never cooked turkey before. By partially fulfilling the user intent corresponding to the action of searching for a turkey recipe (i.e., not further determining / modifying the user intent based on the user statement "This is my first time cooking a turkey"), the latency of rendering the immediate response (or a portion thereof) in response to the user query "how to cook a turkey" can be reduced. By providing a more refined portion of the response immediately following the immediate response, a more accurate or expected response can be provided in response to the user's words.

Claims

1. A method implemented using one or more processors, the method comprising: Receive user queries, including natural language queries, which are generated based on user interface input at the client device; as well as In response to receiving the user query: The first generative model is used to process the user query to generate a natural language response to the user query. This causes at least a first portion of the natural language response to be rendered; Generate a prompt, the generated prompt comprising the user query, the first part of the natural language response, and a natural language request to generate a refined response to the user query and to follow the first part of the natural language response; This causes the generated prompts to be processed by a second generative model, resulting in a refined natural language response that includes the refined portion after the first portion. as well as This causes the refined portion of the refined natural language response to be rendered.

2. The method as described in claim 1, wherein, The user interface input used to generate the user query is either typed input or verbal input.

3. The method as described in any of the preceding claims, wherein, The second generative model includes a larger number of parameters than the first generative model.

4. The method as described in any of the preceding claims, wherein, The first part of the natural language response is the first sentence of the natural language response.

5. The method as described in any of the preceding claims, wherein, The first generative model is trained or fine-tuned to avoid making factual statements.

6. The method as described in any of the preceding claims, wherein, The first part of the natural language response does not include factual statements about one or more entities queried by the user.

7. The method of any of the preceding claims, further comprising: Estimate the latency between the rendering of the first portion of the natural language response and the reception of the refined portion of the refined natural language response; as well as The speed at which the first part of the natural language response will be rendered is determined based on the latency; The rendering of the first part of the natural language response includes rendering the first part audibly at the first speed.

8. The method as described in any of the preceding claims, wherein, Before generating the entire refined response, at least a portion of the first part of the natural language response is rendered.

9. The method as described in any of the preceding claims, wherein, The first generative model is stored at the client device, and the second generative model is stored at a server device located remotely from the client device.

10. The method of claim 9, wherein, The memory constraints of the client device prevent the second generative model from being utilized on the client device.

11. The method as claimed in any of the preceding claims, wherein, Causing at least the first portion of the natural language response to be rendered includes causing only the first portion of the natural language response to be rendered.

12. The method of any of the preceding claims, further comprising: Before the generated prompt is processed by the second generative model, the estimated delay for receiving the refined response is determined; The first portion is determined from the natural language response based on the estimated latency.

13. The method of claim 12, wherein, The estimated delay is determined based on: The measured or anticipated current server load associated with one or more servers hosting the second generative model; and / or The length of the user query.

14. The method as claimed in any of the preceding claims, wherein, The natural language response includes a second part following the first part, and wherein the refined portion of the refined natural language response is rendered after the rendering of the first part, while the second part of the natural language response is not rendered in between.

15. A method implemented using one or more processors, the method comprising: Receive user queries, including natural language queries, which are generated based on user interface input at the client device; In response to receiving the user query: Generate a natural language response to the user's query; In response to the query, at least a first portion of the natural language response is rendered at the client device; Generate a text prompt, the generated text prompt including the user query, the first part of the natural language response, and a natural language request to generate a refined response to the user query and to follow the first part of the natural language response; While the first part of the natural language response is being rendered: This leads to the use of a generative model to process the generated text prompts, thereby producing a refined natural language response, which includes a refined portion that follows the first portion; as well as This causes the refined portion of the refined natural language response to be rendered after the first portion of the natural language response has been rendered.

16. The method of claim 15, wherein, The natural language response is generated using an additional generative model, wherein the generative model includes a larger number of parameters than the additional generative model.

17. The method of claim 16, wherein, The additional generative model is trained or fine-tuned to avoid making factual statements.

18. The method according to any one of claims 15 to 17, wherein, The natural language response is generated using a template, wherein one or more fields of the template are populated with information from the user query.

19. The method according to any one of claims 15 to 18, wherein, The first part of the natural language response includes an inaccurate factual statement, and the refined part of the refined natural language response includes sentences that correct or modify the inaccurate factual statement.

20. A method implemented using one or more processors, the method comprising: Receive user queries, including natural language queries, which are generated based on user interface input at the client device; In response to receiving the user query, Generate a natural language response to the user's query; This causes at least a first portion of the natural language response to be rendered via the client device; While the first part of the natural language response is being rendered via the client device: Generate a prompt, the generated prompt including the user query, the first part of the natural language response, and a natural language request to generate a refined response to the user query and to follow the first part of the natural language response, and This leads to the use of a generative model to process the generated prompts, resulting in a refined natural language response that includes the refined portion after the first portion; Determine that the refined portion did not receive the refined natural language response when the rendering of the first portion was completed; as well as In response to determining that no refined natural language response was received when the rendering of the first portion was completed, the refined portion... This causes the sentence in the second part of the natural language response following the first part to be rendered, and This causes the refined portion of the refined natural language response to be rendered after the sentence in the second part.

21. The method of claim 19, wherein, The first part of the natural language response does not include factual statements, and the second part of the natural language response includes at least factual statements about one or more entities queried by the user.

22. A method implemented using one or more processors, the method comprising: Receive user queries, which are generated based on user interface input at the client device; as well as In response to receiving the user query: The user query is processed using a first generative model that is computationally more efficient than the second generative model, in order to generate at least an initial portion of the response to the user query. This causes the initial portion of the response to be rendered at the client device; Generate a prompt that includes the user's query; This causes the generated prompt to be processed by the second generative model to generate at least a subsequent part of the response; as well as This causes the subsequent portion of the response to be rendered at the client device after at least a portion of the initial portion of the response has been rendered.

23. The method of claim 22, further comprising: Estimate the latency between the rendering of the initial portion of the response and the rendering of the subsequent portion of the response; as well as The speed at which the initial portion of the response is to be rendered is determined based on the latency. The rendering of the initial portion of the response includes causing the initial portion of the response to be audibly rendered at the first speed.

24. The method of claim 22 or 23, wherein, Generating the prompt further includes incorporating the initial portion of the generated response into the prompt.

25. The method of claim 24, wherein, Generating the prompt further includes incorporating a natural language request that generates a response to the user query and includes content following the initial portion of the response into the prompt.

26. The method of claim 22 or claim 23, wherein, The second generative model is fine-tuned to predict the initial portion of the response generated using the first generative model.

Images