High-efficiency, low-latency LLM-based assistant hybrid inference

Abstract

The implementation leverages a hybrid approach using both smaller and larger LLMs to generate and refine content in response to user queries / requests for content generation. In various implementations, a smaller LLM is used to handle user queries for content generation, generating initial content in response to those queries. Text hints can be generated using both the user queries and the initial content, and these hints can be configured to further include requests for focused editing. A larger LLM can be used to handle such text hints to generate focused editing of the initiated content, refining the initial content and resulting in revised content (with improved accuracy) in response to user queries for content generation.

Description

Background Technology

[0001] Various generative models have been proposed that can be used to process natural language (NL) content and / or other inputs to generate outputs that reflect generative content in response to the inputs. For example, large language models (LLMs) have been developed that can be used to process NL content and / or other inputs to generate LLM outputs that reflect generative NL content and / or other generative content in response to the inputs. For example, LLM can be used to process NL content such as "how to change DNS settings on Acme router" to generate LLM output that reflects several responsive NL sentences, such as: "First, type the router's IP address in a browser; the default IP address is 192.168.1.1. Then enter the username and password; the defaults are admin and admin. Finally, select the advanced settings tab and find the DNS settings section." However, current utilization of generative models has one or more drawbacks.

[0002] As an example, many generative models can be very large, often including billions of parameters. For example (Exceeding 100 billion, 250 billion, or 500 billion parameters). Due to the sheer size of such generative models, significant amounts of memory, processors, power, and / or other computational resources may be required to process the inputs and generate the corresponding generative outputs. This resource utilization can be significant on a per-input basis and is particularly pronounced when hundreds or thousands of inputs are being processed per minute, per second, or other intervals. Furthermore, the large size of such generative models can lead to significant latency in generating the corresponding generative outputs and, consequently, in rendering the corresponding generative content. Such latency can result in prolonged user-computer interaction.

[0003] Smaller-sized counterparts to such generative models do exist, such as separately trained counterparts with fewer parameters, or pruned and / or quantized counterparts generated by applying one or more pruning and / or quantization techniques to a larger counterpart. For example, smaller counterparts to a larger model may have 25%, 33%, 50%, 66%, or other percentage fewer parameters than the larger model. However, such smaller-sized counterparts may be less robust and / or accurate than their larger-sized counterparts. Therefore, while processing inputs using such smaller-sized counterparts may be computationally more efficient and / or can be performed with less latency, there is a greater risk that the corresponding generative outputs generated by processing the inputs may be inaccurate and / or inadequately specified. Summary of the Invention

[0004] The implementation disclosed herein involves using a first generative model and a second generative model with different computational efficiencies to generate and refine content in response to user requests, respectively. For example, by having a smaller number of parameters, the first generative model can be more computationally efficient than the second generative model. As a non-limiting example, the first generative model could be a smaller large language model (LLM) with fewer than 100 billion parameters, while the second generative model could be a larger LLM including more than 200 billion parameters. In other examples, the first LLM could be a smaller LLM that includes 20%, 30%, 40%, 50%, or other percentage fewer parameters than a larger LLM.

[0005] By utilizing both a first generative model and a second generative model, the implementations of this disclosure improve the quality and / or accuracy of content provided in response to user requests, while reducing latency in providing such content. For example, various implementations utilize the computationally more efficient first generative model in generating initial content in response to user requests, enabling the initial content to be presented to the user with reduced latency, thereby shortening the overall duration of user / computer interaction. Various implementations also utilize the less computationally efficient second generative model in refining the initial content, enabling dynamic editing of the initial content to be presented / rendered to the user without additional user input or with a reduced amount of user input for manual editing. This mitigates the occurrence of inaccurate and / or inadequately specified initial content, and consequently mitigates computational and / or network inefficiencies caused by further user editing or subsequent (or new) user requests to address inaccuracies and / or inadequacies in the generated initial content.

[0006] In these and other ways, the implementations disclosed herein seek to reduce the overall duration of user / computer interaction by leveraging a computationally more efficient first generative model. Furthermore, those implementations also utilize a computationally less efficient second generative model to at least selectively refine the initial content generated based on the output generated using the first generative model, thereby mitigating inaccuracies and / or inadequate specification in the initial content by leveraging a more robust and / or accurate second generative model. Even further, those implementations seek to ensure that the refinement of the initial content is performed within the constraints of the initial content, so that the efficiency achieved by leveraging the computationally more efficient first generative model is not negated.

[0007] As a non-limiting example of work, a user request for generating content (also referred to as a "user request for content generation" or "user query for content generation") is received at a client device based on user interface input from a user. This user request queries for "write a 275-word email in a kind tone that talks about the need for innovation in a technology company." In response to the received user request for generating content, a first generative model (e.g., a computationally more efficient / smaller LLM) can be used to process the request to generate content, producing a first generative model output. Initial content responding to the user request can be generated based on the first generative model output, and this initial content can be visually rendered via the client device's user interface for user review and / or for receiving edits from the user. The initial content may include, for example, text (e.g., a message, email, report, post, etc.) that is responding to the user request for generating content and can be decoded from the first generative model output.

[0008] In the above non-limiting working example, text prompts can be generated based on user requests for content generation and the output of a first generative model, where the text prompts may further include requests for one or more focused edits. A request for one or more focused edits can be a request to generate a single focused edit (e.g., a single focused edit of a single text fragment such as a sentence, paragraph, or other type of text segment) in each of one or more iterations. For example, a request for one or more focused edits may include natural language text that requests a second generative model to perform one or more iterations of content refinement, where, for each iteration of content refinement, the model output of the second generative model is used to provide / derive a single focused edit of the single text fragment (e.g., a sentence, paragraph, etc.). Alternatively, a request for one or more focused edits can be a request to generate all one or more focused edits at once. Alternatively, a request for one or more focused edits can be a request to generate one or more focused edits in each iteration of content refinement.

[0009] In some implementations, text prompts may include user requests to generate content, initial content output (e.g., rendered via a user interface), and requests for one or more focused edits. For example, a request for one or more focused edits (sometimes referred to as a "request for focused edit(s)") could be a request to edit a single sentence (or paragraph, section, etc.) of the initial content, which is visually rendered via a user interface and generated based on the output of a first generative model. In other words, a request for one or more focused edits is not intended to edit the entire initial content all at once, nor is it a request to generate entirely new content based on a user request to generate content. Instead, it is a request to edit only a portion of the initial content (e.g., one or more text fragments, such as one or more sentences / paragraphs). Generating text prompts in these and other ways allows the robustness and / or accuracy of a computationally less efficient second generative model to be utilized, while limiting the amount of processing that would be performed using the second generative model and / or ensuring that the efficiency achieved by utilizing a computationally more efficient first generative model is not offset.

[0010] The generated text prompts can be fed to a second generative model (e.g., a less computationally efficient / larger LLM), causing the second generative model to process the generated text prompts to produce its output. Based on the second generative model output, revised content can be generated (e.g., by decoding the second generative model output), where the revised content includes LLM-based edits (sometimes referred to as "focused edits") of the initial content. For example, the LLM-based edits to the initial content reflect updated sentences that can be used to replace sentences in the initial content. The LLM-based edits to the initial content (e.g., sentence replacements) can be rendered automatically at the aforementioned user interface, or sentences can be highlighted or marked using graphical elements indicating the presence of changes, where the graphical elements can be (e.g.,...). For example Selectable via a single tap, single click, or other single user interface input, and when selected, causes visual rendering / execution of LLM-based editing of the initial content.

[0011] In some implementations, a smaller LLM can be a refined, quantized, and / or pruned version of a larger LLM. In other implementations, a smaller LLM is not a quantized and / or pruned version of a larger LLM, but is completely independent of the larger LLM. For example, a smaller LLM may have a different architecture relative to a larger LLM, and / or may be trained on a unique training dataset relative to a larger LLM. For example, the input dimension of a smaller LLM may be smaller than that of a larger LLM, the output dimension of a smaller LLM may be smaller than that of a larger LLM, and / or a smaller LLM may include various intermediate layers of different sizes and / or types relative to those of a larger LLM.

[0012] Smaller LLMs can be computationally more efficient than larger LLMs. For example, processing requests using a smaller LLM can occur with less latency compared to using a larger LLM. As another example, processing requests using a smaller LLM can utilize less memory, processor, and / or power resources compared to using a larger LLM. In some implementations, the smaller LLM can reside on a device at the client device, while the larger LLM can be located remotely at the client device. For example, the larger LLM can reside on a server device communicating with the client device.

[0013] Responding to user requests by using a smaller LLM (rather than a larger LLM) to generate initial content, and enabling visual rendering of the generated initial content, allows for a faster fulfillment of the information needs of the user who made the request. Utilizing a larger LLM to perform one or more focused edits on the initial content can produce refined / revised content with improved accuracy and satisfactory quality compared to the initial content. One or more focused edits can be visually rendered for users to review revisions to the initial content, for example, automatically and sentence-by-sentence.

[0014] In some implementations, users can use one or more user edits to edit the initial content and / or the refined / revised content. For example, once no focused edit is received (i.e., a focused edit generated based on the output of the second generative model), users can be allowed to edit the refined content. In other examples, users can be allowed to edit the initial content via the user interface as long as it is rendered at the user interface, and / or users can continue to edit the initial content while one or more focused edits are applied to it.

[0015] As a specific example, a first focus edit (in one or more focus edits) that replaces or modifies a text fragment (e.g., a single sentence, a single paragraph, or other type of text fragment) in the initial content can be rendered visually and dynamically via a display screen, followed by a second focus edit (in one or more focus edits) that replaces or modifies additional text fragments in the initial content. In this specific example, if the user wants to provide user editing while the second focus edit is being rendered via the display screen, the user can be allowed to edit multiple portions of the initial content (which can exclude specific portions of the initial content currently shown as modified / replaced by the second focus edit, as such specific portions are locked and therefore uneditable). In this specific example, the user can also be allowed to edit text fragments that have already been replaced or modified by the first focus edit. In other words, when focus editing is applied to content displayed via the display screen (which can be initial content or intermediate content showing sentences or other text fragments modified or replaced based on focus editing), the portions of the content modified by focus editing can be locked and not allowed to be edited by the user, while the rest of the content can be unlocked and allowed to be edited by the user. Intermediate content can refer to any content that is visually rendered (via the display screen) after the initial content but before the final content (which incorporates all focused edits determined using, for example, a larger LLM).

[0016] In some implementations, user requests for generating content can be processed to determine one or more query features, context features, and / or attribute features associated with the client device and / or the user providing the request. For example, when the request includes ( For example When generating natural language queries (either automatically or based on user interface input), one or more query features may include: terms of the query; embeddings of the terms of the query (e.g., generated using a separate encoder); the topic or domain reflected by the query; and / or other features that can be derived from the query. As another example, when the request includes a query with an image, query features may include: automatically generated captions for the image; descriptors of objects automatically detected in the image; and / or other features that can be derived from the image. Contextual features may be, or may include, first features related to, for example, the tone of the user request (if the user request is audible) determined based on audio data captured from the user request. For example, attribute features may be determined based on the user's profile.

[0017] In some implementations, smaller or larger LLMs are sequence-to-sequence models, which are transformer-based and / or may include encoders and / or decoders. A non-limiting example of an LLM is Google's Pathways Language Model (PaLM). Another non-limiting example of an LLM is Google's Language Model for Dialogue Applications (LaMDA).

[0018] The foregoing is presented as an overview of only some of the implementations disclosed herein. These and other implementations are disclosed in more detail herein. For example, additional and / or alternative implementations are disclosed herein, such as those involving the use of a second generative model to generate one or more focused edits. In some implementations, to generate one or more focused edits, a second generative model can be used for a first iteration of content refinement to process the aforementioned text hints (which include initial content, user requests for content generation, and requests for one or more focused edits) to generate a first second generative model output from which the first focused edit is determined. The first focused edit can be rendered at the user interface immediately after the first focused edit is determined. The first focused edit can be automatically applied to the text hints to update them, and the updated text hints can be processed using a second generative model for a second iteration of content refinement to generate a second second generative model output from which the second focused edit is determined. The second focused edit can be rendered at the user interface immediately after the second focused edit is determined. The second focus edit can be automatically applied to the updated text hint to further update the updated text hint, and the second generative model can be used to process the further updated text hint for the third iteration of content refinement to generate a third second generative model output from which the third focus edit is determined. This focus edit determination process can be repeated until the Nth second generative model output indicates that no focus edit is needed, where N is the total number of focus edits (i.e., N is a positive integer greater than zero).

[0019] Various implementations may include a non-transitory computer-readable storage medium storing instructions executable by a processor to perform one or more methods such as those described herein. Other implementations may include a system comprising a memory and one or more hardware processors operable to execute instructions stored in the memory to perform one or more methods such as those described herein. Attached Figure Description

[0020] Figure 1A A block diagram depicts an example environment that illustrates various aspects of this disclosure and in which some implementations of the disclosure herein may be implemented.

[0021] Figure 1B Examples are shown illustrating various aspects of this disclosure and in which some implementations of the disclosure can be implemented, including content generation and revision.

[0022] Figure 2A An example of a user interface that receives a user request for generated content, according to various aspects of this disclosure, is shown.

[0023] Figure 2B The illustration shows a response to receiving according to various aspects of this disclosure. Figure 2A An example of a user interface that generates initial content based on user requests.

[0024] Figure 2C The illustrations show various aspects of this disclosure. Figure 2B An example of a user interface that provides the initial focus for editing the first content.

[0025] Figure 2D The illustrations show various aspects of this disclosure. Figure 2B An example of a user interface for the first focused editing of the initial content.

[0026] Figure 2E An example of a user interface illustrating the rationale for the second focus editing, according to various aspects of this disclosure, is shown.

[0027] Figure 2F An example of a user interface for second-focus editing is shown according to various aspects of this disclosure.

[0028] Figure 2G An example of a user interface illustrating the rationale for third-focused editing, according to various aspects of this disclosure, is shown.

[0029] Figure 2H An example of a user interface for third-focus editing is shown according to various aspects of this disclosure.

[0030] Figure 3 A flowchart illustrating an example method for generating content in response to a user request for generated content, according to various aspects of this disclosure, is provided.

[0031] Figure 4 A flowchart is depicted illustrating another example method for generating content in response to a user's request for generated content, according to various aspects of this disclosure.

[0032] Figure 5 Example architectures of computing devices based on various implementations are depicted. Detailed Implementation

[0033] The following description, provided with reference to the accompanying drawings, is intended to help understand various implementations of this disclosure. It should be understood that different features from different embodiments can be combined and / or interchanged with each other. Furthermore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. For clarity and brevity, descriptions of well-known or repetitive functions and constructions may be omitted.

[0034] The terms and words used in the following description and claims are not limited to their bibliographical meaning and are used solely by the inventors to achieve a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the disclosure as defined by the appended claims and their equivalents.

[0035] Figure 1A This is a block diagram illustrating various aspects of this disclosure and in which the implementations disclosed herein can be carried out. For example... Figure 1A As shown, environment 100 may include client computing device 10 (“client device”) and server computing device 12 (“server device”) communicating with client computing device 10 via one or more networks 13. One or more networks 15 may include, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, and / or any other suitable network.

[0036] The client computing device 10 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a vehicle computing device (e.g., an in-vehicle entertainment system), an interactive speaker, a smart appliance such as a smart TV, and / or a wearable device that includes a computing device (e.g., glasses with a computing device, a virtual or augmented reality computing device), and this disclosure is not limited thereto.

[0037] In various implementations, the client computing device 10 may include a user input engine 101 configured to detect user input provided by a user of the client computing device 10 using one or more user interface input devices. For example, the client computing device 10 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to the user's spoken words or other sounds in the environment of the client computing device 10. Additionally or alternatively, the client computing device 10 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or motion detected in the field of view of one or more of the visual components. For example (Gestures) correspond to each other.

[0038] Additionally or alternatively, the client computing device 10 may be equipped with one or more touch-sensitive components. For example Keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons wait The one or more touch-sensitive components are configured to capture signals corresponding to touch input directed at the client computing device 10. Some examples of queries that can be included in a request as described herein can be queries based on user input provided by the user of the client computing device 10 and detected by the user input engine 101. For example, a query can be a typed query entered via a physical or virtual keyboard, a suggested query selected via a touchscreen or mouse, a spoken voice query detected by the microphone of the client device, or an image query based on an image captured by the visual components of the client device.

[0039] In various implementations, the client computing device 10 may include a local LLM-based assistant 11, a rendering engine 110, and / or storage 115. In various implementations, the rendering engine 110 may be configured to provide audible and / or visually presented content to a user of the client computing device 10 using one or more user interface output devices. For example (Natural language-based responses generated by LLM). For example, the client computing device 10 may be equipped with one or more speakers that enable the provision of audible content to a user via the client computing device 10. Additionally or alternatively, the client computing device 10 may be equipped with a display or projector that enables the provision of visual content to a user via the client computing device 10.

[0040] In various implementations, the native LLM-based assistant 11 may include multiple components, including an automatic speech recognition (ASR) engine 111, a natural language understanding (NLU) engine 112, a performance engine 112, and / or a text-to-speech (TTS) engine 114. The native LLM-based assistant 11 may optionally include a content generation system (e.g., Figure 1B 18) or a portion thereof (e.g., the content generation engine of a content generation system). In some implementations, the user R of the client computing device 10 may have a registered account associated with a local LLM-based assistant 11 and / or other third-party applications. Third-party applications may include, for example, note-taking applications, shopping applications, messaging applications, and / or any other suitable applications (or services) installed on (or accessible via) the client computing device 10.

[0041] Server computing device 12 may be, for example, a web server, a proxy server, a VPN server, or any other type of server as required. Server computing device 12 may include a cloud-based LLM-based assistant 120. In various implementations, the cloud-based LLM-based assistant 120 installed on server computing device 12 may communicate with a local LLM-based assistant 11 installed on client computing device 10. The cloud-based LLM-based assistant 120 may include cloud-based components that are the same as or similar to multiple local components of the local LLM-based assistant 11 installed locally on client computing device 1. For example, cloud-based LLM-based assistant 131 may include a cloud-based ASR engine 121, a cloud-based NLU engine 122, a cloud-based fulfillment engine 123, and / or a cloud-based TTS engine 124.

[0042] Optionally, the cloud-based LLM-based assistant 120 may include a content generation system (e.g., Figure 1B 18) or a portion thereof (e.g., content refinement engine). The local LLM-based assistant 11 and the cloud-based LLM-based assistant 120 may be referred to as "LLM-based assistant".

[0043] ASR engine 111 (and / or cloud-based ASR engine 121) can use one or more streaming ASR models (e.g., recurrent neural network (RNN) models, transformer models, and / or any other type of ML model capable of performing ASR) to process the audio data stream captured from spoken utterances and generated by the microphone of the client computing device 10 to generate a corresponding ASR output stream. It is noteworthy that streaming ASR models can be used to generate the corresponding ASR output stream while generating the audio data stream.

[0044] NLU engine 112 and / or cloud-based NLU engine 122 can use one or more NLU models (e.g., Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or syntax-based rules to process the corresponding ASR output stream to generate a corresponding NLU output stream. Performance engine 113 and / or cloud-based performance engine 123 can cause the corresponding NLU output stream to be processed to generate a corresponding performance data stream. The corresponding performance data stream can correspond, for example, to a corresponding given assistant output predicted in response to spoken utterances captured in the corresponding audio data stream processed by ASR engine 111 (and / or cloud-based ASR engine 121).

[0045] A TTS engine (e.g., 114 and / or 124) can use a TTS model to process a corresponding text content stream (e.g., text defined by an LLM-based assistant 11) to generate synthesized speech audio data including computer-generated synthesized speech. The corresponding text content stream may correspond to, for example, one or more given assistant outputs, one or more given assistant outputs in modified form, and / or any other text content described herein. The aforementioned ML model may be an on-device ML model stored locally at the client computing device 10, a remote ML model executed remotely from a server computing device (e.g., at a remote server device 12), or a shared ML model accessible to both the client computing device 10 and / or the remote system (e.g., the remote server computing device 12). In an additional or alternative implementation, the corresponding synthesized speech audio data stream corresponding to one or more given assistant outputs, one or more given assistant outputs in a modified form, and / or any other text content described herein may be pre-cached in a memory or one or more databases accessible by the client computing device 10, so that the LLM-based assistant does not need to use the TTS engine 114 (or 124) to generate the corresponding synthesized speech audio data.

[0046] In various implementations, the corresponding ASR output stream may include, for example, a stream of ASR hypotheses (e.g., term hypotheses and / or transcription hypotheses) predicted to correspond to spoken utterances captured in the corresponding audio data stream, one or more corresponding prediction metrics (e.g., probability, log-likelihood, and / or other values) for each ASR hypothesis included in the ASR hypothesis stream, multiple phonemes predicted to correspond to spoken utterances captured in the corresponding audio data stream, and / or other ASR outputs. In some versions of those implementations, ASR engines 111 and / or 121 may select one or more ASR hypotheses from the ASR hypotheses as the corresponding identified text corresponding to the spoken utterances (e.g., selected based on corresponding prediction metrics).

[0047] In various implementations, the corresponding NLU output stream may include, for example, an annotated stream of identified text, which includes one or more annotations to the identified text for one or more (e.g., all) terms of the identified text, one or more corresponding prediction metrics (e.g., probability, log-likelihood, and / or other values) included in the NLU output stream, and / or other NLU outputs. For example, NLU engines 112 and / or 122 may include part-of-speech taggers (not depicted) configured to annotate terms using syntactic roles. Additionally or alternatively, NLU engines 112 and / or 122 may include entity taggers (not depicted) configured to annotate entity references in one or more segments of the identified text, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, (real and imagined) locations, etc. In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some implementations, a knowledge graph may include nodes representing known entities (and in some cases, entity attributes), and edges connecting the nodes and representing relationships between entities. Entity annotators may annotate references to entities at a high-granularity level (e.g., to enable the identification of all references to entity categories (such as people)) and / or at a lower-granularity level (e.g., to enable the identification of all references to a specific entity (such as a specific person)). Entity annotators may rely on the content of natural language input to resolve specific entities and / or may optionally communicate with the knowledge graph or other entity databases to resolve specific entities.

[0048] Additionally or alternatively, NLU engines 112 and / or 122 may include a coreference resolver (not depicted) configured to group or “cluster” references to the same entity based on one or more contextual cues. For example, the coreference resolver may be used to resolve the term “them” in the input “buythem” to “buy theater tickets” based on a mention of “theater tickets” in a client device notification rendered immediately before the natural language input “buy them” is received. In some implementations, one or more components of NLU engines 112 and / or 122 may rely on annotations from one or more other components of NLU engines 112 and / or 122. For example, in some implementations, an entity annotator may rely on annotations from the coreference resolver to annotate all references to a particular entity. Furthermore, for example, in some implementations, the coreference resolver may rely on annotations from the entity annotator to cluster references to the same entity.

[0049] Additionally or alternatively, NLU engines 112 and / or 122 may include a content generation request determination engine (e.g., Figure 1B (1121) In this context, the content generation request determination engine is configured to determine whether a verbal request (or text request) is a request for content generation based on the corresponding ASR output stream and / or the corresponding NLU output stream. Note that in some implementations, the content generation request determination engine may be separate from the NLU engine (i.e., not included in the NLU engine). In some implementations, in response to determining that a verbal request (or text request) is a request for content generation, the content generation request determination engine may determine the type of content (e.g., email, message, image, memo, summary, etc.).

[0050] In some implementations, the fulfillment engine 113 or 123 may include a content generation engine (e.g., Figure 1B In section 181), the content generation engine processes the corresponding NLU output stream to determine initial content in response to a verbal request (or text request). The determined content in response to the verbal request (or text request) can be rendered at the user interface, for example, by rendering engine 110. For example, the content generation engine can use a first generative model (e.g., the smaller LLM described above, which is more computationally efficient) to process the corresponding NLU output stream as input to generate a first generative model output (also referred to as the "smaller generative model output") from which the initial content in response to the verbal request (or text request) is determined.

[0051] Additionally or alternatively, the execution engine 113 or 123 may include a content refinement engine (e.g., Figure 1B(183) The content refinement engine processes initial content to determine refined / revised content in response to a verbal (or textual) request. For example, the content refinement engine may use a second generative model (e.g., the larger LLM described above, which is less computationally efficient) to iteratively process a text prompt that includes both the initial content and the request for a focus edit as input, to generate one or more second generative model outputs (also referred to as "one or more larger generative model outputs") from which one or more focus edits are determined. Note that the text prompts processed using the second generative model as input can be different in different iterations of content refinement. For example, a text prompt processed using the second generative model for a first iteration of content refinement ("first iteration") may include the initial content and the request for a focus edit, while a text prompt processed using the second generative model for a second iteration of content refinement ("second iteration") may include the initial content with a first sentence updated based on a first focus edit (generated based on the model output of the second generative model for the first iteration) and includes the request for a focus edit.

[0052] In some implementations, a request for focused editing can be, for example, a text request for editing one and only one sentence in the initial content during each iteration of content refinement (e.g., by a larger LLM). Alternatively, a request for focused editing can be, for example, a text request for editing one and only one paragraph in the initial content during each iteration of content refinement (e.g., by a larger LLM). However, the description of a request for focused editing is not limited to this.

[0053] In various implementations, rendering engine 110 can cause initial content to be visually rendered via the user interface of client computing device 10, wherein the initial content is rendered in response to a verbal request (for content generation). Rendering engine 110 can further cause one or more focused edits of the initial content to be automatically rendered via the user interface of client computing device 10. In some implementations, one or more focused edits can be rendered simultaneously at the user interface. For example, the initial content can be automatically edited to reflect all one or more focused edits at once, resulting in the initial content being replaced with refined content that incorporates all one or more focused edits into the initial content.

[0054] In some other implementations, one or more focused edits can be rendered sequentially at the user interface. For example, initial content can be automatically edited to render / reflect a first focused edit (from one or more focused edits) at a first time, a second focused edit (from one or more focused edits) at a second time after the first time, ..., and an Nth focused edit (from one or more edits) at the Nth time (assuming N is the total number of one or more focused edits). Optionally, the time interval between different moments of rendering two adjacent focused edits (e.g., the first and second focused edits) can be predetermined. For example, the time interval between the first and second times can be pre-configured to 1.0 second or any other suitable length.

[0055] Optionally or alternatively, in response to receiving a first focused edit (decoded from the output of the corresponding model of the second generative model), the initial content can be automatically edited to render / reflect the first focused edit, and in response to receiving a second focused edit (decoded from the output of the corresponding model of the second generative model), the second focused edit can be automatically rendered. In other words, the first time (when the first focused edit is initiated or rendered) and the second time (when the second focused edit is initiated or rendered) can be the time for completing the first iteration of content refinement and the second iteration of content refinement based on the second generative model, respectively, and additionally based on factors such as the network connection for the client computing device 10 to receive the first and second focused edits from the second generative model (in the case that the second generative model is accessed or stored in the server computing device 12).

[0056] In some implementations, a first focus edit of the initial content can be determined in response to a first larger generative model output generated by the second generative model during the first iteration based on the processing of the first text prompt (which includes both the initial content and the request for focus editing), and this first focus edit can be rendered by the rendering engine 110 at the user interface displaying the initial content. In some implementations, the first focus edit may be accompanied by a reason for the first focus edit, and the reason for the first focus edit can be rendered by the rendering engine 110. The reason for the first focus edit can be rendered simultaneously with the first focus edit and remains rendered while the first focus edit is dynamically displayed (e.g., showing the deletion of the first initial sentence and the addition of the first replacement (or revised) sentence). Details of the rendering will be provided elsewhere in this disclosure and are omitted here for clarity. In some implementations, both the reason for the first focus edit and the first focus edit can be derived from the output generated by the second generative model during the first iteration of content refinement (which corresponds to the first text prompt including both the initial content and the request for focus editing).

[0057] In response to a second larger generative model output generated during a second iteration of content refinement by the second generative model based on the processing of a second text prompt (which includes the initial content refined to incorporate the first focus edit and the request for focus edit), a second focus edit on the initial content can be determined, and this second focus edit is rendered at the user interface by rendering engine 110. In some implementations, the second focus edit may be accompanied by a reason for the second focus edit, and the reason for the second focus edit can be rendered by rendering engine 110. In some implementations, the reason for the second focus edit and the second focus edit can be determined from the output generated by the second generative model during the second iteration of content refinement (which corresponds to the second text prompt). In some implementations, the second text prompt can be automatically generated in response to the determination of the first focus edit. In other words, in response to the first focus edit determined based on the output generated by the second generative model during the second iteration of content refinement (e.g., the output of the second larger generative model), a second text prompt can be generated to include a request for focus edit and the initial content refined to incorporate the first focus edit.

[0058] The focus editing determination and rendering process can be repeated N times, where N can be a predetermined positive integer or N can be determined based on the processing using a second generative model, until, for example, the second generative model determines that no focus editing exists.

[0059] In other words, in response to the Nth larger generative model output generated by the second generative model during the Nth iteration of content refinement based on the processing of text prompts (which include both the initial content refined using the first to (N-1)th focus edits and / or requests for focus edits), the Nth focus edit on the initial content can be determined, and this Nth focus edit is rendered at the user interface by the rendering engine 110. In some implementations, the Nth focus edit may be accompanied by a reason for the Nth focus edit, and the reason for the focus edit may be rendered at the user interface by the rendering engine 110. In some implementations, optionally, a prompt may (but not necessarily) be additionally rendered by the rendering engine 110 to notify the user that content refinement of the initial content has been completed.

[0060] In some implementations, instead of rendering all or all of the one or more focused edits, a subset of one or more focused edits can be selectively rendered. For example, the output of the second generative model that determines / generates the corresponding focused edit may include the confidence score of the corresponding focused edit, and when the confidence score of the corresponding focused edit does not meet the recommended rendering threshold score (e.g., 0.7 in 1.0), the rendering engine 110 may determine not to render the corresponding focused edit at the user interface, causing all other focused edits with confidence scores that meet the recommended rendering threshold score to be rendered at the user interface.

[0061] In some implementations, optionally, the output of the first generative model, which determines / generates the initial content, may further include inferences indicating / identifying one or more specific sentences in the initial content to be refined. Such inferences indicating one or more specific sentences in the initial content to be refined may also be included in text prompts transmitted to the second generative model, such that the second generative model can be used to provide one or more LLM-based edits for editing one or more specific sentences respectively.

[0062] In some implementations, instead of performing an Nth iteration of content refinement, the second generative model can be configured to perform a single iteration of content refinement, wherein one or more model outputs can be generated by the second generative model during that single iteration (e.g., all at once). One or more model outputs can each be decoded or derived to determine a corresponding focus edit. For example, the second generative model can be used to process the aforementioned text prompt as input to generate multiple model outputs (e.g., model output O1, model output O2, and model output O3) all at once, wherein a focus edit can be derived from model output O1, an additional focus edit from model output O2, and a further focus edit from model output O3. The focus edits, additional focus edits, and further focus edits can be rendered to the user visually (e.g., simultaneously, or in the order they are decoded and received by the computing device, which renders focus edits on the initial content). Optionally, only one or more model outputs among the multiple model outputs that have a confidence score satisfying the aforementioned rendering recommendation threshold score are selectively rendered to the user. For example, if only the model output O2 includes a confidence score (e.g., 0.8) higher than the rendering recommendation threshold score (e.g., 0.7), then only the additional focus edit is transmitted from the server device hosting the second generative model to the computing device to render the focus edit and is rendered to the user via the display of the computing device, while the focus edit and further focus edits are not transmitted and are not rendered to the user.

[0063] In some implementations, training instances for training the first generative model can be generated based on user requests for content generation and on initial content incorporating one or more focused edits (determined using a second generative model). For example, training instances may include user requests for content generation as input to the training instances and initial content incorporating all one or more focused edits (which may be referred to as "final refined content") as the ground truth output. The training instances can be applied to train the first generative model by: processing the user requests for content generation as input using the first generative model to generate the training model output; comparing the training model output with the ground truth output to determine the difference; and adjusting one or more parameters / weights of the first generative model based on the difference between the training model output and the ground truth output. Note that different user requests for content generation can be collected / stored, and correspondingly, different focused edits corresponding to the different user requests for content generation can be collected / stored. In this way, different training instances can be generated as described above. These different training instances can be applied to train the first generative model or to fine-tune the first generative model in a specific domain (e.g., when writing a report, or when writing a summary, etc.).

[0064] In some implementations, optionally, in addition to the user request for content generation and the initial content, the text prompts to be provided to / transmitted to the second generative model to initiate the iterative process of focused editing generation may further include word count limits (e.g., 200 words, 500 words, etc.) so that the refined content can be relatively concise or can meet the word count limit requirements.

[0065] In some implementations, the aforementioned user interface (e.g., rendering the initial or refined content visually at its location) Figure 1B150 in the document may include one or more user controls (e.g., selectable graphical elements that enable user interaction with the initial content and / or refined content). The one or more user controls may enable the user to provide one or more user edits to the initial content and / or refined content (i.e., if the refined content is the final refined content, the initial content is incorporated with one or more focused edits, or if the refined content is the intermediate refined content, the initial content is incorporated as a portion of one or more focused edits). In these implementations, one or more training instances may be generated for training the first generative model to include, for example, (1) a user request for content generation as input to the training instance and (2) initial content incorporated with one or more focused edits (determined using the second generative model) and incorporated with one or more user edits as the ground truth output. Training of the first generative model using these one or more training instances may be similar to the descriptions provided in other parts of this disclosure, and repeated descriptions are omitted here.

[0066] As a working example, refer to Figure 1B A user (e.g., user R) can provide a verbal request 14 to client computing device 1 (e.g., “draft an email to Victor to tell him about my vacation”, or “draft a document about the summit in Zurich and share it with Victor”, etc.). The verbal request 14 can be processed by ASR engine 111 (and / or 121) for example using one or more streaming ASR models to generate a corresponding ASR output stream. Based on the corresponding ASR output stream, the verbal request (e.g., “draft an email to Victor to tell him about my vacation”) is converted into a text request 16 (e.g., “draft an email to Victor to tell him about my vacation” in natural language). In some implementations, the text request 16 (e.g., “draft an email to Victor to tell him about my vacation” in natural language) can be processed by the NLU engine 112 (and / or 122) using one or more NLU models to generate a corresponding NLU output stream.

[0067] In the above working example, the content generation request determination engine 1121 can determine, based on the corresponding ASR output stream and / or the corresponding NLU output stream, that the verbal request 14 (“draft an email to Victor to tell him about my vacation”) is a request for content generation. In response to determining that the verbal request 14 (“draft an email to Victor to tell him about my vacation”) is a request for content generation, the rendering engine 110 can cause the content generation user interface (UI) 150 to be rendered at the client computing device 10. Optionally, in response to determining that the verbal request 14 (“draft an email to Victor to tell him about my vacation”) is a request for content generation, the type of content (e.g., email) is determined, and the rendering engine 110 can render the content generation UI based on the determined type of content to be generated. In other words, different content generation UIs can be pre-configured and rendered for different types of content to be generated.

[0068] As a non-limiting example, when it is determined that the content to be generated is of email type, a first content generation UI can be pre-configured and rendered, and when it is determined that the content to be generated is of report type, a second content generation UI can be pre-configured and rendered, wherein the first content generation UI and the second content generation UI may include different graphical elements. For example, the first content generation UI may include a first selectable graphical element that, when selected, causes a list of email addresses (e.g., as a drop-down menu) to be rendered at the first content generation UI for the user to select, add, or delete recipients. The second content generation UI may include a second selectable graphical element that, when selected, causes a header (or other aspect of the report) to be rendered at the second content generation UI for the user to select and / or edit.

[0069] If the content generation request determination engine 1121 determines, based on the corresponding ASR output stream and / or the corresponding NLU output stream, that the verbal request 14 is not a request for content generation, then the text request 16 may be routed to the NLU engine 112 (if the content generation request determination engine 1121 is not included in the NLU engine 112) or may be routed to other components of the NLU engine 112 (if the content generation request determination engine 1121 is included in the NLU engine 112) to determine the intent and / or associated parameters of the action (e.g., excluding content generation, such as “turn off the light”) to be determined and performed using the fulfillment engine 113 corresponding to the verbal request 14.

[0070] The content generation UI 150 can be a pre-configured user interface of an LLM-based assistant, wherein the pre-configured user interface may include a request display area with a content request input field 161 (e.g., Figure 2A (203 in the original text). Content request input field 161 may, for example, list the text request 16 provided by the user at content request input field 161 or converted from the verbal request 14 “draft an email to Victor to tell him about my vacation” (e.g., “draft an email to Victor to tell him about my vacation” in natural language). In some implementations, content request input field 161 may allow the user to edit the text request 16 before submitting it to content generation system 18 (which may include content generation engine 181) for generating initial content 19A in response to the verbal request (e.g., an initial email to Victor describing the user’s vacation arrangements as provided in the verbal request). For example, content generation system 18 may be included in LLM-based assistant 11.

[0071] In some implementations, the user does not need to submit a text request 16 to the content generation system 18 for the initial content to be generated in response to a verbal request. For example, the user can provide the verbal utterance 14 “draft an email to Victor to tell him about my vacation” to the LLM-based assistant 11 (or provide a text request 16 via keyboard, mouse, touchpad, etc.), and in response to determining that the verbal utterance 14 “draft an email to Victor to tell him about my vacation” is a request for content generation (“content generation request”) and that the verbal utterance 14 “draft an email to Victor to tell him about my vacation” is a complete request (e.g., by determining that no additional user input is received within a predetermined subsequent time period (e.g., 5 seconds) after the verbal utterance 14), a text request 16 for the initial content to be generated (e.g., 19A) converted from the verbal request 14 can be automatically submitted to the content generation engine 181.

[0072] Initial content 19A in response to a verbal request can be generated, for example, using the first generative model described above (e.g., the first / smaller LLM 190A), based on the processing of language recognition of spoken utterance 14 (e.g., “draft an email to Victor to tell him about my vacation”). This could be an initial email sent to Victor offering the user’s vacation arrangements as described verbally. In response to the generated initial content 19A, it can be automatically rendered at user interface 150. Note that by utilizing the (computationally more efficient) smaller LLM 190A to derive the initial content 19A, latency in rendering the initial content 19A can be reduced. For example, the initial content 19A can be rendered shortly after user interface 150 is rendered.

[0073] Continuing with the above working example, text prompt 17 can be generated based on initial content 19A in response to verbal request 14 and a request 108 for focused editing of the initial content. For example, text prompt 17 may include initial content 19A in response to verbal request 14 and a request 108 for focused editing of the initial content, wherein the request 108 for focused editing of the initial content may be a natural language request to edit a single sentence (or other fragment) in the initial content using a second generative model during a single iteration of content refinement. Alternatively or additionally, text prompt 17 may include one or more additional requests, which may include formatting requests (e.g., font, font size, specific sentence or conversation, etc.), word count limit requests, writing style requests (e.g., tone), etc.

[0074] The text hint 17 can be provided to the content refinement engine 183, whereby this provision causes a second generative model (e.g., a larger / second LLM 190B) to process the text hint 17 as input for one or more iterations of content refinement. In some implementations, the text hint 17 can change for each of multiple iterations (e.g., N iterations) of content refinement, and the changed text hint 17 can be processed as input using the second generative model to generate a corresponding model output from which a corresponding focused edit 17-n (which edits a single sentence in the initial content) is derived.

[0075] For example, during the first iteration of N iterations, a second generative model can be used to process a text prompt 17, including initial content and a request for focus editing, as input to generate an output from which a first focus edit 17-1 is generated. The first focus edit 17-1 can be rendered at user interface 150, for example, to replace the first sentence in the initial content 19A rendered at user interface 150. The first focus edit 17-1 (generated during the first iteration) can also be applied to update the initial content 19A in the text prompt 17, allowing a second text prompt to be generated to include the initial content 19A incorporating the first focus edit 17-1 and / or a request for focus editing. During the second iteration, a second generative model can be used to process the second text prompt to generate an output from which a second focus edit 17-2 is generated. The second focus edit 17-2 can be applied to the initial content 19A incorporating the first focus edit 17-1 (if such an edit is not restored or rejected by the user), and can also be applied to update the second text prompt to initiate a third iteration of content refinement using the second generative model. This process can be repeated until a predefined iteration is reached (e.g., the Nth iteration), or until the output of the second generative model indicates that no focused edit is generated.

[0076] In some implementations, once the focus edits 17-n (where n=1, 2, ..., N) are determined, they can be rendered immediately at the user interface, allowing the user to view the automatic rendering of a series of one or more focus edits (also known as "LLM-based edits") on the initial content.

[0077] Continuing with the above working example, in some implementations, after all focused edits (e.g., 17-1, 17-2, ..., 17-N) have been applied to the initial content 19A, refined / revised content 19C (also referred to as "refined content") can be presented to the user. In some implementations, user edits (if any) of the refined content 19C can be received from the user to generate the final content. For example, the content generation UI 150 may include a specific selectable element (e.g., a button, not depicted) that, when selected, causes either the refined content 19C (if no further user edits are received) or the final content (if further user edits are received) to be transmitted to a third-party application (e.g., an email application).

[0078] In some implementations, optionally, selection of a specific selectable element may further cause the third-party application to launch in a specific state in which the user interface of the third-party application 105 is rendered to display the transmitted revised content 19C (or, if user editing is received, the final content transmitted). In some implementations, alternatively, selection of a specific selectable element may further cause the third-party application to automatically send the transmitted revised content 19B or the transmitted final content 19C to the recipient (e.g., “Victor” identified in verbal request 14) without causing the third-party application 105 to launch in a specific state.

[0079] Figure 2A An example of a user interface that receives a user request for generated content, according to various aspects of this disclosure, is shown. Figure 2B The illustration shows a response to receiving according to various aspects of this disclosure. Figure 2A An example of a user interface that generates initial content based on user requests. Figure 2C The illustrations show various aspects of this disclosure. Figure 2B An example of a user interface that provides the initial focus for editing the first content. Figure 2D The illustrations show various aspects of this disclosure. Figure 2B An example of a user interface for the first focused editing of the initial content. Figure 2E An example of a user interface illustrating the rationale for the second focus editing, according to various aspects of this disclosure, is shown. Figure 2F An example of a user interface for second-focus editing is shown according to various aspects of this disclosure. Figure 2G An example of a user interface illustrating the rationale for third-focused editing, according to various aspects of this disclosure, is shown. Figure 2H An example of a user interface for third-focus editing is shown according to various aspects of this disclosure.

[0080] like Figure 2AAs shown, a content generation user interface (UI) 200 is provided. The content generation UI 200 may be (but does not necessarily need to be) an LLM-based assistant 201. The content generation UI 200 may include a user request section 203, which includes a content request input field 203a. The content request input field 203a may receive text input from a user, such as “Write a 275-word email in a kind tone that talks about the need for innovation in a technology company”. Alternatively, the content request input field 203a may display a text request (e.g., “Write a 275-word email in a kind tone that talks about the need for innovation in a technology company” in natural language), which is a translation of a verbal request from the user (e.g., the user utterance “Write a 275-word email in a kind tone that talks about the need for innovation in a technology company”). In various implementations, the content request input field 203a enables a user to transmit text input (or a text request derived from a verbal request from the user) for processing using a first generative model (e.g., to generate initial content in response to the text input (or the text request derived from a verbal request)). Figure 2B The text input from the user (or a text request converted from a verbal request) can be edited before 208A in the content generation UI 200. For example, the initial content 208A can be visually rendered within the preview section 200A of the content generation UI 200 for the user to review and / or edit.

[0081] In some implementations, optionally, in response to determining that text input from the user (or a text request converted from a verbal request from the user) is complete (e.g., by determining that no additional input has been received within a predetermined time period after the text input or text request was received), the text input (or text request) can be automatically (i.e., in the absence of user input that triggers such transmission) to the first generative model. In some implementations, optionally, in response to the user selecting user request submission element 203b, text input from the user (or a text request converted from a verbal request from the user) can be transmitted to the first generative model, wherein the user request submission element is selectable and, upon selection, causes the text shown in the content request input field 203a (i.e., the text input or the text request converted from a verbal request) to be transmitted to the first generative model for processing using the first generative model.

[0082] For example, the first generative model could be a trained LLM of relatively small size, such as a “smaller LLM” with a first number of parameters (e.g., approximately millions or billions). Due to the relatively small size of the first generative model (i.e., the trained LLM with the first number of parameters), the memory, processor, power, and / or other computational resources required to process text input (such as “Write a 275-word email in a kind tone that talks about the need for innovation in a technology company”) using the first generative model to generate initial content can be relatively insignificant. Furthermore, due to the relatively small size of the first generative model, the latency in generating the initial content, and therefore in rendering the initial content, can be relatively insignificant. Such insignificant latency allows the user to immediately review and read the initial content while using a second generative model of relatively large size (such as a “larger LLM” with a second number of parameters (e.g., approximately billions, tens of billions, or hundreds of billions)) to transmit and / or process the initial content for refinement.

[0083] In various implementations, text prompts can be generated to include the initial content 208A generated using the first generative model, and to include a request for one or more focused edits. For example, a request for one or more focused edits could be “review the following initial content and suggest the most important edit by replacing a sentence in the initial content,” or “review the following draft text and suggest the most important edit in the form of 'original sentence' -> 'newsentence'. Provide a short justification for your edit, prefix it with 'Justification'. Replace only one sentence.” Note that requests for one or more focused edits may or may not be rendered via the content generation UI 200. However, it should be further noted that the specific content of the request for one or more focused edits is not limited herein and can include any other appropriate content. The generated text prompts can be transmitted to a second generative model, where the transmission can cause multiple iterations of content refinement to be performed using the second generative model.

[0084] For example, referring to Figures 1C and 1D, during the first iteration of content refinement, the first sentence 211A in the initial content can be edited by replacing it with the updated first sentence 211B, while all other sentences in the initial content remain unchanged. As a non-limiting example, the first sentence 211A could be “I think this approach is important, and I believe it has the potential to revolutionize the way we do business.”, and the updated first sentence 211B could be “This approach is not only important, but I believe it has the potential to revolutionize the way we do business.” The replacement of the first sentence 211A with the updated first sentence 211B can be rendered within user interface 200 (e.g., at preview portion 200A of user interface 200). After replacing the first sentence 211A with the updated first sentence 211B, the user can be allowed (e.g., via a restore button displayed at the content generation UI 200, not depicted) to restore the replacement. In some implementations, the first reason 211C can also be determined based on the output of the second generative model during the first iteration of content refinement, and displayed within the user interface 200. For example, the first reason 211C can be rendered when the first sentence 211A is highlighted, and can remain rendered while words in the first sentence 211A are automatically removed one by one, and continues to remain rendered while words in the updated first sentence 211B are rendered one by one. As a non-restrictive example, the first reason 211C could include the following: "The original sentence is repetitive with the use of phrases such as 'I think' and 'I believe'. The suggested edit removes the repetition and makes the sentence more assertive and impactful."

[0085] Referring to Figures 1E and 1F, during the second iteration of content refinement, the second sentence 213A in the initial content (which differs from the first sentence 211A) can be edited by replacing it with the updated second sentence 213B. As a non-limiting example, the second sentence 213A could be “I think it is important to remember that not every innovation comes from a new product or service,” and the updated second sentence 213B could be “It is crucial to remember that innovation isn't always about introducing a new product or service.” The replacement of the second sentence 213A with the updated second sentence 213B can be rendered within user interface 200 (e.g., at the preview portion 200A of user interface 200). In some implementations, the second reason 213C can also be determined based on the output of the second generative model during the second iteration, and this second reason can be displayed within user interface 200. As a non-restrictive example, the first reason 213C could include the following: “The original sentence is a bit weak due to the use of 'I think'. The suggested edit removes this phrase to make the sentence more assertive. Additionally, the phrase 'not every innovation comes from' is replaced with 'isn't always about', which is more engaging and conversational.”

[0086] Referring to Figures 1G and 1H, during the third iteration of content refinement, the third sentence 215A in the initial content can be edited by replacing it with the updated third sentence 215B. As a non-limiting example, the third sentence 215A could be “I think it is important that technology companies continue to innovate and explore new ways to serve their customers,” and the updated third sentence 215B could be “It is vital that technology companies persist in innovating and exploring new ways to serve their customers.” The replacement of the third sentence 215A with the updated third sentence 215B can be rendered within the user interface 200 (e.g., at the preview portion 200A of the user interface 200). In some implementations, the third reason 215C can also be determined based on the output of the second generative model during the third iteration and displayed within the user interface 200. As a non-restrictive example, the third reason 215C could include the following: “The phrase ‘I think it is important’ is somewhat weak and subjective. The revision makes the statement more assertive and emphasizes the urgency of need for continuous innovation in technology companies.”

[0087] like Figures 2C to 2HAs shown, the first sentence 211A (replaced by the updated first sentence 211B in the first focus edit) can occur in the initial content 208A after the second sentence 213A. The second sentence 213A (replaced by the updated second sentence 213B in the second focus edit) can occur in the initial content 208A before the third sentence 215A (replaced by the updated third sentence 215B in the third focus edit) and before the first sentence 211A. This is likely because the first focus edit has a higher importance / priority level than the second focus edit, and the second focus edit has a higher importance / priority level than the third focus edit. In some implementations, the content generation UI 200 may include a refinement pause element (not shown), which is selectable and, when selected, causes a pause in refinement performed using the second generative model.

[0088] Note that, although Figures 2C to 2H Three iterations of content refinement are shown, but a higher or lower number of iterations may exist. In some implementations, the total number of iterations can be predetermined (e.g., by the user via an iteration number selection button displayed in the content generation UI 200, or via a default iteration number specified in a text prompt or provided to the second generative model). In some implementations, the total number of iterations may not be predetermined, and iterations will stop in response to the second generative model's output for a particular iteration indicating that no focused edit has been generated or that no focused edit is necessary.

[0089] Note further that different text prompts can be processed (e.g., using a second generative model) during different iterations of content refinement. For example, a text prompt processed during the first iteration may include initial content 208A and a request for focus editing; a text prompt processed during the second iteration may include initial content 208A incorporating the first focus edit (generated during the first iteration) and a request for focus editing; and a text prompt processed during the third iteration may include initial content 208A incorporating both the first focus edit (generated during the first iteration) and the second focus edit (generated during the second iteration) and / or a request for focus editing. In other words, a text prompt processed using the second generative model in the Nth iteration may include: (i) initial content 208A incorporating the first, second, ..., and (N-1)th focus edits, and / or (ii) a request for focus editing. Additionally, text prompts may include word count limits to control the length of the refined content, or may include other user-specified requests or restrictions (e.g., user-specified formatting requests, etc.).

[0090] Now go to Figure 3The diagram depicts a flowchart illustrating an example method for generating content in response to a user request for generated content, according to various aspects of this disclosure. For convenience, the operation of method 300 is described with reference to a system performing the operation. This system of method 300 includes a computing device (…). For example The system may include one or more processors, memories, and / or other components of the client computing device 10 (Figure 1), one or more servers, and / or other computing devices. For example, the system may include the aforementioned LLM-based assistants (11 and / or 120). Furthermore, although the operations of method 300 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0091] At box 301, the system receives a user query for content generation via a client device. This user query (which may be simply referred to as a "user query") can be based on user interface input at the client device, such as typing or voice input. User queries can be, for example, voice queries, typing queries, image-based queries, etc. For example (Including voice input and image) multimodal queries.

[0092] In some implementations, when a user query includes content that is not in text format, the system can convert the user query into text format or another format. For example, if the query is a voice query (e.g., directed to an automated assistant such as the LLM-based Assistant 11 mentioned above), the system can perform automatic speech recognition (ASR) to convert the user query into text format. As another example, a user query can be a typed query (e.g., it can be entered into a text field). Figure 2A The content request input field 203a receives a typed query, such as "draft an email to Victor to tell him about my vacation". Content request input field 203a may, but does not necessarily, be associated with an LLM-based assistant.

[0093] As another example, suppose the query is a multimodal query that includes an image of an avocado and the voice input "is this healthy?". In this example, the system can perform ASR to convert the voice input into text, perform image processing on the image to identify the presence of an avocado, and perform coreference resolution to replace "this" with "an avocado", thus producing the text-formatted query "is an avocado healthy?".

[0094] At box 303, the system performs one or more actions in response to receiving a user query for content generation, wherein performing one or more actions includes: processing the user query using a first generative model (e.g., a first LLM) to generate a first LLM output (303A); and causing the initial content (e.g., an email or other text) generated based on the first LLM output to be rendered via a user interface of a client device (303B).

[0095] At box 305, the system generates a text prompt based on the user query for content generation and the first LLM output, wherein the text prompt further includes a request for one or more focused edits (which may also be referred to as "request for focused edit(s)", "request for focused edits", "request for focused edit", etc.).

[0096] In some implementations, in addition to requests for one or more focused edits, the text prompt may also include a user query for content generation, as well as initial content generated based on the first LLM output. Requests for one or more focused edits may be natural language requests to replace segments (e.g., text fragments or other parts) in the initial segment. For example, requests for one or more focused edits may be natural language requests to replace or modify a single sentence in the initial content. Alternatively, requests for one or more focused edits may be natural language requests to replace a single paragraph in the initial content. Alternatively, requests for one or more focused edits may be natural language requests to replace a specific or target segment of the initial content (e.g., “analysis session”). Alternatively, requests for one or more focused edits may be natural language requests to perform one or more iterations of content refinement and to determine or generate focused edits for individual text fragments (sentences, paragraphs, etc.) in each iteration. Alternatively, requests for one or more focused edits may be natural language requests to perform a single iteration of content refinement and to determine all one or more focused edits at once during that single iteration. However, the description of the content and format of requests for one or more focused edits is not limited to this and may be done in any applicable manner.

[0097] In some implementations, the text prompt further includes a sentence-specific refinement request that refines a specific initial sentence within the initial content. This sentence-specific refinement request can be generated based on the first LLM output indicating the specific sentence to be refined.

[0098] In some implementations, generating text prompts based on user queries and initial content may include: filtering the initial content to redact privacy information from the initial content, thereby generating a redacted version of the initial content; and generating text prompts to include the redacted version of the user query and the initial content.

[0099] At box 307, the system provides text prompts to a second LLM that is computationally less efficient than the first LLM. In some implementations, the first LLM may be a smaller LLM with fewer than 100 billion parameters, while the second LLM may be a larger LLM with more than 200 billion parameters. In some implementations, the first LLM may be a smaller LLM that includes 20%, 30%, 40%, 50%, or other percentage fewer parameters than the larger LLM. In some implementations, both the first and second LLMs may communicate with the aforementioned LLM-based assistant. For example, the LLM-based assistant may access the first LLM to generate content in response to a user query for content generation; and the LLM-based assistant may access the second LLM to refine the content (e.g., by generating and forwarding text prompts that include the user query for content generation and initial content generated by the first LLM in response to the user query for content generation).

[0100] In some implementations, a smaller LLM can be a distilled, quantized, and / or pruned version of a larger LLM. In other implementations, a smaller LLM is not a quantized and / or pruned version of a larger LLM, but is completely independent of the larger LLM. For example, a smaller LLM may have a different architecture relative to a larger LLM, and / or may be trained on a unique training dataset relative to a larger LLM. For example, the input dimension of a smaller LLM may be smaller than that of a larger LLM, the output dimension of a smaller LLM may be smaller than that of a larger LLM, and / or a smaller LLM may include various intermediate layers of different sizes and / or types relative to those of a larger LLM.

[0101] Smaller LLMs can be computationally more efficient than larger LLMs. For example, processing requests using a smaller LLM can occur with less latency compared to using a larger LLM. As another example, processing requests using a smaller LLM can utilize less memory, processor, and / or power resources compared to using a larger LLM. In some implementations, the smaller LLM can reside on a device at the client device, while the larger LLM can be located remotely at the client device. For example, the larger LLM can reside on a server device communicating with the client device.

[0102] In some implementations, the first LLM can be local on the client device, and the second LLM can be remote on the client device. In other implementations, both the first and second LLMs can be local on the client device. In still other implementations, both the first and second LLMs can be remote on the client device.

[0103] Due to the relatively small size of the first LLM, the memory, processor, power, and / or other computing resources required to process user queries for content generation (in order to generate initial content) using the first LLM can be relatively insignificant. Furthermore, due to the relatively small size of the first LLM, the latency in generating the initial content, and therefore in rendering the initial content, can be relatively insignificant. This insignificant latency allows users to review, read, and revise the initial content while using a second LLM with a relatively large size (e.g., with approximately billions, tens of billions, or hundreds of billions of parameters) to transmit and / or process the initial content for refinement. By utilizing the second LLM, the implementation can also ensure or improve the accuracy of the content provided in response to user queries for content generation.

[0104] At box 309, the system receives one or more focus edits in response to providing a text prompt to replace one or more initial sentences in the initial content with one or more updated sentences, wherein the one or more focus edits may be generated based on processing the text prompt using a second LLM for one or more iterations. At box 311, the system causes the one or more focus edits on the initial content to be visually rendered via the user interface, thereby producing revised content in response to a user query for content generation.

[0105] In some implementations, the system can cause one or more focused edits to be visually rendered via a user interface on an initial version of the document by: causing a first focused edit to be visually rendered on the initial content via a user interface, and causing a second focused edit to be visually rendered via a user interface. The first focused edit may be generated during a first iteration in one or more iterations, and may replace a first initial sentence in the initial content with a first updated sentence. The second focused edit may be generated during a second iteration in one or more iterations, and may replace a second initial sentence in the initial content with a second updated sentence. The second initial sentence is different from the first initial sentence. For example, the second initial sentence may occur earlier in the initial content than the first initial sentence. In other words, the one or more focused edits to the initial content do not occur based on the order of sentences in the initial content. Instead, the order in which the one or more focused edits to the initial content occur may be based on the importance level of sentences in the initial content.

[0106] In some implementations, while one or more focus edits (i.e., focus edits determined using a second LLM, which may be referred to as "LLM-based edits") are being applied to initial content, the system may receive a first user edit from a user (e.g., a human user) via a user interface. In these implementations, while one or more focus edits are being applied to initial content, the system may cause the first user edit to be applied to the initial document.

[0107] In some implementations, the system can receive a second user edit of the revised content via a user interface (i.e., after all or one focused edit has been applied to the initial content). In these implementations, the system can cause the second user edit to be visually applied to the revised content.

[0108] In some implementations, the system may further generate training instances for the first LLM. The training instances may include user queries about content generation as input to the training instances, and revised content as ground truth output. In some implementations, the system may further use the generated training instances to train the first LLM. For example, the system may train the first LLM by: using the first LLM to process user queries about content generation as input to generate the output of the first LLM; comparing the output of the first LLM with the ground truth output; and updating one or more weights of the first LLM based on the comparison between the output of the first LLM and the ground truth output.

[0109] Now go to Figure 4The diagram depicts a flowchart illustrating another example method for generating content in response to a user request for generated content, according to various aspects of this disclosure. The system of method 400 includes a computing device ( For example This refers to one or more processors, memories, and / or other components of the client computing device 10 (Figure 1), one or more servers, and / or other computing devices. Furthermore, although the operations of method 400 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0110] At box 401, the system receives a text prompt generated based on initial content generated in response to a user query for content generation, wherein the text prompt may further include a request for focus editing. The user query may be audio input or typed input (or other applicable types of input) received via a client device. The initial content may be visually rendered as a preview of content generation via the user interface of the client device for review by a user of the client device (e.g., a human user providing the user query). For example, a request for focus editing may be replacing a single sentence (or other text fragment, such as a paragraph, in the initial content with an updated sentence in each iteration of content refinement), or it may be replacing multiple sentences using a single iteration of content refinement. Requests for focus editing are not limited herein. For example, a request for focus editing may also be a text request that replaces or modifies multiple sentences in each of one or more iterations of content refinement.

[0111] In some implementations, the text prompt may include initial content and a request for focus editing. In other implementations, the initial content may be filtered to exclude or edit privacy information, resulting in filtered initial content. In these implementations, due to privacy concerns, the text prompt may include filtered initial content (instead of the initial content) and a request for focus editing before being received (e.g., at a server device).

[0112] At box 403, the system performs one or more iterations of content refinement based on text prompts and using an LLM to determine one or more focused edits to the initial content. In some implementations, the initial content in response to a user query for content generation is generated based on an additional LLM that processes the user query for content generation using fewer parameters than the original LLM.

[0113] In some implementations, the LLM can be accessible via a server device, and the LLM can be used during the first iteration of content refinement to process text prompts to generate a first output from which the second generative model generates the first focus edit. In these implementations, prior to the second iteration of content refinement, the text prompts can be modified to include or reflect the first focus edit. For example, in addition to including a request for focus edits, the text prompts can be modified to include initial content with a first sentence replaced or revised based on the first focus edit. During the second iteration of content refinement, the second generative model can process such modified text prompts as input to generate a second output from which the second generative model determines / generates the second focus edit. This content refinement process can be repeated until a predetermined number of content refinement iterations are reached, or until the output of the second generative model indicates that no focus edits have been generated.

[0114] In some implementations, an LLM can be a larger LLM with a larger number of parameters (e.g., approximately 100 billion), and an additional LLM can be a smaller LLM with a smaller number of parameters (e.g., approximately 1 billion).

[0115] In some implementations, a smaller LLM can be a quantized and / or pruned version of a larger LLM. In other implementations, a smaller LLM is not a quantized and / or pruned version of a larger LLM, but is completely independent of the larger LLM. For example, a smaller LLM may have a different architecture relative to a larger LLM, and / or may be trained on a unique training dataset relative to a larger LLM. For example, the input dimension of a smaller LLM may be smaller than that of a larger LLM, the output dimension of a smaller LLM may be smaller than that of a larger LLM, and / or a smaller LLM may include various intermediate layers of different sizes and / or types relative to those of a larger LLM.

[0116] Smaller LLMs can be computationally more efficient than larger LLMs. For example, processing requests using a smaller LLM can occur with less latency compared to using a larger LLM. As another example, processing requests using a smaller LLM can utilize less memory, processor, and / or power resources compared to using a larger LLM. In some implementations, the smaller LLM can reside on a device at the client device, while the larger LLM can be located remotely at the client device. For example, the larger LLM can reside on a server device communicating with the client device.

[0117] At box 405, the system provides one or more focus edits or a subset of one or more focus edits to the initial content, wherein the provision causes the one or more focus edits or the subset of focus edits to be rendered at the user interface of the client device.

[0118] In some implementations, the system provides one or more focused edits or a subset of one or more focused edits on the initial content by: providing a first focused edit on the initial content upon completion of the first iteration of content refinement and before completion of the second iteration of content refinement, and providing a second focused edit different from the first focused edit upon completion of the second iteration of content refinement.

[0119] Note that during each iteration of one or more iterations of content refinement, the output of the second generative model may include confidence scores for recommending corresponding focused edits to be rendered. Instead of providing all one or more focused edits, a subset of these edits may be provided for rendering based on the confidence scores of each focused edit from that subset corresponding to a confidence score that meets the rendering recommendation threshold score, and based on the confidence scores of other focused edits not from that subset corresponding to confidence scores that do not meet the rendering recommendation threshold score.

[0120] Now go to Figure 5 The diagram depicts a block diagram of an example computing device 510 that can be optionally used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based LLM-based assistant component, and / or other components may include one or more components of the example computing device 510.

[0121] Computing device 510 typically includes at least one processor 514 that communicates with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow users to interact with computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0122] User interface input device 522 may include a keyboard, pointing devices (such as a mouse, trackball, touchpad or graphics tablet, scanner, touchscreen integrated into the display), audio input devices (such as a voice recognition system, microphone), and / or other types of input devices. Generally, the term "input device" is intended to include all possible types of means and methods for inputting information into computing device 510 or into a communication network.

[0123] User interface output device 520 may include a display subsystem, printer, fax machine, or non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is intended to include all possible types of means and methods for outputting information from computing device 510 to a user or another machine or computing device.

[0124] Storage subsystem 524 stores the functional programming and data constructs of some or all of the modules described herein. For example, storage subsystem 524 may include selected aspects for performing the methods disclosed herein and logic for implementing the various components depicted in Figure 1.

[0125] These software modules are typically executed by processor 514 alone or in conjunction with other processors. The memory 525 used in storage subsystem 524 may include multiple memories, including main random access memory (RAM) 530 for storing instructions and data during program execution and read-only memory (ROM) 532 for storing fixed instructions. File storage subsystem 526 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing certain functionalities may be stored in file storage subsystem 526 within storage subsystem 524, or in other machines accessible to processor 514.

[0126] Bus subsystem 512 provides mechanisms for enabling various components and subsystems of computing device 510 to communicate with each other as intended. Although bus subsystem 512 is schematically shown as a single bus, alternative implementations of bus subsystem 512 may use multiple buses.

[0127] The computing device 510 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly changing nature of computers and networks, therefore... Figure 5 The description of the computing device 510 is intended only as a specific example for illustrating some implementation methods. Many other configurations of the computing device 510 are possible, and these configurations are related to… Figure 5 The computing device depicted has more or fewer components compared to the one described.

[0128] In situations where the system described herein collects or otherwise monitors personal information about a user, or where personal information and / or the monitored information can be used, the user may be provided with the following opportunity: to control whether a program or feature collects user information ( For example This includes information about a user's social networks, social actions or activities, occupation, user preferences, or current geographic location; or control over whether and / or how content might be more relevant to the user received from content servers. Furthermore, some data may be processed in one or more ways before it is stored or used, resulting in the removal of personally identifiable information. For example, a user's identity may be processed to the point that their personally identifiable information cannot be determined, or, where geographic location information is available, the user's geographic location may be generalized (e.g., to the city, zip code, or state level) to make it impossible to determine the user's specific geographic location. Therefore, users have control over how information about them is collected and / or used.

[0129] Some other implementations disclosed in this paper recognize that training generative models may require a large amount of ( For example The LLM requires millions of training instances. Due to the large number of training instances, many will lack the input and / or output properties expected when deploying the generative model for use. For example, some LLM training instance outputs may be undesirably grammatically incorrect, undesirably overly concise, undesirably overly robust, etc. Furthermore, for example, some LLM training instance inputs may lack expected contextual data, such as user attributes associated with the input, conversation history associated with the input, etc. Because of the many missing expected input and / or output properties in LLM training instances, LLM will generate many output instances that similarly lack expected output properties after training and upon deployment.

[0130] Additionally, some implementations include one or more processors of one or more computing devices. For example The processor includes a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU), wherein one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause the execution of any of the methods described above. Some implementations also include one or more transient or non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the methods described above. Some implementations also include a computer program product comprising instructions executable by one or more processors to perform any of the methods described above.

Claims

1. A method implemented by one or more processors, the method comprising: The client device receives user queries for content generation. In response to receiving the user query for content generation, The user query is processed using a first generative model to generate the first generative model output. This causes the user interface of the client device to render the initial content based on the output of the first generative model; A text prompt is generated based on the output of the first generative model, the text prompt further including a request for one or more focused edits; The text prompt is provided to a second generative model, which is less computationally efficient than the first generative model. In response to providing the text prompt, one or more focused edits are received to replace one or more initial fragments in the initial content with one or more updated fragments. The one or more focused edits are generated by performing one or more iterations of content refinement based on the text prompts using the second generative model; as well as This causes the one or more focused edits to the initial content to be rendered visually via the user interface, thereby producing revised content in response to the user query that generated the content.

2. The method according to claim 1, wherein, Each of the one or more initial segments corresponds to a sentence, and wherein causing the one or more focused edits to visually render the initial version of the document via the user interface includes: This causes a first focused edit of the initial content to be visually rendered via the user interface, wherein the first focused edit is generated during a first iteration of content refinement and replaces a first initial sentence in the initial content with a first updated sentence, and This causes a second focus edit to be visually rendered via the user interface, wherein the second focus edit is generated during a second iteration of content refinement and replaces a second initial sentence in the initial content with a second updated sentence, which is different from the first initial sentence.

3. The method according to claim 2, wherein, During the first iteration, the second generative model is used to process the text prompt as input to generate the first focused edit, and during the second iteration, the second generative model is used to process a second text prompt, different from the text prompt, as input to generate the second focused edit.

4. The method according to claim 3, wherein, The second text prompt corresponds to the text prompt that incorporates the first focused edit of the initial content.

5. The method according to any one of the preceding claims, further comprising: While one or more focus edits are being applied to the initial content, a first user edit is received via the user interface; as well as While one or more focus edits are being applied to the initial content, the first user edit is applied to the initial document.

6. The method according to any one of the preceding claims, further comprising: Receive a second user edit of the revised content via the user interface; as well as This causes the second user to edit the revised content.

7. The method according to any one of the preceding claims, wherein, The request for one or more focused edits is a request to replace a single sentence or paragraph in the initial content during each of the one or more iterations.

8. The method according to any one of the preceding claims, wherein, The text prompt includes the initial content generated based on the output of the first generative model.

9. The method according to any one of the preceding claims, wherein, The text prompts include specific sentence refinement requests that refine specific initial sentences within the initial content.

10. The method according to claim 9, wherein, The specific sentence refinement request is generated based on the output indication of the first generative model to refine the specific sentence.

11. The method according to any one of the preceding claims, wherein, Generating the text prompt based on the user query and the initial content includes: The initial content is filtered to edit out privacy information from it, thereby generating an edited version of the initial content; and Generate the text prompt to include the edited version of the initial content.

12. The method according to any one of the preceding claims, further comprising: Generate training instances for the first generative model, wherein the training instances include: The user query generated from the content, as input to the training instance, and The revised content is output as the true value.

13. The method of claim 12, further comprising: The first generative model is trained using the generated training instances, including: The first generative model is used to process the user query, which is the input to the content generation, to produce the output of the first generative model. The output of the first generative model is compared with the true value output, and One or more weights of the first generative model are updated by comparing the output of the first generative model with the true value output.

14. A method implemented by one or more processors, the method comprising: Receives a text prompt generated based on a user query generated for content and initial content in response to the user query generated for content, the text prompt further including a request for one or more focused edits. The user query is received via a client device, and The initial content is rendered visually via the user interface of the client device. Based on the text prompts and using a generative model to perform one or more iterations of content refinement, determine one or more focused edits to the initial content; and Provide one or more focus edits or a subset of the one or more focus edits, the provision causing the one or more focus edits or the subset to be rendered at the user interface of the client device.

15. The method according to claim 14, wherein, The initial content in response to the user query for content generation is generated based on the user query for content generation using an additional generative model that has fewer parameters than the generative model.

16. The method according to claim 14 or claim 15, wherein, The request for one or more focused edits is to replace a single sentence in the initial content with the updated sentence.

17. The method according to any one of claims 14 to 16, wherein, Providing one or more focus edits or the subset thereof includes: Upon completion of the first iteration of content refinement and before completion of the second iteration of content refinement, a first focused edit of the initial content is provided, and When the second iteration of content refinement is completed, a second focus edit, different from the first focus edit, is provided.

18. The method according to any one of claims 14 to 17, wherein, In addition to the requests for one or more focused edits, the text prompt also includes the initial content generated based on the output of the first generative model corresponding to the user query for content generation.

19. The method according to any one of claims 14 to 18, wherein, The initial content is modified to exclude privacy information, and the text prompt includes the modified initial content and the request for one or more focus edits.

20. A method implemented by one or more processors, the method comprising: The client device receives verbal user queries regarding content generation. In response to receiving the verbal user query regarding content generation Process the spoken user query to generate a natural language speech recognition of the spoken user query. The speech recognition of the spoken user query is processed using a first generative model to generate the output of the first generative model, and This causes the user interface of the client device to render the initial content based on the output of the first generative model; Generate text prompts, the text prompts including the speech recognition of the user query for content generation, the initial content generated based on the output of the first generative model, and requests for one or more focused edits; The text prompt is provided to a second generative model, which is less computationally efficient than the first generative model. In response to providing the text prompt, one or more focused edits are received to replace one or more initial fragments in the initial content with one or more updated fragments, said one or more focused edits being generated based on one or more iterations of content refinement using the second generative model to process the text prompt, and This causes the one or more focused edits to the initial content to be rendered visually via the user interface, thereby producing revised content in response to the verbal user query for content generation.

21. A system comprising one or more processors operable to execute instructions configured to cause execution of any one of claims 1 to 20.

22. A computer-readable storage medium storing one or more computer instructions, said computer instructions being executable by one or more processors to perform the method according to any one of claims 1 to 20.

Images