Modifying subparts of large language model output

Abstract

Implementations are described herein for using an LLM to modify less than all of a rendered LLM output. In various implementations, a first LLM response is used by a client application to provide a first rendered LLM output. The client application can provide (i) an indication of a sub-portion of the first rendered LLM output selected by a user, and (ii) a request to modify the selected sub-portion. A sub-portion of the first LLM response corresponding to the selected sub-portion of the first rendered LLM output can be used to assemble a second LLM prompt, which can be processed using one or more LLMs to generate a second LLM response. The second LLM response can be operable to provide a second rendered LLM output that includes at least a portion of the first rendered LLM output outside of the selected sub-portion and the modified selected sub-portion.

Description

[0001] Large Language Models (LLMs) can be used to process sequences of input words as input to generate sequences of output words as an LLM response. These sequences of input / output words are typically in the form of text strings, although they can also take other forms, such as embeddings, numbers, etc. Some LLM responses can include a relatively large amount of detail and / or a large amount of natural language. For example, a user might request to use an LLM to generate multi-segment summaries of a document, detailed invitations to a birthday party, business emails, etc. Summary of the Invention

[0002] If a user is not satisfied with the rendered LLM output, they can copy it to a text editor and then manually edit the text. Alternatively, the user can issue a subsequent natural language request to modify the entire LLM output, such as adding more details, changing the "tone" of the LLM output, replacing words with other lexical units (e.g., synonyms), etc. However, it is possible that the user is satisfied with some parts of the LLM output but not others.

[0003] This document describes an implementation for modifying selected sub-sections—i.e., less than the entirety of the rendered LLM output—using an LLM. More specifically, but not exclusively, this document describes an implementation for determining which sub-sections of the rendered LLM output have been selected by the user and modifying those selected sub-sections based on user requests to generate modified versions of those selected sub-sections of the rendered LLM output. Users can select sub-sections of the rendered LLM output in various ways, such as using pointer devices, touchscreens, and / or keyboards to highlight content (text and / or images), verbally indicating specific sections (e.g., "shorten the second paragraph," "update the map to give driving directions instead of subway directions"), etc. In some implementations, the rendered LLM output may also be provided in editable text fields or other similar interfaces. This allows users to directly edit the rendered LLM output, instead of requiring them to copy and paste the rendered LLM output into a text editor, word processor, or other application that allows them to edit content.

[0004] The techniques described in this paper offer various advantages. Users who wish to modify a portion (e.g., a sentence, paragraph) of an LLM output (e.g., a business email, invitation, advertising copy, etc.) but want to keep another part of the LLM output unaffected no longer need to copy the entire LLM output to a separate content editing application. Instead, users can provide a specific sub-section of the LLM output they wish to edit, along with their modification request, as input to the LLM (or another LLM). Subsequent LLM prompts that include the user's modification request and the selected sub-section of the LLM output will be shorter than those that include the entire LLM output and the user's modification request. Consequently, significantly fewer computational resources (e.g., processor cycles, memory) can be consumed, especially considering that LLMs typically have hundreds of billions of parameters, meaning that longer input sequences would require much longer processing times.

[0005] The techniques described in this paper also enable users to modify one modality of the LLM output while leaving another modality unchanged. Suppose a user requests an LLM to generate a multimodal document comprising both text and images. If the user only wants to modify (e.g., replace, change) the images but not the text, the user can select the images and issue commands to modify those selected images accordingly. This can trigger the application of a text-to-image generative model (e.g., a diffusion model or similar model) to the user's request and the selected images, rather than re-invoking the LLM that was used to generate the initial multimodal LLM output. Alternatively, the system can search an image repository for replacement images, which does not necessarily require additional LLM processing.

[0006] In some implementations, a method may be implemented by one or more processors and may include: processing a first LLM prompt using a large language model (LLM) to generate a first LLM response; providing the first LLM response to a client application, wherein the first LLM response is operable by the client application to provide a first rendered LLM output; receiving from the client application: an indication of a sub-part of the first rendered LLM output that has been selected using one or more input devices, and a request for a modified version of the selected sub-part of the first rendered LLM output; extracting a sub-part of the first LLM response corresponding to the selected sub-part of the first rendered LLM output. The process involves: assembling a selected sub-part of the first LLM response with data indicating a request to modify a selected sub-part of the first rendered LLM output as a second LLM prompt; processing the second LLM prompt using the same LLM or a different LLM to generate a second LLM response; and providing the second LLM response to the client application, wherein the second LLM response is operable by the client application to provide a second rendered LLM output, the second rendered LLM output including at least a portion of the first rendered LLM output other than the selected sub-part of the first rendered LLM output and a modified version of the selected sub-part of the first rendered LLM output.

[0007] In various implementations, the first LLM response may include a raw text string containing "metadata instructions" for formatting the first rendered LLM output at the client. In various implementations, selection may include receiving start and end character positions from the client application, these positions identifying segments of the raw text string outside the metadata instructions. In various implementations, a request for a modified version of a selected sub-section of the first rendered LLM output may include a request to add one or more details to the selected sub-section of the first rendered LLM output. In various implementations, a request for a modified version of a selected sub-section of the first rendered LLM output may include a request to modify or replace one or more details of the selected sub-section of the first rendered LLM output.

[0008] In various implementations, a request for a modified version of a selected sub-part of the first rendered LLM output may include a request to add content supporting one or more details of the selected sub-part of the first rendered LLM output. In various implementations, the method may include: formulating a search query based on the one or more details of the selected sub-part of the first rendered LLM output; retrieving one or more documents from a search engine in response to the search query; and incorporating data from the one or more documents in response to the search query into a second LLM hint.

[0009] In various implementations, a request for a modified version of a selected sub-portion of the first rendered LLM output may include a natural language request. In various implementations, the first LLM response may include metadata instructions for rendering one or more images, and the selected sub-portion of the first rendered LLM output includes one or more rendered images. In various implementations, a request for a modified version of a selected sub-portion of the first rendered LLM output may include a request to replace one or more rendered images in the rendered images with one or more alternative images.

[0010] In various implementations, a request to replace one or more rendered images in a rendered image with one or more alternative images may include a natural language request to retrieve one or more alternative images having specified visual features.

[0011] In various implementations, a request for a modified version of a selected sub-part of the first rendered LLM output may include a natural language request for generating a modified version of one or more rendered images from the rendered images, and processing the second LLM prompt with the same or different LLM may include processing the natural language request with a visual LLM to generate a modified version of one or more rendered images from the rendered images.

[0012] Additionally, some implementations include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause the execution of any of the methods described above. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the methods described above. It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0013] Figure 1 A block diagram depicts an example environment that illustrates various aspects of this disclosure and in which some of the implementations disclosed herein can be implemented.

[0014] Figure 2 Examples of how the various components described herein can collaborate to perform selected aspects of this disclosure are illustrated schematically.

[0015] Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3E Examples of graphical user interfaces (GUIs) in use according to the various implementations described in this article are illustrated schematically.

[0016] Figure 4 A flowchart illustrating an example method of practicing a selected aspect of this disclosure is depicted.

[0017] Figure 5 Example architectures of computing devices based on various implementations are described. Detailed Implementation

[0018] This document describes an implementation for modifying selected sub-sections of the LLM output—i.e., less than the entire rendered LLM output—using LLM. More specifically, but not exclusively, this document describes an implementation for determining which sub-sections of the LLM output have been selected by the user and modifying those selected sub-sections based on user requests to generate modified versions of those selected sub-sections of the LLM output. Users can select sub-sections of the LLM output in various ways, such as using pointer devices, touchscreens, and / or keyboards to highlight content (text and / or images), verbally indicating specific sections (e.g., "shorten the second paragraph," "update the map to give driving directions instead of subway directions"), etc.

[0019] In various implementations, when a user issues a natural language query, the query can be used to assemble an LLM hint, which is then processed by the LLM to generate an LLM response. In various implementations, this LLM response can include a sequence of terms containing raw or plain text. This raw or plain text can include both responsive content and “metadata instructions” for formatting and / or otherwise rendering the responsive content. Metadata instructions can include, for example, markup language (e.g., markdown, LaTeX, XML, HTML, etc.) notation that describes how the text should be rendered (e.g., font, spacing) and / or how other content should be presented. For example, some LLMs can generate LLM output that includes metadata instructions for rendering images, such as by using Uniform Resource Locators (URLs) to identify those images.

[0020] An LLM response can be operable to enable client applications—such as automated assistants or chatbots that can access human-computer dialogue using an LLM as a backend—to render the LLM response as a visually appealing and / or informationally rich rendered LLM output. For example, the rendered LLM output may include the raw text content from the LLM response, rendered along with images or other content identified in the metadata directives also included in the LLM response, as described in the metadata directives.

[0021] As used herein, "rendered LLM output" (or simply "rendered output") generally refers to the formattable content that is ultimately presented to the user, including graphics / images / videos, etc. The user can select sub-sections of the rendered LLM output, issue requests to modify those selections, and be presented with modified rendered LLM output, where the selected sub-sections are changed based on the user's request. "LLM response" or "raw LLM response" refers to a sequence of terms directly generated using LLM. These terms may include, for example, sequences of text ultimately used to format and / or otherwise generate the rendered LLM output, metadata instructions, etc.

[0022] In various implementations, users can select a sub-section (i.e., less than all) of the rendered LLM output and manipulate only that selected sub-section by issuing subsequent requests to the LLM. For example, if the rendered LLM output contains three paragraphs, a user can select the middle paragraph and issue various types of requests to manipulate only the selected sub-section, while leaving the rest of the rendered LLM output (and the underlying raw LLM response) unchanged. In the example where the user selects fragments of text (e.g., sentences or paragraphs), these subsequent requests can include, for example, requests to make the selected text shorter, longer, more casual, more formal or professional, simpler, more detailed, with a different tone (e.g., more interesting, more serious), or rephrased (e.g., randomly, based on different bundle search results, etc.).

[0023] Other examples of how the selected content can be modified in this way include, but are not limited to, removing the selected content, adding more details or elaborating on the details contained in the selected content, replacing words with synonyms or other selected words or phrases, etc. In some implementations where the selected content states a position, theory, argument, or opinion, the user can request additional information / text to prove and / or support that position, theory, argument, or opinion. In some implementations, the user can request citations or references to support the claimed factual statements. In other implementations where the user has selected content rendered specifically using metadata directives (e.g., images), the metadata directives themselves can be extracted and used as part of subsequent LLM prompts.

[0024] The rendered LLM output typically includes additional formatting information based on the aforementioned metadata instructions contained in the underlying original LLM response. For example, HTML / XML code and / or Document Object Model (DOM) nodes may be incorporated into the LLM response (e.g., injected into the LLM response to replace parts of it) to make it operable for rendering by an HTML browser or other similarly configured user interface. When this occurs, in some cases, a mapping may be created that can later be used to identify the portion of the original LLM response corresponding to a sub-part of the final rendered LLM output selected by the user. In various implementations, when the user selects a sub-part of the rendered LLM output, this sub-part indication may be used along with this mapping to extract the corresponding portion of the underlying LLM response used to generate the rendered LLM output. As a non-limiting example, when HTML code is incorporated into the LLM response, HTML tags may be annotated with additional attributes (e.g., character offsets) that identify where the content of this HTML DOM node originates in the initial LLM response.

[0025] Indications of a sub-section of the rendered LLM output selected by the user (e.g., start and end character positions) can be used to extract a portion of the initial LLM response. This extracted portion can then be assembled into a subsequent LLM hint along with a subsequent request from the user. In some implementations, additional implicit requests or commands can also be incorporated into the subsequent LLM hint designed to trigger the selected aspects of this disclosure. For example, an additional implicit request could be a request to "modify only the provided extract of the previous LLM response according to the user's command. Leave the rest of the previous LLM response unchanged." In some implementations, implicit requests such as these, along with an LLM response having the selected sub-section and the corresponding user command, can be used to train and / or fine-tune the LLM. This subsequent LLM hint can then be processed using the same or a different LLM to generate a subsequent LLM response. The sub-section of this subsequent LLM response corresponding to the portion of the previously rendered LLM output selected by the user can be modified according to the user's subsequent request. In some implementations, the remainder of the subsequent LLM response can remain unaffected and therefore not necessarily be processed using LLM, which saves considerable resources.

[0026] Even if a user might only want to change a selected sub-section of the rendered LLM output, it's still possible that a user-requested change needs to be propagated to parts of the rendered LLM output outside the selected sub-section. Suppose the rendered LLM output includes a proposed meeting agenda. Further suppose the user selects a first date in the agenda and requests that it be replaced with a second date different from that first date. If that first date is included elsewhere in the proposed agenda outside the user-selected section, then the second date cannot be used to replace instances that might produce inconsistent meeting agendas.

[0027] Therefore, in some implementations, the LLM can be trained and / or fine-tuned to process commands, taking into account differences between facts or details contained within and outside the user's selections. For example, an explicit user request to replace a first date with a second date in a selected portion of the rendered LLM output (generated from the underlying LLM response) can trigger the generation of an implicit request to replace the first date with the second date elsewhere in the rendered LLM output, even in portions not selected by the user. This feature can be particularly useful when the LLM is used to generate complex structured languages ​​such as source code, mathematical proofs, etc. For example, a user can select a specific code snippet (e.g., a line or block) of the LLM-generated source code and request a change to the variable name contained within that selection. The same variable name can then be changed throughout the LLM-generated source code—either in the user's selections or elsewhere. In some such implementations, instances of variables to be changed found outside the user's selections can be presented to the user one at a time as a list, etc., allowing the user to review and approve (or reject) each proposed replacement individually.

[0028] As previously mentioned, an LLM response may include metadata instructions for rendering one or more images. Suppose a user selects a sub-portion of the resulting rendered LLM output, including one or more rendered images, and issues a request to replace one or more of the selected rendered images with one or more alternative images. In some implementations, the user's selection of an image can be mapped to metadata instructions for retrieving an image contained in the underlying raw LLM response. In some implementations, when the metadata instructions for retrieving an image are detected to correspond to the selected portion of the rendered LLM output, a search for alternative images can be automatically triggered, for example, using a search query based on the user's accompanying request to modify the image. For example, if a user selects an image of a tiger and requests to replace the tiger image with an image of a bear, this could trigger the formulation and / or submission of an image search query for an image of a bear. It is worth noting that only the metadata instructions need to be modified; this does not necessarily require subsequent application of the LLM.

[0029] In some cases, the request may include a request to retrieve one or more alternative images having specified visual features. For example, if the image is of the Eiffel Tower during the day, a user could request an alternative image depicting the Eiffel Tower at night. Alternatively, if the request includes a natural language request to generate a modified version of one or more of the rendered images, the natural language request can be processed using a text-to-image generative model (typically including both an LLM for converting input text into a latent representation and a generative image model) to generate a modified version of one or more of the rendered images or a completely new image.

[0030] Now go to Figure 1 The diagram depicts a block diagram of an example environment 100 illustrating various aspects of this disclosure and in which implementations disclosed herein may be carried out. Example environment 100 includes a client device 110, a natural language (NL)-based response system 120, and a search system 140. Although shown separately, in some implementations, all or all aspects of the NL-based response system 120 and all or all aspects of the search system 140 may be implemented as part of an integrated system.

[0031] In some implementations, all or all aspects of the NL-based response system 120 may be implemented locally at the client device 110. In additional or alternative implementations, all or all aspects of the NL-based response system 120 may be obtained from, for example... Figure 1The depicted client device 110 is implemented remotely (e.g., at a remote server). In those implementations, the client device 110 and the NL-based response system 120 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks (“LANs”, including Wi-Fi LANs, mesh networks, Bluetooth, near field communication, etc.) or wide area networks (“WANs”, including the Internet) .

[0032] Client device 110 may be one or more of the following, for example: desktop computer, laptop computer, tablet computer, mobile phone, vehicle computing device (e.g., in-vehicle communication system, in-vehicle entertainment system, in-vehicle navigation system), independent interactive speaker (optionally with a display), smart home appliance (such as a smart TV), and / or user's wearable device including a computing device (e.g., user's watch with computing device, user's glasses with computing device, virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0033] Client device 110 may execute one or more applications, such as application 115, through which queries can be submitted and / or (e.g., audibly and / or visually) rendered NL-based summaries and / or other responses to queries. Application 115 may be an application separate from the operating system of client device 110 (e.g., an application installed "on top of" the operating system) – or it may alternatively be implemented directly by the operating system of client device 110. For example, application 115 may be a web browser installed on top of the operating system, or it may be an application integrated as part of the functionality of the operating system. Application 115 may interact with NL-based response system 120.

[0034] In various implementations, client device 110 may include a user input engine 111 configured to detect user input provided by a user of client device 110 using one or more user interface input devices. For example, client device 110 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to the user's spoken words or other sounds in the environment of client device 110. Alternatively, client device 110 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected in the field of view of one or more visual components. Alternatively, client device 110 may be equipped with one or more touch-sensitive components (e.g., keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to touch input directed at client device 110. Some examples of queries or requests described herein may be queries or requests formulated based on user input provided by a user of client device 110 and detected via user input engine 111. For example, the query or request can be a typed query or request entered via a physical or virtual keyboard, a suggested query or request selected via a touchscreen or mouse, a spoken voice query or request detected by the microphone of the client device, or an image query or request based on an image captured by the visual component of the client device.

[0035] In various implementations, client device 110 may include rendering engine 112 configured to provide content (e.g., NL-based summaries, creative LLM outputs, chat outputs, etc.) for audible and / or visual presentation to a user of client device 110 using one or more user interface output devices. For example, client device 110 may be equipped with one or more speakers enabling the provision of audible content to a user via client device 110. Alternatively, client device 110 may be equipped with a display or projector enabling the provision of visual content to a user via client device 110. In some implementations, the display may be part of a head-mounted display (HMD).

[0036] In some implementations, rendering engine 112 can be configured to generate rendered content based on the raw LLM response. For example, the LLM response may include a sequence of terms that can be manipulated by rendering engine 112 to render audible and / or visual output. In some implementations, this sequence of terms may include a sequence of raw text. Some portions of the raw text sequence may include meaningful content responding to a user's query or request. Other portions of the text sequence may include metadata instructions (e.g., symbols) that may be used by rendering engine 112 (or by UX engine 136, described below) to render the meaningful content in a specific manner (e.g., using a selected font, line breaks, images, formatting, etc.). In some implementations, rendering engine 112 may also be configured to create a mapping between the raw LLM response and the downstream rendered content generated based on that raw LLM response. For example, when incorporating raw LLM content into an HTML DOM node, rendering engine 112 may add attributes (e.g., character offsets) to HTML tags that identify where the content to be displayed using the DOM node is located in the underlying raw LLM response.

[0037] In various implementations, client device 110 may include a context engine 113 configured to determine the context (e.g., current or recent context) of client device 110 and / or its user. In a multi-turn conversation between the user and an automated assistant (or, alternatively, a “virtual assistant,” “chatbot,” etc.), the context of the client device and / or the user can be maintained as a “user state” across multiple turns.

[0038] In some implementations, the context engine 113 may utilize current or recent interactions of the client device 110, the location of the client device 110, profile data of the client device 110's user (e.g., an active user when multiple profiles are associated with the client device 110), and / or other data accessible to the context engine 113 to determine the context and / or update the user's state. For example, the context engine 113 may determine the current context based on one or more recent queries of a search session, profile data, and / or the current location of the client device 110. For example, the context engine 113 may determine the current context "looking for a healthy lunch restaurant in Louisville, Kentucky" based on a recently issued query or request, profile data, and the location of the client device 110.

[0039] As another example, the context engine 113 can determine the current context based on which application is active in the foreground of the client device 110, the current or recent state of the active application, and / or the content currently or recently rendered by the active application. The context determined by the context engine 113 can be used, for example, to supplement or rewrite queries or requests based on user input, to generate implicit queries or requests (e.g., queries or requests independent of user input), and / or to determine the submission of implicit queries / requests and / or to render the results for implicit queries / requests. Furthermore, the user's context across multiple rounds of a search session can be used as user state to enrich, for example, the output rendered by a search chatbot companion device at each round of a multi-round human-computer dialogue session.

[0040] In various implementations, client device 110 may include a selection mapping engine 114 configured to map user-selected sub-parts of the rendered LLM output provided by rendering engine 112 to corresponding sub-parts of the original LLM response used to generate the rendered LLM output. In some implementations, selection mapping engine 114 may utilize previously mentioned HTML attributes (e.g., character offsets) to map user-selected sub-parts of the rendered LLM output provided by rendering engine 112 to corresponding sub-parts of the original LLM response.

[0041] In various implementations, selection mapping engine 114 may provide other components with data indicating the mapping—such as the start and end character indices in the original LLM response—to extract the corresponding portion of the original LLM response. In some cases, selection mapping engine 114 may provide data indicating the mapping to components of the NL-based response system 120—such as selection extraction engine 130—and selection extraction engine 130 may extract the corresponding portion of the original LLM response. In other cases, selection mapping engine 114 may directly use the mapping data to extract the corresponding portion of the original LLM response and provide this extracted portion to components of the NL-based response system 120, such as LLM input engine 126 (discussed in more detail below).

[0042] Furthermore, the client device 110, the NL-based response system 120, and / or the search system 140 may include one or more memories for storing data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components that facilitate communication via one or more networks in network 199. In some implementations, one or more software applications may be locally installed on the client device 110, while in other implementations, one or more software applications may be remotely hosted (e.g., by one or more servers) and may be accessible from the client device 110 via one or more networks in network 199.

[0043] although Figure 1 The aspects described herein relate to a single client device with a single user, but it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, one or more additional client devices of the user and / or additional users may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user may form a coordinated ecosystem of devices that can employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (e.g., via network 199). As another example, a given client device may be utilized by multiple users (e.g., user groups, families) in a shared setting.

[0044] The NL-based response system 120 is shown as including a search results document (SRD) selection engine 122, an LLM selection engine 124, an LLM input engine 126, an LLM response generation engine 128, a selection extraction engine 130, a consistency engine 132, a filtering engine 134, and a user interface (UX) engine 136. Some of these engines may be omitted and / or combined in various implementations.

[0045] SRD selection engine 122 can be configured to generate a NL-based summary response to a query using LLM and search result documents in response to the query. SRD selection engine 122 can also render the NL-based summary in response to the query.

[0046] The LLM selection engine 124 may, for example, select zero or more generative models from a plurality of candidate LLMs. For instance, in some iterations, the system will determine that none of the candidate generative models will be utilized; in some iterations, the system will determine that only one candidate generative model will be utilized; and in some iterations, the system will determine that multiple candidate generative models will be utilized. The LLM selection engine 124 may optionally utilize one or more rules and / or one or more classifiers 125, which are trained to generate outputs identifying which LLMs are best suited to generate a response to the current query or request given the current user state / context.

[0047] LLM input engine 126 can be configured to assemble LLM input hints based on data such as the current query, the current user state / context, past queries, past LLM responses (which may be included in the current user state / context), and the user-selected portions of past rendered LLM outputs for modification. In some implementations, LLM input hints may include sequences of lexical units, which may be words, phrases, or embeddings generated from data such as text, images, audio, etc.

[0048] LLM response generation engine 128 can be configured to apply one or more LLMs stored in LLM database 129 to LLM input prompts generated by LLM input engine 126 to generate LLM responses. LLM responses can take various forms, such as sequences of lexical units that correspond to, represent, or directly convey words, phrases, embeddings, etc. LLMs stored in LLM database 129 can take various forms, such as PaLM, BARD, BERT, LaMDA, Meena, GPT, and / or any other LLM, such as encoder-only, decoder-only, sequence-to-sequence based, and optionally including attention mechanisms or other memory. Visual language models (VLMs) capable of processing images and text can also be included.

[0049] Selection extraction engine 130 can be configured to extract sub-portions of the raw LLM response, which correspond to selected sub-portions of the rendered LLM output provided by rendering engine 112 at client device 110 based on the raw LLM response. As previously described, in some implementations, selection extraction engine 130 may extract these portions of the raw LLM response based on mapping data received from selection mapping engine 114. For example, selection mapping engine 114 may provide a start character position and / or an end character position, and selection extraction engine 130 may extract the sub-portion of the raw LLM response that begins at the start character position and ends at the end character position.

[0050] Consistency engine 132 can be configured to evaluate the remaining portion of the original LLM response beyond the sub-parts extracted by selection extraction engine 130, in order to maintain consistency between the selected and unselected portions of the rendered LLM output. Suppose a user selects a middle paragraph of a scheduling email and issues an NL-based request, “Please change the street number from 359 to 874”. This middle paragraph of the scheduling email (e.g., minus metadata instructions) can be extracted by selection extraction engine 130 and incorporated into a subsequent LLM input prompt by LLM input engine 126. This subsequent LLM input prompt may also include the user's request to change the street number. When the subsequent input request is processed by LLM response generation engine 128 using LLM 129, the resulting LLM response may include the previous LLM response, except that the middle paragraph has been changed to reflect the new street number. However, if the street number is also included in another part of the initially rendered LLM output that the user did not select, another instance of that street number may not be replaced upon request, resulting in scheduling emails with inconsistent street numbers. Therefore, in various implementations, the consistency engine 132 can be configured to ensure that details changed within the selected portion of the initially rendered LLM output (generated using the underlying raw LLM response) are changed elsewhere, where applicable. In some implementations, the consistency engine 132 can heuristically perform its actions, for example, by extracting entities and facts from both the user selection and the remainder of the rendered LLM output and comparing them. In other implementations, the LLM 129 itself can be trained to maintain consistent factual details across the selected and unselected portions of the LLM response.

[0051] Updating a selected sub-section of the rendered LLM output using LLM can sometimes produce unpredictable results. If a user requests more detail for a given selection (e.g., a paragraph), the user may not want the replacement generated using LLM129 to be significantly longer. Therefore, in some implementations, the filtering engine 134 can be configured to ensure that user requests to modify a selected sub-section of the rendered LLM output do not result in potentially negative consequences, excessive changes, etc. For example, in some implementations, if the user's request results in a change in the number of thresholds (e.g., altered characters or words) or too substantial a change (e.g., calculated using edit distance, word count, etc.), the filtering engine 134 can throw an error and / or issue a warning to the user.

[0052] In some implementations, UX engine 136 may be configured to provide raw LLM responses (e.g., sequences of terms mixed with metadata instructions) to client device 110, which can then be manipulated by rendering engine 112 to provide rendered LLM output. Alternatively, in some implementations, UX engine 136 may generate content that can be rendered more directly, such as HTML code including the raw LLM responses and that can be rendered by rendering engine 112 or application 115 as, for example, a webpage.

[0053] Search system 140 is shown as including an SRD engine 142 and a results engine 144. Some of these engines may be omitted or combined with each other in various implementations. For example, SRD engine 142 may utilize index 143 and / or other resources to identify search result documents in response to queries or queries as described herein. For example, SRD engine 142 may use queries or requests formulated by components of NL-based response system 120 to identify search result documents or other content that can be used to modify selected sub-sections of the rendered LLM output. For example, a user may request a search and provide evidence or other documents to support and / or refute details contained in selected sub-sections of the rendered LLM output. Results engine 144 may provide non-LLM-generated search results that may be harvested for content to be presented together with the NL-based summary described herein and / or may be used by LLM response generation engine 128 to generate modified LLM responses.

[0054] In some implementations, when a user requests to add content supporting one or more details contained therein to a selected sub-section of the rendered LLM output, one or more components of the NL-based response system 120 and / or search system 140 can formulate a search query based on one or more details of the selected sub-section of the rendered LLM output. Search system 140 can then retrieve one or more documents in response to the search query. Data from these documents in response to the search query can, for example, be incorporated by LLM input engine 126 into subsequent LLM input prompts for generating a modified version of the previously rendered LLM output.

[0055] Figure 2 schematically depicted Figure 1 Examples of how the various components depicted can collaborate to perform selected aspects of this disclosure are provided. As indicated at the top, in some implementations, the component to the left of the vertical dashed line may be part of an NL-based response system 120. The component to the right of the vertical dashed line may be part of a client device 110. In other implementations, the various components may be implemented elsewhere.

[0056] Starting from the upper right, a first request 250A can be received at user input engine 111, which then provides data indicative of the first request 250A (e.g., the request itself, its generated embeddings, etc.) to LLM input engine 126 of NL-based response system 120. The first request 250A can be typed, transcribed from spoken utterance using ASR, or even an implicit query. In either case, the data indicative of the first request 250A can be assembled by LLM input engine 126 into an LLM prompt (not depicted), which is then processed by LLM response generation engine 128 using one or more LLMs from database 129 to generate a first raw LLM response 252A. As previously described, the first raw LLM response 252A can include a sequence of terms, such as a sequence of raw text including both the content responding to the request and metadata instructions scattered therein. The first raw LLM response 252A can be provided by UX engine 136 to the rendering engine of client device 110. The rendering engine 112 can provide, for example, a display and / or speakers, and a first rendered LLM output 254A, which can include various modal outputs such as audible output, images, text, etc.

[0057] Once rendered at client device 110, the user can select a sub-part 256A of the first rendered LLM output 254A, for example, via user input engine 111. In various implementations, the selected sub-part 256A can be provided to selection mapping engine 114, which in turn provides data to selection extraction engine 130 of NL-based response system 120 indicating a mapping (e.g., start character position and end character position) between the selected sub-part 256A of the first rendered LLM output 254A and the corresponding sub-part of the first original LLM response 252A. Selection extraction engine 130 can then use this mapping to extract the corresponding selected sub-part 258 of the original LLM response 252A.

[0058] Simultaneously, a second request 250B can be received from the user at the client device, such as at the user input engine 111. The second request 250B may include one or more commands for modifying, changing, removing, etc., the selected sub-part 256A. The second request 250B and the selected sub-part 258 of the original LLM response 252A can be provided to the LLM input engine 126, for example, for assembly into another LLM input prompt. In some implementations, this additional LLM input prompt may also include the first request 250A.

[0059] The additional LLM input prompt can then be processed by the LLM response generation engine 128 using LLM 129 to generate a subsequent raw LLM response 252B. The subsequent raw LLM response 252B can then be provided by the UX engine 136 to the rendering engine 112 on the client device 110. The rendering engine 112 can then generate and provide a subsequent rendered LLM output 254B. The subsequent rendered LLM output 254B may include the unmodified portion of the initially rendered LLM output 254A that was not selected by the user, and a modified sub-part 256B that replaces the selected sub-part 256A of the initially rendered LLM output 254A. In some implementations where the consistency engine 132 is deployed, the provided portion of the initially rendered LLM output 254A may also be modified if the details contained therein are inconsistent or conflicting with the details in the modified sub-part 256B of the subsequently rendered LLM output 254B in other respects.

[0060] Figure 3A An example client device 310 in the form of a tablet computer is depicted for interacting with an LLM-based response system 120. The client device 310 includes a display 370 on which a query input field 372 is rendered. A user (not depicted) has entered (by typing or having spoken speech recognized) a request into the query input field 372: “Write an invitation to a slumber birthday party on December 16 for Delia Sue, who is turning 8.” This is based on a raw LLM response (not depicted, e.g., generated by an LLM response generation engine 128). Figure 2 252A in the middle), can be with Figure 2 The rendered LLM output 254A shares various characteristics with the rendered LLM output 354A, which is generated and rendered on the display 370, for example, by the rendering engine 112. The display 370 also renders thumbs-up and thumbs-down buttons that can be operated by the user to provide positive or negative feedback regarding the rendered LLM output 354A, and optional graphical elements 374 that the user can select to initiate selected aspects of this disclosure. Specifically, the user can operate element 374 to initiate… Figure 2 The process described in the text.

[0061] As previously mentioned, in some implementations, the rendered LLM output 354A can be provided within an editable text field or other similar interface. This allows users to directly edit the rendered LLM output 354A (and other rendered LLM outputs described herein), rather than requiring the user to copy and paste the rendered LLM output 354A into a text editor, word processor, or other application that allows users to edit content. In some such implementations, user edits to the rendered LLM output can be annotated, for example, using different fonts, colors, etc., so that the user can keep track of which parts of the rendered LLM output are initial and which parts have been edited by the user. In some implementations, edited versions of the rendered LLM output can be preserved as part of a saved thread, for example, so that they can be used to generate downstream input suggestions. In some implementations, edited portions of the rendered LLM response can then be selected by the user as sub-parts and processed using LLM to generate a modified LLM response. For example, a user can manually edit the rendered LLM output to change details (e.g., date, address, etc.), and then select a sub-section of the rendered LLM output that includes the changed details and request additional modifications (e.g., to make it shorter, longer, more interesting, etc.). The edited sub-section of the rendered LLM response can then be assembled into a subsequent input prompt and processed as described herein.

[0062] exist Figure 3B In the process, the user has selected sub-part 356A of the rendered LLM output 354A (this sub-part can be combined with...). Figure 2 The selected sub-part 256A corresponds to this, and the subsequent request, "Make it start at 5:30 and end at 10:00," is provided in the query input field 372. Therefore, in Figure 3C The text includes a modified sub-part 356B (which can be combined with...). Figure 2 The subsequent rendered LLM output 354B (corresponding to the modified subsection 256B in the original text) has been used to replace the selected subsection 356A. As requested, the modified subsection 356B indicates that the gathering will begin at 5:30 p.m. and end at 10:00 a.m. the following day.

[0063] exist Figures 3A to 3C In the example, the user does not necessarily need to have already selected subsection 256A. Instead, the user may have issued the same command to change the start and end times, and the entire rendered LLM output 354A may have been reprocessed by the LLM response generation engine 128 to obtain the desired result. Figure 3CThe same subsequently rendered LLM output 354B is depicted. However, processing only the selected sub-section 356A instead of the entire rendered LLM output 354B consumes far fewer computational resources, since LLMs typically include hundreds of millions (if not billions) of parameters. Conserving resources in this way can significantly reduce latency and save power, especially given that numerous client computing devices interact with the NL-based response system 120 at any given time interval.

[0064] exist Figure 3D In the process, the user has selected the subsection 356C of the activity list for the birthday party plan in the subsequently rendered LLM output 354B. The user has already provided a request in the query input field 372, "Rewrite this inparagraph form". Therefore, in Figure 3E In this process, another subsequently rendered LLM output 354C, including the modified subsection 356D, has been used to replace the selected subsection 356C. As requested, the modified subsection 356D describes the activities for the party plan in paragraph form.

[0065] Now go to Figure 4 The diagram depicts a flowchart illustrating an example method 400 that implements a selected aspect of this disclosure. For convenience, the operation of method 400 is described with reference to a system performing the operation. This system of method 400 includes one or more processors, memories, and / or other components of a computing device. Furthermore, although the operations of method 400 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0066] In box 402, the system can receive queries or requests. For example, a user can speak or type a natural language request that is processed by the user input engine 111 and provided to the UX engine 136 and / or the LLM input engine 126. In box 404, the system can, for example, assemble a first LLM prompt based on the query via the LLM input engine 126.

[0067] In box 406, the system, for example via LLM response generation engine 128, can use LLM (e.g., 129) to process the first LLM hint to generate a first (raw) LLM response (e.g., Figure 2(Ref. 252A). As described elsewhere herein, the first LLM response (and other “raw” LLM responses described herein) may include sequences of terms, such as sequences of raw text that are mixed with metadata instructions in some cases. Metadata instructions may include formatting instructions (e.g., identified fonts, line breaks, indentation, spacing, etc.) and instructions for rendering other modalities of data such as images, videos, audio, graphics, etc.

[0068] In box 408, the system, for example via UX engine 136, can provide a first LLM response to a client application such as application 115, rendering engine 112, etc. In various implementations, the first LLM response can be provided by the client application through manipulation of the first rendered LLM output (e.g., ...). Figure 2 (see 254A in the original text). For example, rendering engine 112 can be configured to process the first LLM response to generate an HTML DOM hierarchy, which allows relevant content contained in the first LLM response to be rendered in a useful manner, for example, by application 115.

[0069] In box 410, the system can receive, for example, from the client application (i) a first rendered LLM output that has been selected using one or more input devices (e.g., by selecting extraction engine 130). Figure 2 Sub-parts of 254A in (e.g., Figure 2 The instructions in 256A) and (ii) the request for a modified version of the selected sub-section of the first rendered LLM output (e.g., Figure 2 (250B in the example). The indication of a sub-part (e.g., 256A) of the selected first rendered LLM output (e.g., 254A) may include, for example, a start character position and an end character position, or another type of mapping between the selected sub-part (e.g., 256A) of the first rendered LLM output and the corresponding sub-part of the first raw LLM response (e.g., 252A) generated in box 406.

[0070] In box 412, the system can, for example, extract or select, based on an instruction received in box 410, a sub-part of the first LLM response corresponding to a selected sub-part (e.g., 256A) of the first rendered LLM output (e.g., 254A). Figure 2(258 in the example). In box 414, the system, for example via LLM input engine 126, may assemble a selected sub-part (e.g., 258) of a first LLM response (e.g., 252A) with data indicating a request (e.g., 250B) for modifying the selected sub-part (258) of the first rendered LLM output as a second LLM hint. Figure 2 In 252B).

[0071] In box 416, the system, for example via LLM response generation engine 128, can use the same LLM (e.g., 129) or a different LLM to process the second LLM prompt to generate a second LLM response (e.g., Figure 2 (e.g., 252B). In box 418, the system may provide a second LLM response (e.g., 252B) to the client application, for example via UX engine 136. In various implementations, this second LLM response may be operable by the client application, for example via rendering engine 112, to provide a second rendered LLM output (e.g., 252B). Figure 2 (e.g., 254A) The second rendered LLM output includes at least a portion of the first rendered LLM output other than a selected sub-part of the first rendered LLM output and a modified version (e.g., 256B) of a selected sub-part of the first rendered LLM output (e.g., 254A) (e.g., 256A).

[0072] Now go to Figure 5 This diagram depicts a block diagram of an example computing device 510 that can be optionally utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automation assistant component, and / or other components may include one or more components of the example computing device 510.

[0073] Computing device 510 typically includes at least one processor 514 that communicates with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow users to interact with computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0074] User interface input device 522 may include a keyboard, pointing devices (such as a mouse, trackball, touchpad or graphics tablet, scanner, touchscreen integrated into the display), audio input devices (such as a voice recognition system, microphone), and / or other types of input devices. Generally, the term "input device" is intended to include all possible types of means and methods for inputting information into computing device 510 or into a communication network.

[0075] User interface output device 520 may include a display subsystem, printer, fax machine, or non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is intended to include all possible types of means and methods for outputting information from computing device 510 to a user or another machine or computing device.

[0076] Storage subsystem 524 stores the functional programming and data structures provided by some or all of the modules described herein. For example, storage subsystem 524 may include selected aspects and implementations of the methods disclosed herein. Figure 1 or Figure 2 The logic of the various components described in the document.

[0077] These software modules are typically executed by processor 514 alone or in conjunction with other processors. The memory 525 used in storage subsystem 524 may include multiple memories, including main random access memory (RAM) 530 for storing instructions and data during program execution and read-only memory (ROM) 532 for storing fixed instructions. File storage subsystem 526 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored in file storage subsystem 526 within storage subsystem 524 or in other machines accessible to processor 514. Bus subsystem 512 provides mechanisms for enabling various components and subsystems of computing device 510 to communicate with each other as intended. Although bus subsystem 512 is schematically shown as a single bus, alternative implementations of bus subsystem 512 may use multiple buses.

[0078] The computing device 510 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, [the following applies]. Figure 5 The description of the computing device 510 is intended only as a specific example for illustrating some implementation methods. Many other configurations of the computing device 510 are possible, and these configurations are related to... Figure 5 The computing device depicted has more or fewer components compared to the one described.

[0079] In situations where the systems described herein collect or otherwise monitor personal information about users, or may utilize personal information and / or monitored information, users may be given the opportunity to control whether programs or features collect user information (e.g., information about a user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from content servers. Furthermore, some data may be altered before it is stored or used, resulting in the removal of personally identifiable information. For example, a user's identity may be processed to the point that their personally identifiable information cannot be determined, or, if geographic location information is available, the user's geographic location may be generalized (e.g., to the city, zip code, or state level), making it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or used.

Claims

1. A method implemented using one or more processors, comprising: A large language model LLM is used to process the first LLM prompt to generate the first LLM response; The first LLM response is provided to the client application, wherein the first LLM response can be manipulated by the client application to provide a first rendered LLM output; Received from the client application: An indication of the first sub-section of the rendered LLM output that has been selected using one or more input devices, and A request for a modified version of a selected sub-section of the first rendered LLM output; Extract the sub-part of the first LLM response that corresponds to the selected sub-part of the first rendered LLM output; The selected sub-part of the first LLM response is assembled with the data of the request indicating the modification of the selected sub-part of the first rendered LLM output as a second LLM prompt; The second LLM prompt can be processed using the same LLM or a different LLM to generate a second LLM response; as well as The second LLM response is provided to the client application, wherein the second LLM response is operable by the client application to provide a second rendered LLM output, the second rendered LLM output including at least a portion of the first rendered LLM output other than a selected sub-portion of the first rendered LLM output and the modified version of the selected sub-portion of the first rendered LLM output.

2. The method as described in claim 1, wherein, The first LLM response includes a raw text string that includes metadata instructions for formatting the first rendered LLM output at the client application.

3. The method as described in claim 2, wherein, The selection includes receiving a start character position and an end character position from the client application, the start character position and the end character position identifying segments of the original text string outside the metadata instructions.

4. The method as described in any of the preceding claims, wherein, The request for a modified version of a selected sub-section of the first rendered LLM output includes a request to add one or more details to the selected sub-section of the first rendered LLM output.

5. The method as described in any one of the preceding claims, wherein, The request for a modified version of a selected sub-section of the first rendered LLM output includes a request to modify or replace one or more details of the selected sub-section of the first rendered LLM output.

6. The method as described in any of the preceding claims, wherein, The request for a modified version of a selected sub-part of the first rendered LLM output includes a request to add content to the selected sub-part of the first rendered LLM output that supports one or more details of the selected sub-part of the first rendered LLM output.

7. The method of claim 6, further comprising: The search query is formulated based on one or more details of the selected sub-part of the first rendered LLM output; Retrieve one or more documents from the search engine in response to the search query; as well as Data from the one or more documents in response to the search query is incorporated into the second LLM prompt.

8. The method as described in any of the preceding claims, wherein, The request for a modified version of a selected sub-section of the first rendered LLM output includes a natural language request.

9. The method as described in any of the preceding claims, wherein, The first LLM response includes metadata instructions for rendering one or more images, and a selected sub-part of the first rendered LLM output includes one or more rendered images.

10. The method of claim 9, wherein, The request for a modified version of a selected sub-part of the first rendered LLM output includes a request to replace one or more rendered images in the rendered images with one or more alternative images.

11. The method of claim 10, wherein, The request for replacing one or more rendered images with one or more alternative images includes a natural language request for retrieving one or more alternative images having specified visual features.

12. The method of claim 9, wherein, The request for a modified version of a selected sub-part of the first rendered LLM output includes a natural language request for generating a modified version of one or more of the rendered images, and processing the second LLM prompt using the same or a different LLM includes processing the natural language request using a text-to-image generative model to generate the modified version of one or more of the rendered images.

13. A system comprising one or more processors and a memory storing instructions, the instructions being responsive to execution by the one or more processors to cause the one or more processors to: A large language model LLM is used to process the first LLM prompt to generate the first LLM response; The first LLM response is provided to the client application, wherein the first LLM response can be manipulated by the client application to provide a first rendered LLM output; Received from the client application: An indication of the first sub-section of the rendered LLM output that has been selected using one or more input devices, and A request for a modified version of a selected sub-section of the first rendered LLM output; Select the sub-part of the first LLM response that corresponds to the selected sub-part of the first rendered LLM output; The selected sub-part of the first LLM response is assembled with the data of the request indicating the modification of the selected sub-part of the first rendered LLM output as a second LLM prompt; The second LLM prompt can be processed using the same LLM or a different LLM to generate a second LLM response; as well as The second LLM response is provided to the client application, wherein the second LLM response is operable by the client application to provide a second rendered LLM output, the second rendered LLM output including at least a portion of the first rendered LLM output other than a selected sub-portion of the first rendered LLM output and the modified version of the selected sub-portion of the first rendered LLM output.

14. The system of claim 13, wherein, The first LLM response includes a raw text string that includes metadata instructions for formatting the first rendered LLM output at the client application.

15. The system of claim 14, wherein, The instructions for extraction include instructions for receiving start character positions and end character positions from the client application, the start character positions and end character positions identifying segments of the original text string outside the metadata instructions.

16. The system as claimed in any one of claims 13 to 15, wherein, The request for a modified version of a selected sub-section of the first rendered LLM output includes a request to add one or more details to the selected sub-section of the first rendered LLM output.

17. The system as claimed in any one of claims 13 to 16, wherein, The request for a modified version of a selected sub-section of the first rendered LLM output includes a request to modify or replace one or more details of the selected sub-section of the first rendered LLM output.

18. The system as claimed in any one of claims 13 to 17, wherein, The request for a modified version of a selected sub-part of the first rendered LLM output includes a request to add content to the selected sub-part of the first rendered LLM output that supports one or more details of the selected sub-part of the first rendered LLM output.

19. The system of claim 18, further comprising instructions for: The search query is formulated based on one or more details of the selected sub-part of the first rendered LLM output; Retrieve one or more documents from the search engine in response to the search query; and Data from the one or more documents in response to the search query is incorporated into the second LLM prompt.

20. At least one non-transitory computer-readable medium, comprising instructions that, in response to execution by one or more processors, cause the one or more processors to: A large language model LLM is used to process the first LLM prompt to generate the first LLM response; The first LLM response is provided to the client application, wherein the first LLM response can be manipulated by the client application to provide a first rendered LLM output; Received from the client application: An indication of the first sub-section of the rendered LLM output that has been selected using one or more input devices, and A request for a modified version of a selected sub-section of the first rendered LLM output; Select the sub-part of the first LLM response that corresponds to the selected sub-part of the first rendered LLM output; The selected sub-part of the first LLM response is assembled with the data of the request indicating the modification of the selected sub-part of the first rendered LLM output as a second LLM prompt; The second LLM prompt can be processed using the same LLM or a different LLM to generate a second LLM response; as well as The second LLM response is provided to the client application, wherein the second LLM response is operable by the client application to provide a second rendered LLM output, the second rendered LLM output including at least a portion of the first rendered LLM output other than a selected sub-portion of the first rendered LLM output and the modified version of the selected sub-portion of the first rendered LLM output.

Images