Multimedia content management for large language models and / or other generative models

Abstract

Implementations relate to managing multimedia content obtained by large language models (LLMs) and / or generated by other generative models. A processor of a system can receive a natural language (NL)-based input requesting multimedia content, generate a response to the NL-based input, and cause the response to be presented. In some implementations, and in generating the response, the processor can process an LLM input using an LLM to generate an LLM output, and based on the LLM output, determine at least multimedia content to be included in the response. Further, the processor can evaluate the multimedia content to determine whether the multimedia content should be included in the response. In response to determining that the multimedia content should not be included in the response, the processor can cause the response including alternative multimedia content or other textual content to be presented.

Description

Background Technology

[0001] Various generative models have been proposed that can process natural language (NL) content and / or other inputs to generate outputs that reflect generative content in response to the input. For example, large language models (LLMs) have been developed that can process NL content and / or other inputs to generate LLM outputs that reflect generative NL content and / or other generative content in response to the input. These LLMs are typically trained on massive amounts of diverse data, including, but not limited to, data from web pages, ebooks, software code, e-news articles, and machine translation data. Therefore, these LLMs utilize the underlying data on which they were trained when performing these various NLP tasks. For example, when performing language generation tasks, these LLMs can process natural language (NL)-based input received from a client device and generate responses in response to that NL-based input that will be presented on the client device. In many cases, these LLMs enable the inclusion of text content in the response. In some cases, these LLMs can also additionally or alternatively include multimedia content, such as images, in the response (e.g., by causing image retrieval to be performed, by causing an image generation model to generate images, etc.).

[0002] However, some multimedia content, even if it can be acquired by these LLMs for inclusion in the response, should not be presented as part of the response. For example, some multimedia content that can be acquired by these LLMs may jeopardize the data security of the user of the client device or other users. As another example, some multimedia content that can be acquired by these LLMs may be used to facilitate illegal activities. Therefore, there is a need in the art to mitigate and / or eliminate situations where multimedia content acquired by these LLMs jeopardizes user data security and / or is used to facilitate illegal activities in a computationally efficient manner. Summary of the Invention

[0003] The implementation described herein relates to managing multimedia content acquired by a large language model (LLM) and / or generated by other generative models, which will be presented as part of a response generated by the LLM. The system's processor can: receive natural language (NL)-based input requesting multimedia content, generate a response to the NL-based input, and cause the response to be presented. In some implementations, and while generating the response, the processor can process the LLM input using the LLM to generate an LLM output, and based on the LLM output, determine at least the multimedia content to be included in the response. In some versions of those implementations, the processor can evaluate the multimedia content to determine whether the multimedia content should be included in the response to the NL-based input. In response to determining that the acquired and / or generated multimedia content should not be included in the response, the processor can cause the response, which includes alternative multimedia content or other text content and replaces the multimedia content, to be presented in response to the NL-based input.

[0004] For example, suppose the NL-based input is a prompt like "show me an image of a nerdy attorney." In this example, the processor can determine that, considering the potential scientific background required to become a patent attorney, a patent attorney is the most nerdy type of lawyer, and can obtain an image of the actual patent attorney to be included in the response. However, before the response including the image of the actual patent attorney is presented in response to the NL-based input, the processor can evaluate the multimedia content to determine whether it should be included in the response. In evaluating the multimedia content, the processor can utilize various evaluation models and / or evaluation techniques described herein to determine whether the multimedia content should be included in the response to be presented on the client device, and / or to determine the reasons why the multimedia content should not be included in the response to be presented on the client device. In this example, the evaluation might indicate that including the image of the actual patent attorney may jeopardize the data security of the actual patent attorney included in the image. Therefore, the processor can attempt to acquire another image of the patent attorney (e.g., a generative image of the patent attorney that does not depict the actual patent attorney) to be included in the response and in place of the initially acquired image, and / or can determine pre-defined textual content and / or other textual content to be included in the response and in place of the initially acquired image. As a result, these techniques mitigate and / or eliminate, in a computationally efficient manner, situations where multimedia content acquired by these LLMs jeopardizes user data security and / or is used to facilitate illegal activities.

[0005] In additional or alternative versions of those implementations, the processor may determine, based on one or more signals, whether to continue acquiring the multimedia content while it is being acquired and / or generated. In response to determining that acquisition of the multimedia content should not continue, the processor may cancel acquiring the multimedia content and may cause a response, including alternative multimedia content or other text content, to be presented in response to the NL-based input. The one or more signals may include, for example, an NL-based input context relative to the multimedia content requested by the NL-based input, a multimedia content context relative to the NL-based input, a response context relative to the text content, and / or other signals.

[0006] For example, suppose the NL-based input is a prompt to "show me an image of a nerdy attorney." In this example, the processor can again determine that, considering the potential scientific background required to become a patent attorney, a patent attorney is the most academic type of lawyer, and can initiate the retrieval of an image of the actual patent attorney to be included in the response. However, while retrieving the image of the actual patent attorney to be presented in response to the NL-based input, the processor can evaluate one or more signals, without considering the context of the actual multimedia content to be retrieved, to determine whether the multimedia content to be retrieved should be included in the response to be presented at the client device. In this example, if the processor determines that the image of the actual patent attorney should not be presented in the context of the received NL-based input, if the multimedia content query for retrieving the image of the actual patent attorney should not have been used (e.g., because the user's location information is included in the multimedia content query), if the multimedia content query for retrieving the image of the actual patent attorney should not have been used given the surrounding text content to be included in the response (e.g., identifying the actual patent attorney's name and / or business location), etc., the retrieval of the image of the actual patent attorney can be aborted. Therefore, the processor can attempt to acquire another image of the patent attorney (e.g., a generative image of the patent attorney that does not depict the actual patent attorney) to be included in the response and in place of the initially acquired image, and / or can determine pre-defined textual content and / or other textual content to be included in the response and in place of the initially acquired image. As a result, these techniques mitigate and / or eliminate, in a computationally efficient manner, situations where multimedia content acquired by these LLMs jeopardizes user data security and / or is used to facilitate illegal activities, while also balancing latency.

[0007] In additional or alternative versions of those implementations, the processor may, upon receiving the NL-based input, determine whether to retrieve the requested multimedia content based on the NL-based input itself and before initiating retrieval of the requested multimedia content. In response to determining that retrieval of the requested multimedia content is not initiated, the processor may cause a response, including alternative multimedia content or other text content, to be presented in response to the NL-based input. It is noteworthy that, compared to the implementations described above, the processor may not even attempt to retrieve the multimedia content at all.

[0008] For example, suppose the NL-based input is a prompt like "show me an image of a nerdy attorney." In this example, the processor can again determine that, considering the potential scientific background required to become a patent attorney, a patent attorney is the most academic type of lawyer, and can initiate the retrieval of an image of the actual patent attorney to be included in the response. However, before initiating the retrieval of the image of the actual patent attorney to be presented in response to the NL-based input, the processor can determine that the multimedia content requested by the user in the NL-based input should not be retrieved. Therefore, the processor can determine the preset text content and / or other text content to be included in the response and in place of the initially retrieved image (e.g., "how about I generate an image of a nerdy attorney"). As a result, these techniques mitigate and / or eliminate, in a computationally efficient manner, situations where multimedia content retrieved by these LLMs jeopardizes user data security and / or is used to facilitate illegal activities, while also balancing latency.

[0009] The above description is provided as an overview of some implementations of this disclosure. Further descriptions of those implementations and others will follow in more detail. Attached Figure Description

[0010] Figure 1 A block diagram depicts an example environment that illustrates various aspects of this disclosure, and in which some of the implementations disclosed herein can be implemented.

[0011] Figure 2 It describes the use of various implementation methods from Figure 1 Various components are used to manage the example process flow of multimedia content obtained from a large language model (LLM).

[0012] Figure 3A flowchart illustrating example methods for managing multimedia content acquired by a large language model (LLM) according to various implementations is provided.

[0013] Figure 4 A flowchart is depicted illustrating another example approach to managing multimedia content acquired by a large language model (LLM) based on various implementations.

[0014] Figure 5 A flowchart is depicted illustrating yet another example approach to managing multimedia content acquired by a large language model (LLM) based on various implementations.

[0015] Figure 6A , Figure 6B , Figure 6C and Figure 6D Various non-restricted examples of managing multimedia content acquired from a large language model (LLM) are described, based on different implementation methods.

[0016] Figure 7 Example architectures of computing devices based on various implementations are described. Detailed Implementation

[0017] Turn now Figure 1 This document depicts a block diagram of an example environment illustrating various aspects of this disclosure, in which the implementations disclosed herein can be carried out. This example environment includes a client device 110 and a multimedia content management system 120. In some implementations, all or aspects of the multimedia content management system 120 may be implemented locally at the client device 110. In additional or alternative implementations, all or aspects of the multimedia content management system 120 may be implemented as follows: Figure 1 As depicted, this is implemented remotely from the client device 110 (e.g., at a remote server). In those implementations, the client device 110 and the multimedia content management system 120 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks (“LANs”, including Wi-Fi, mesh networks, Bluetooth, near field communication, etc.) or wide area networks (“WANs”, including the Internet).

[0018] Client device 110 may be, for example, one or more of the following: desktop computer, laptop computer, tablet computer, mobile phone, vehicle computing device (e.g., in-vehicle communication system, in-vehicle entertainment system, in-vehicle navigation system), independent interactive speaker (optionally with a display), smart home appliance (such as a smart TV), and / or user's wearable device including a computing device (e.g., user's watch with computing device, user's glasses with computing device, virtual reality or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0019] Client device 110 may execute one or more software applications via application engine 115, through which NL-based input can be submitted and / or responses to the NL-based input can be presented (e.g., audibly and / or visually). Application engine 115 may execute one or more software applications separate from the operating system of client device 110 (e.g., an application installed "on top of" the operating system) – or may alternatively be implemented directly by the operating system of client device 110. For example, application engine 115 may execute a web browser or automation assistant installed on top of the operating system of client device 110. As another example, application engine 115 may execute a web browser software application or automation assistant software application integrated as part of the operating system of client device 110. Application engine 115 (and the one or more software applications executed by application engine 115) may interact with or otherwise provide access to (e.g., as a front end) the multimedia content management system 120.

[0020] In various implementations, client device 110 may include a user input engine 111 configured to detect user input provided by a user of client device 110 using one or more user interface input devices. For example, client device 110 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to a user's spoken words or other sounds in the environment of client device 110. Additionally or alternatively, client device 110 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or motions (e.g., gestures) detected within the field of view of the one or more visual components. Additionally or alternatively, client device 110 may be equipped with one or more touch-sensitive components (e.g., keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to typing and / or touch input directed at client device 110.

[0021] Some examples of NL-based input described herein can be queries for responses based on user input provided by the user of client device 110 and detected by user input engine 111. For example, the query could be a typing query via a physical or virtual keyboard, a suggested query selected via a touchscreen or mouse of client device 110, a spoken query detected via a microphone of client device 110 (and optionally directed to an automated assistant performing at least partially at client device 110), or an image or video query based on visual data captured by the vision components of client device 110 (or NL-based input generated by processing the image using, for example, an object detection model, a captioning model, etc.). Other examples of NL-based input described herein can be prompts for content based on user input provided by the user of client device 110 and detected by user input engine 111. For example, the prompt may be a typing prompt entered via a physical or virtual keyboard, a suggested prompt selected via the touchscreen or mouse of the client device 110, a verbal prompt detected by the microphone of the client device 110, or an image or video prompt based on an image or video captured by the visual component of the client device 110.

[0022] In various implementations, client device 110 may include a rendering engine 112 configured to present content (e.g., a unimodal response consisting only of text content or only of multimedia content, a multimodal response consisting of both text content and multimedia content, an indication of a source associated with a portion of the unimodal and / or multimodal response, and / or other content) for an audible and / or visual presentation to a user of client device 110 using one or more user interface output devices. For example, client device 110 may be equipped with one or more speakers that enable audible content to be provided for an audible presentation to a user via client device 110. Additionally or alternatively, client device 110 may be equipped with a display or projector that enables text content or other visual content (e.g., images, videos, etc.) to be provided for a visual presentation to a user via client device 110.

[0023] In various implementations, client device 110 may include a context engine 113 configured to determine the client device context (e.g., the current or recent context) and / or the user context of the client device 110's users (or the active users of the client device 110 when the client device 110 is associated with multiple users). In some of those implementations, the context engine 113 may determine the context based on data stored in the client device data database 110A. The data stored in the client device data database 110A may include, for example, user interaction data characterizing the current or recent interactions of the client device 110 and / or the user of the client device 110, location data characterizing the current or recent location of the client device 110 and / or the geographic area associated with the user of the client device 110, user attribute data characterizing one or more attributes of the user of the client device 110, user preference data characterizing one or more preferences of the user of the client device 110, user profile data characterizing the profile of the user of the client device 110, and / or any other data that the scene engine 113 can access via the client device data database 110A or otherwise.

[0024] For example, context engine 113 may determine the current context based on the current state of the conversation (e.g., considering one or more recent NL-based inputs provided by the user during the conversation), profile data, and / or the current location of client device 110. For instance, context engine 113 may determine the current context of “visitor looking for upcoming events in Louisville, Kentucky” based on recently issued queries, profile data, and the expected future location of client device 110 (e.g., based on recently booked hotel accommodations). As another example, context engine 113 may determine the current context based on which software application is active in the foreground of client device 110, the current or recent state of that active software application, and / or the content currently or recently presented by that active software application. The context determined by the context engine 113 can be used, for example, to supplement or rewrite NL-based input specified by user input, generate implicit NL-based input (e.g., implicit queries or prompts specified independently of any explicit NL-based input provided by the user of client device 110), and / or determine the submission of implicit NL-based input and / or present results (e.g., responses) for implicit NL-based input.

[0025] In various implementations, client device 110 may include an implicit input engine 114 configured to: generate implicit NL-based input independently of any explicit user-submitted NL-based input provided by a user of client device 110; optionally submit implicit NL-based input independently of any explicit user-submitted NL-based input; and / or optionally render search results or responses for implicit NL-based input independently of any explicit NL-based input requesting search results or responses. For example, implicit input engine 114 may use one or more past or current contexts from context engine 113 to generate implicit NL-based input, determine to submit the implicit NL-based input, and / or determine to render search results or responses in response to the implicit NL-based input. For example, implicit input engine 114 may automatically generate and submit implicit queries or implicit hints based on the one or more past or current contexts. Furthermore, the implicit input engine 114 can automatically push search results or responses generated in response to the implicit query or implicit prompt so that they are automatically presented, or it can automatically push notifications of search results or responses, such as optional notifications, which, when selected, cause the search results or responses to be presented. Additionally or alternatively, the implicit input engine 114 can submit corresponding implicit NL-based input at regular or irregular intervals, causing the corresponding search results or responses to be automatically provided (or their notifications automatically provided). For example, implicit NL-based input could be “patent news” based on one or more past or current contexts indicating a user’s general interest in a patent, which is submitted periodically, and the corresponding search results or responses can be automatically provided (or their notifications automatically provided). It should be noted that, given the existence of new / fresh search result documents over time, the corresponding search results or responses may change over time.

[0026] Furthermore, the client device 110 and / or the multimedia content management system 120 may include one or more memories for storing data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components that facilitate communication via the one or more networks 199. In some implementations, the one or more software applications may be locally installed on the client device 110, while in other implementations, the one or more software applications may be remotely hosted (e.g., by one or more servers) and accessible by the client device 110 via the one or more networks 199.

[0027] although Figure 1The aspects described herein relate to a single client device with a single user, but it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, one or more additional client devices of the user and / or additional users may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user may form a device ecosystem that can employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (e.g., via network 199). As another example, a given client device may be utilized by multiple users in a shared setting (e.g., a group of users, a home, a workplace, a hotel, etc.).

[0028] Multimedia Content Management System 120 Figure 1 The diagram illustrates a fine-tuning engine 130, an LLM engine 140, a text content engine 150, a multimedia content engine 160, and a responsive modification engine 170. Some of these engines may be combined and / or omitted in various implementations. Furthermore, these engines may include various sub-engines. For example, the fine-tuning engine 130 in... Figure 1 The diagram shows training instance engine 131 and training engine 132. Additionally, LLM engine 140... Figure 1 The diagram illustrates an explicit LLM engine 141 and a conversational LLM engine 142. Furthermore, the multimedia content engine 160... Figure 1 The diagram illustrates a multimedia content tagging engine 161, a generative multimedia content prompting engine 162, a generative multimedia content model selection engine 163, and a multimedia content retrieval engine 164. Additionally, a response modification engine 170... Figure 1 The engine is described as including an evaluation engine 171 and a pause processing engine 172. Similarly, some of these sub-engines can be combined and / or omitted in various implementations. Therefore, it should be understood that... Figure 1 The various engines and sub-engines of the multimedia content management system 120 illustrated in the diagram are depicted to describe certain functions and are not intended to be limiting.

[0029] In addition, the multimedia content management system 120 Figure 1The diagram illustrates connections to various databases, such as a training instance database 130A, an LLM database 140A, a curated multimedia content database 160A, and an evaluation model database 170A. While specific engines and / or sub-engines are depicted as capable of accessing specific databases, it should be understood that this is for illustrative purposes and not intended to be limiting. For example, in some implementations, each of the various engines and / or sub-engines of the multimedia content management system 120 may be able to access each of the various databases. Furthermore, some of these databases may be combined and / or omitted in various implementations. Therefore, it should be understood that... Figure 1 The various databases connected to the multimedia content management system 120 as shown in the diagram are depicted to describe certain data accessible to the multimedia content management system 120 and are not intended to be limiting.

[0030] Moreover, the multimedia content management system 120 in Figure 1 The multimedia content management system 120 is illustrated as being connected to other systems, such as search system 180 and generative system 190. In addition to the multimedia content included in the curated multimedia content database 160A, the multimedia content retrieval engine 164 can generate and transmit requests to search system 180 and / or generative system 190 to retrieve multimedia content to be included in responses to NL-based inputs, as described herein. In some implementations, search system 180 and / or generative system 190 are first-party systems, while in other implementations, search system 180 and / or generative system 190 are third-party systems. As used herein, the term "first-party" or "first-party entity" refers to the entity that controls, develops, and / or maintains the multimedia content management system 120, while the term "third-party" or "third-party entity" refers to an entity different from the entity that controls, develops, and / or maintains the multimedia content management system 120.

[0031] As described in more detail in this article (e.g., regarding...) Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6A , Figure 6B , Figure 6C and Figure 6DThe multimedia content management system 120 can be used to generate responses to corresponding NL-based inputs received at the client device 110. The responses described herein may include text content in response to corresponding NL-based inputs, multimedia content in response to corresponding NL-based inputs, or both. The multimedia content described herein may include multimedia content items, such as images, video clips, audio clips, GIFs, and / or any other multimedia content suitable for presentation at the client device 110. In an implementation where multimedia content is retrieved using the search system 180, the multimedia content may be considered "non-generative multimedia content." In other words, the non-generative multimedia content described herein may include existing multimedia content that can be retrieved via the search system 180. In an implementation where multimedia content is retrieved using the generative system 180, the multimedia content may be considered "generative multimedia content." In other words, the generative multimedia content described herein may include multimedia content generated by the generative system 190 in response to a prompt received from the multimedia content management system 120 regarding such generative multimedia content (e.g., "instantaneously" or "real-time"). Unless otherwise expressly stated, non-generative multimedia content and generative multimedia content are collectively referred to as "multimedia content" in this document.

[0032] It is worth noting that before receiving any NL-based input from the user of the client device 110, the LLM (e.g., an LLM stored in the LLM database 140A and subsequently used by the conversational LLM engine 142 to generate responses to corresponding NL-based inputs received from the user of the client device) can be fine-tuned to acquire the multimedia content. In these implementations, the training instance engine 131 can acquire multiple training instances to be used to fine-tune the LLM. For example, to enable the LLM to acquire non-generative multimedia content, one or more of the multiple training instances may include: (1) corresponding NL-based input; and (2) corresponding multimedia content tags indicating corresponding multimedia content items in response to the corresponding NL-based input. These training instances may further include corresponding text content, which is interleaved with the corresponding multimedia content tags, to further enable the LLM to logically arrange the non-generative multimedia content relative to the text content. As another example, in order for the LLM to acquire generative multimedia content, one or more of the plurality of training instances may include: (1) a corresponding NL-based input; and (2) a corresponding generative multimedia content cue indicating a corresponding generative multimedia content item in response to the corresponding NL-based input. Similarly, these training instances may further include corresponding text content, which is interleaved with the corresponding generative multimedia content cue, to further enable the LLM to logically arrange the generative multimedia content relative to the text content.

[0033] In some versions of these implementations, one or more of the plurality of training instances can be curated by, for example, a developer associated with the multimedia content management system 120. For example, any corresponding text content for NL-based input and response can be obtained from the dialogue log, and the developer can manually add corresponding multimedia content tags and / or corresponding generative multimedia content cues. In additional or alternative implementations, one or more of the plurality of training instances can be generated using an automated process. For example, any corresponding text content for NL-based input and multimodal response can be obtained from the dialogue log, and corresponding multimedia content tags and / or corresponding generative multimedia content cues can be automatically inserted into the training instances. Upon acquisition and / or generation, the training instance engine 131 can store the plurality of training instances in the training instance database 130A.

[0034] It is worth noting that the corresponding generative multimedia content cues included in each training instance can be more detailed than the corresponding multimedia content tags included in each additional training instance. This subsequently enables the LLM to generate detailed cues related to various concepts included in the NL-based input of the training instance. For example, if the NL-based input for a given training instance is “I'm planning a trip to Rome next summer, what are the must-see attractions?”, then the corresponding text content could include information about “The Colosseum in Rome”, followed by the multimedia content tag “{tag:[image of The Colosseum in Rome] image {url: ...}}”, and then additional corresponding text content in response to that NL-based input. It is worth noting that the multimedia content tag does not include a lot of details because a typical search system (e.g., the image search system of search system 180) can easily obtain images of the "Colosseum" by performing a simple image search using the image query term "Colosseum".Conversely, if the NL-based input for a given training instance is “narrate a Kentucky Derby race with Mohammed Ali as the winning jockey riding Secretariat as he uses a Louisville Slugger as a riding crop, and include pictures or a video of the race,” then the corresponding text content could include a fictional story about how Mohammed Ali, as a Kentucky Derby rider, prepared with the Secretariat through a rigorous training program, followed by “{prompt: [generate an image of Mohammad Ali breezing Secretariat with the Twin Spires of Churchill Downs in the background during the month of May] image generator {url:…}} ({prompt: [generate an image of Mohammad Ali breezing Secretariat with the Twin Spires of Churchill Downs in the background during the month of May] image generator {url:…}} The system provides a generative multimedia content prompt, followed by additional corresponding text content in response to the NL-based input. It is noteworthy that this generative multimedia content prompt does indeed include a significant amount of detail, as typical generative systems (e.g., the image generator of generative system 190) often require more information than typical search systems to accurately reflect the scene included in the NL-based input.

[0035] Furthermore, training engine 132 can retrieve a given training instance from training instance database 130A and cause the LLM to process that given training instance. It is noteworthy that, since the given training instance includes corresponding multimedia content tags, corresponding generative multimedia content prompts, and / or the text content, the LLM is effectively fine-tuned to perform the specific task of determining when to include multimedia content in the response and where to include the multimedia content in the response. Training engine 132 can continue to fine-tune the LLM in this manner until one or more conditions are met. These one or more conditions may include, for example, whether the LLM has been fine-tuned based on a threshold number of training instances, whether a threshold duration has elapsed since the start of the fine-tuning process, whether the LLM's performance has reached a threshold performance level, and / or other conditions. Training engine 132 can then cause the LLM to be deployed to generate responses to NL-based input received from a user on client device 110 (e.g., regarding...). Figure 3 , Figure 4 and Figure 5 As described herein (e.g., regarding...). Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6A , Figure 6B , Figure 6C and Figure 6D Additional descriptions are provided for the LLM engine 140, text content engine 150, multimedia content engine 160, and responsive modification engine 170.

[0036] It is worth noting that by fine-tuning the LLM in this way, the Multimedia Content Management System 120 effectively enables any multimedia content to be acquired (e.g., via search system 180) and / or generated (e.g., via generative system 190). In other words, the amount of multimedia content that can be acquired and / or generated by the Multimedia Content Management System 120 is virtually unlimited. However, some multimedia content that can be acquired and / or generated by the Multimedia Content Management System 120 should not be presented on the user's client device 110. For example, some multimedia content described herein may jeopardize the data security of the user or other users, be used to facilitate illegal activities, and / or be otherwise unsuitable for presentation on the client device 110. Therefore, the Multimedia Content Management System 120 described herein is provided to manage the multimedia content presented on the client device 110 to ensure that the data security of the user and other users is maintained, to mitigate and / or eliminate illegal activities, and to provide other technical advantages. For example, the Multimedia Content Management System 120 can achieve the above-mentioned technical advantages in various implementations by using parallel processing in a so-called "one-shot" manner, thereby reducing the latency that causes the response to be presented on the client device. These and other technological advantages are discussed in this article (e.g., regarding...). Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6A and Figure 6B (To be described in more detail)

[0037] Turn now Figure 2 It describes the use of from Figure 1 An example process flow 200 is used to manage multimedia content acquired by a large language model (LLM) using various components. For example, assume that the user input engine 111 of the client device detects NL-based input 201. For example, assume that NL-based input 201 is a prompt "show me an image of a nerdyattorney". Although Figure 2 The process flow 200 is described in relation to the NL-based input 201 being an explicitly NL-based input, but it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, the NL-based input 201 may additionally or alternatively be an implicit NL-based input (e.g., as described with respect to the implicit input engine 114).

[0038] Further assuming that the NL-based input 201 is provided to the explicit LLM engine 141. The explicit LLM 141 may be a form of LLM that processes the NL-based input 201 (and optionally the content 202 determined by the content engine 113 of the client device) to generate an LLM input 203. The LLM input 203 can then be provided to the conversational LLM engine 142 to generate an LLM output 204. In other words, the explicit LLM 141 can process the raw NL-based input 201 and place it in a structured form more suitable for processing by the conversational LLM engine 142. The explicit LLMs and / or conversational LLMs utilized by these respective engines may include, for example, any LLM stored in the LLM database 140A, such as PaLM, BARD, BERT, LaMDA, Meena, GPT, and / or any other LLM, such as any other encoder-only, decoder-only, sequence-to-sequence based LLM, and optionally include attention mechanisms or other memories, and be fine-tuned as described herein (e.g., regarding...). Figure 3 The multimedia content tags are generated. It is worth noting that when generating LLM input 203, the explicit engine 141 can further process a prompt that indicates that the original NL-based input 201 (and optionally the context 202) should be placed in a structured form that is more suitable for processing by the conversational LLM engine 142.

[0039] In some implementations, the explicit LLM engine 141 may generate one or more queries based on the NL-based input 201, submit the queries to one or more search systems (e.g., search system 180), and process the search result documents when generating the LLM input 203. Continuing the example above, the explicit LLM engine 141 may generate and submit a first query, “scholarly lawyer,” to obtain search results indicating that, considering the potential scientific background required to become a patent lawyer, the most scholarly type of lawyer is a patent lawyer. Therefore, this information can be included not only in the LLM input 203 for subsequent determination of the text content of the response, but also in the LLM input 203 for subsequent determination of the multimedia content to be included with the response.

[0040] Furthermore, when generating LLM output 204, the conversational LLM engine 142 can generate LLM output 204 as, for example, a probability distribution over a sequence of lexical units, such as words, phrases, or other semantic units, predicted in response to NL-based input 201, non-generative multimedia content tags for obtaining non-generative multimedia content predicted in response to NL-based input 201, and / or generative multimedia content cues for obtaining generative multimedia content predicted in response to NL-based input 201. As mentioned above, the LLM can include millions or billions of weights and / or parameters, which are learned by training the LLM on a large amount of diverse data. This enables the LLM to generate LLM output as a probability distribution over a sequence of lexical units. Furthermore, the LLM can be fine-tuned (e.g., as per...). Figure 1 As described, this enables LLM to generate LLM outputs that include sequences of lexical terms on non-generative multimedia content tags and / or generative multimedia content cues.

[0041] Further assuming that LLM output 204 is provided to both text content engine 150 and multimedia content engine 160. In this example, text content engine 150 may determine, based on the probability distribution over a sequence of lexical units (e.g., over words, phrases, or other semantic units), text content 205 (if any) to be included in the response 208 to the NL-based input. Continuing the example above, where the NL-based input 201 is the prompt “show me an image of a nerdyattorney,” text content 205 may include “sure, here is an image of a patent attorney, the nerdiest type of attorney there is” and / or other text content.

[0042] Similarly, in this example, the multimedia content engine 160 can determine the multimedia content 206 to be included in the response 208 in response to the NL-based input 201 based on the probability distribution over the sequence of lexical units (e.g., over non-generated multimedia content tags and / or generated multimedia content cues). As described above, the conversational LLM used by the conversational LLM engine 142 to generate the LLM output 204 can be fine-tuned to generate non-generated multimedia content tags and / or generated multimedia content cues (e.g., as about...). Figure 1(As described). The multimedia content tagging engine 161 can parse the LLM output 204 itself and / or the text content 205 to identify any non-generated multimedia content tags. Furthermore, the generative multimedia content prompting engine 162 can parse the LLM output 204 itself and / or the text content 205 to identify any generative multimedia content prompts. Continuing the example above, where the NL-based input 201 is the prompt “show me an image of a nerdy attorney”, the LLM output 204 itself and / or the text content 205 can include non-generated multimedia content tags (e.g., {tag:[Image of a patent attorney] image {url: …}}), because there are numerous readily available images depicting patent attorneys (e.g., via search system 180). Additionally or alternatively, the LLM output 204 itself and / or the text content 205 may include a generative multimedia content prompt (e.g., {prompt: [patent attorney; nerdy; draftingpatent application] image generator {url: …}} ({prompt: [patent attorney; nerdy; drafting patent application] image generator {url: …}})), which will be submitted to a generative model (e.g., an image generator) to generate a generative image of the patent attorney.

[0043] It is worth noting that in the LLM output 204 itself and / or text content 205, non-generated multimedia content tags and / or generated multimedia content prompts may be interleaved relative to the text content 205. However, it should be noted that non-generated multimedia content tags and / or generated multimedia content prompts may not be included in the response 208 presented to a user providing NL-based input 201. More precisely, non-generated multimedia content tags and / or generated multimedia content prompts are replaced (or superseded) by multimedia content 206 corresponding to non-generated multimedia content items obtained based on non-generated multimedia content tags (e.g., via search system 180) and / or generated multimedia content items generated based on generated multimedia content prompts (e.g., via generated system 190).

[0044] In implementations where the LLM output 204 itself and / or text content 205 include non-generative multimedia content tags, the multimedia content engine 160 can determine the type of search system 180 to be used to retrieve non-generative multimedia content items, such as an image search system (e.g., based on the non-generative multimedia content tag {tag: [Image of apatent attorney] image {url: …}} containing "image"). However, it should be understood that other search systems are also envisioned herein (e.g., video search systems, audio search systems, animated image search systems, etc.). Furthermore, in implementations where the LLM output 204 itself and / or text content 205 include generative multimedia content hints, the generative multimedia content model selection engine 163 can utilize the generative multimedia content hints to select a given generative multimedia content model from multiple different generative multimedia content models to process the generative multimedia content hints. As mentioned above regarding… Figure 1 The plurality of different generative multimedia content cues may include a first-party generative multimedia content model and / or a third-party generative multimedia content model. Furthermore, the plurality of different generative multimedia content cues may include an image generator, a video generator, an audio generator, and / or any other generative model capable of processing cues to generate generative multimedia content.

[0045] While generative multimedia content prompts may include an indication of the type of generative model to be used to generate generative multimedia content items (e.g., based on the generative multimedia content prompt {prompt: [patent attorney;nerdy; drafting patent application] image generator {url: …}} containing "imagegenerator"), there may be multiple image generator models capable of generating generative images. For example, these multiple different generative multimedia content prompts may include image generators, video generators, audio generators, and / or other generative models of different sizes (e.g., including generative models with billions of parameters (e.g., 100 billion, 250 billion, 500 billion, etc.) or millions of parameters (e.g., 100 million, 250 million, 500 million, etc.)). Specifically, the generative multimedia content model selection engine 163 may utilize the type of generative multimedia content to be generated (e.g., as indicated by the generative multimedia content prompt) to select a given generative multimedia content model to process the generative multimedia content prompt, and also considers the different sizes of the models to reduce computational resource consumption in human-computer interaction.

[0046] In an implementation where the LLM output 204 itself and / or the text content 205 includes non-generated multimedia content tags, the multimedia content retrieval engine 164 can determine a multimedia content query based on the non-generated multimedia content tags and submit the multimedia content query to a multimedia content search system (e.g., to one or more of search systems 180 and via one or more of networks 199). In response to the submission of the multimedia content query, the multimedia content retrieval engine 164 can receive non-generated multimedia content items as multimedia content 206 to be included in response 208. Furthermore, in an implementation where the LLM output 204 itself and / or the text content 205 includes generated multimedia content prompts, the multimedia content retrieval engine 164 can cause non-generated multimedia content tags to be submitted to a given generated multimedia content model (e.g., via generated systems 190 and via one or more of networks 199). In response to the submission of generated multimedia content prompts to a given generated multimedia content model, the multimedia content retrieval engine 164 can receive generated multimedia content items as multimedia content 206 to be included in response 208.

[0047] In various implementations, and before rendering engine 112 renders response 208, multimedia content management system 120 may determine whether response 208 should be rendered or whether response 208 should be modified before it is rendered (e.g., as indicated by 207). In some versions of these implementations, evaluation engine 171 may use evaluation models stored in evaluation model database 170A to process NL-based input 201 and text content 205 (if any) and / or multimedia content 206 to generate evaluation output. Furthermore, evaluation engine 171 may, based on this evaluation output, determine whether response 208 should be rendered or whether response 208 should be modified before it is rendered. In response to determining that response 208 should be rendered, rendering engine 112 may cause response 208 to be rendered. However, in response to determining that the response should not be rendered, response 208 may be modified before any content is rendered for display to a user providing NL-based input 201.

[0048] The evaluation model stored in the evaluation model database 170A may include, for example, a visual language model (VLM) or other machine learning (ML) model. In an implementation where the evaluation model corresponds to a VLM, the evaluation engine 171 may use the VLM to process the NL-based input 201, and text content 205 (if any) and / or multimedia content 206 to generate a VLM output as the evaluation output. Furthermore, the evaluation engine 171 may determine, based on the VLM output, whether the response 208 should be rendered. Notably, when processing the NL-based input 201, and text content 205 (if any) and / or multimedia content 206 to generate the VLM output, the evaluation engine 171 may further cause the VLM to process a prompt, which includes a request for the VLM to determine whether the response 208 should be rendered at the client device 110 given the NL-based input 201. Not only can the VLM output include an indication of whether response 208 should be presented at client device 110 and given NL-based input 201, the VLM output may additionally or alternatively include a reason for whether response 208 should be presented at client device 110 and given NL-based input 201.

[0049] Continuing the example above, where the NL-based input 201 is the prompt “show me an image of a nerdy attorney”, further assuming that the text content 205 includes “sure, here is an image of a patent attorney, the nerdiest type of attorney there is” and the multimedia content 206 includes the image of the patent attorney as described above. In this example, the evaluation engine 171 can use VLM to process the prompt “show me an image of a nerdy attorney”, the text content 205 “sure, here is an image of a patent attorney, the nerdiest type of attorney there is”, and / or the multimedia content 206 including the image of the patent attorney. Further assuming that the VLM processes the prompt “should the image be rendered given the context of the NL-based input and the textual content included in the response?” In this example, it is further assumed that the VLM output indicates that multimedia content 206 including an image of a patent attorney should not be presented, because the combination of the image and the text of response 208 may be considered offensive. Furthermore, it is assumed that the image is non-generative and includes an actual patent attorney, which could jeopardize the data security of the actual patent attorney depicted in the image.

[0050] However, the evaluation engine 171 can initiate additional iterations of the processing using the conversational LLM engine 142 to generate additional LLM output 204, instead of outputting some preset text content (e.g., "I'm sorry, I can't provide the requested image"). It is worth noting that in these additional iterations of the processing using the conversational LLM engine 142, the additional LLM output processed to generate the additional LLM output may include a portion of the VLM output (e.g., reasons why response 208 should be presented at client device 110 and given the NL-based input 201). Therefore, in these additional iterations of the processing, any generated alternative multimedia content subsequently acquired to be included in response 208 to be presented at client device 110 in response to the NL-based input 201 is unlikely to be considered offensive or jeopardize the data security of any actual patent attorney. For example, alternative multimedia content may include a generated image of the patent attorney (i.e., not an actual patent attorney, but a cartoon-like patent attorney). Furthermore, the response modification engine 170 can again determine whether alternative multimedia content should be included in the presented response 208. This process can be repeated iteratively until... N The iteration (where, N This is a positive integer, which can be fixed (e.g., defined by the developers associated with the multimedia content management system 120) or dynamic (e.g., based on one or more of the following: LLM lexical constraints, LLM time constraints, or LLM computational constraints)). If executed N If the next iteration fails to identify suitable multimedia content, the pause processing engine 172 can pause any further processing, and preset text content can be provided as a response 208.

[0051] Although the above examples are described with respect to the evaluation model being VLM, it should be understood that they are for illustrative purposes and not intended to be limiting. For example, in additional or alternative versions of those implementations, evaluation engine 171 may use a caption model stored in evaluation model database 170A to process multimedia content 206 to generate one or more corresponding captions for multimedia content 206. In these implementations, evaluation engine 171 may cause the LLM to process one or more of the corresponding captions of multimedia content 206 and NL-based input 201 (and optionally text content 205, if any) to generate additional LLM outputs as evaluation outputs. Similarly, when processing this content, the LLM may also handle the prompt "should the image be rendered given the context of the NL-based input and the textual content included in the response?". In these implementations, the evaluation output may include not only an indication of whether response 208 should be presented at client device 110 given NL-based input 201, but also, additionally or alternatively, a reason for whether response 208 should be presented at client device 110 given NL-based input 201. As another example, in additional or alternative versions of those implementations, evaluation engine 171 may retrieve tags stored associated with multimedia content 206 (e.g., assuming multimedia content 206 is non-generative multimedia content and tags are actually stored associated with multimedia content 206), and use one or more of the tags to replace the corresponding annotations to generate the evaluation output.

[0052] Furthermore, while the above examples are described relative to non-generated content that can be obtained from a search system, it should be understood that they are for illustrative purposes and not intended to be limiting. In additional or alternative implementations, non-generated multimedia content items can be obtained from a curated multimedia content database 160A. In these implementations, if the entity identified in the multimedia content tag is a specific type of entity, for example, it may be considered sensitive, personal, controversial, etc., then the multimedia content retrieval engine 163 can submit a multimedia search query based on the curated multimedia content database 160A. For example, if the multimedia content tag indicates that an image of the U.S. president should be included in response 208, then an official presidential portrait from the curated multimedia content database 160A can be obtained as multimedia content 206. However, it should be understood that whether the LLM output 204 itself and / or the text content 205 includes multimedia content tags (rather than the aforementioned generated multimedia content prompts) may depend on the user-provided NL-based input 201 and / or the LLM-generated LLM output 204 and / or text content 205.

[0053] Turn now Figure 3 A flowchart illustrating an example method 300 for managing multimedia content acquired by a Large Language Model (LLM) is provided. For convenience, the operation of method 300 is described with reference to a system performing these operations. This system of method 300 includes a computing device (e.g., Figure 1 Client device 110 Figure 1 Multimodal response system 120 Figure 7 The computing device 710, one or more servers, and / or other computing devices may include one or more processors, memories, and / or other components. Furthermore, although the operations of method 300 are shown in a specific order, this does not imply limitation. One or more operations may be reordered, omitted, and / or added.

[0054] At box 352, the system receives natural language (NL)-based input associated with a client device that requests multimedia content. For example, the NL-based input may be detected via a user input engine of the client device (e.g., via user input engine 111) and provided to the system via the client device. In some implementations, multimedia content may be explicitly requested by the NL-based input (e.g., “include pictures,” “include videos,” or the like). In other implementations, multimedia content may be implicitly requested by the NL-based input (e.g., based on a determined intent or context associated with the user providing the NL-based input).

[0055] At box 354, the system uses a large language model (LLM) to process the LLM input to generate LLM output, the LLM input including at least the NL-based input. In some implementations, the LLM input can be generated using an explicit LLM (e.g., via explicit LLM engine 141) that structures the NL-based input and / or other content (e.g., context, other cues besides the NL-based input, etc.) for LLM processing. In additional or alternative implementations, the LLM input can be the raw NL-based input received by the system. Furthermore, in some implementations, the LLM output can be generated using a conversational LLM (e.g., via conversational LLM engine 142) that is fine-tuned as described herein (e.g., with respect to fine-tuning engine 130).

[0056] At box 356, the system, based on the LLM output, determines at least the multimedia content to be included in the response to the NL-based input. It is noteworthy that the LLM, fine-tuned as described herein, can generate corresponding multimedia content tags (e.g., to be submitted to search system 180) and / or corresponding generative multimedia content prompts (e.g., to be submitted to generative system 190) for retrieving the multimedia content. Therefore, the system can determine the multimedia content to be included in the response to the NL-based input based on the corresponding multimedia content tags (e.g., by parsing the LLM output based on multimedia content tag engine 161 to identify the corresponding multimedia content tags) and / or the corresponding generative multimedia content prompts (e.g., by parsing the LLM output based on generative multimedia content prompt engine 162 to identify the corresponding generative multimedia content prompts). Additionally or alternatively, the system can determine, based on the LLM output, the text content to be included in the response to the NL-based input and to be included in the response along with the multimedia content (e.g., via text content engine 160). However, it should be noted that some responses may include multimedia content without any text content.

[0057] At box 358, the system obtains the multimedia content to be included in the response to the NL-based input. For example, in an implementation where the multimedia content to be included in the response includes non-generated multimedia content, corresponding multimedia content tags can be used to determine one or more non-generated multimedia content queries (e.g., via multimedia content tagging engine 161), and the non-generated multimedia content can be obtained based on the one or more non-generated multimedia content queries (e.g., by submitting the one or more non-generated multimedia content queries to search system 180 via multimedia content retrieval engine 164). As another example, in an implementation where the multimedia content to be included in the response includes generated multimedia content, corresponding generated multimedia content prompts can be used to determine one or more generated multimedia content prompts (e.g., via generated multimedia content prompting engine 162), one or more generative systems can be selected to process the one or more generated multimedia content prompts (e.g., via generated multimedia content model selection engine 163), and the generated multimedia content can be obtained based on the one or more generated multimedia content prompts (e.g., by submitting the one or more generated multimedia content prompts to generative system 190 via multimedia content retrieval engine 164).

[0058] At box 360, the system determines whether to include multimedia content in the response to be presented at the client device. The system may base this on, for example, the use of one or more evaluation models (e.g., via evaluation engine 171 and as described above). Figure 2 The generated evaluation output (described) determines whether multimedia content should be included in the response to be presented at the client device. In other words, even if the system is able to acquire multimedia content, it can determine whether the multimedia content is suitable to be included in the response (e.g., based on the evaluation output). If, at the iteration in box 360, the system determines that multimedia content should not be included in the response to be presented at the client device, the system proceeds to box 362.

[0059] At box 362, the system determines whether to include alternative multimedia content in the response to be presented at the client device. In other words, even if the system determines that the initially acquired multimedia content is not suitable for inclusion in the response, the system may attempt to acquire alternative multimedia content that is suitable for inclusion in the response (e.g., based on subsequent evaluation of the alternative multimedia content by evaluation engine 171). In some implementations, the system may be configured (e.g., by a developer associated with the multimedia content management system 120) to attempt to acquire alternative multimedia content. N This iteration retrieves alternative multimedia content suitable for inclusion in the response. It is worth noting that... NIt can be a fixed (e.g., defined by the developers associated with the multimedia content management system 120) or dynamic (e.g., based on one or more of the following: LLM lexical constraints, LLM temporal constraints, or LLM computational constraints) positive integer. In other implementations, the system may not attempt to fetch any alternative multimedia content, and the operation in box 362 may be omitted.

[0060] If, at the iteration in box 362, the system determines that alternative multimedia content should be included in the response to be presented at the client device, the system proceeds to box 364. At box 364, the system determines the alternative multimedia content to be included in the response to the NL-based input. In some implementations, the system may determine the alternative multimedia content based on the initially generated LLM output and at least a portion of the evaluation output (generated based on processing at least the initially acquired multimedia content), and / or the system may determine the alternative multimedia content based on additional LLM output generated based on processing the NL-based input and at least a portion of the evaluation output (generated based on processing at least the initially acquired multimedia content). It is noteworthy that the evaluation output may include indications regarding whether the multimedia content should be presented and / or indications regarding why the multimedia content was determined to be either or should not be presented. Therefore, by considering the evaluation output when determining the alternative multimedia content, the system should avoid determining alternative multimedia content that should not be presented for the same reasons as the initially acquired multimedia content. For example, in an implementation where alternative multimedia content is determined based on the initially generated LLM output and at least a portion of the evaluation output (generated based on processing at least the initially acquired multimedia content), the system can modify the one or more multimedia content queries and / or the one or more multimedia content prompts to ensure that they indeed elicit the same multimedia content as the initially acquired multimedia content. Similarly, for example, in an implementation where alternative multimedia content is determined based on additional LLM output—generated based on processing NL-based input and at least a portion of the evaluation output (generated based on processing at least the initially acquired multimedia content)—the system can cause the additional LLM output to be generated knowing why the multimedia content was determined to be either or should be presented, thereby including different non-generative multimedia content tags and / or generative multimedia content prompts in the additional LLM output.

[0061] At box 366, the system causes the response, which includes alternative multimedia content and replaces the multimedia content, to be presented at the client device. For example, the system may cause the response to be visually presented at the client device via the client device's display. The system returns to box 352 and may perform additional iterations of method 300 in response to receiving additional NL-based input associated with the client device (or additional client devices) at an additional iteration of box 352.

[0062] If, at the iteration in box 362, the system determines that alternative multimedia content should not be included in the response to be presented at the client device, the system proceeds to box 368. At box 368, the system determines the preset text content and / or other text content to be included in the response and in place of the multimedia content. In other words, if the system determines that the initially acquired multimedia content and / or any subsequently acquired alternative multimedia content is unsuitable for inclusion in the response (e.g., based on...), the system will proceed to box 368. N If the next iteration does not produce any alternative multimedia content, the system may pause any further attempts to acquire multimedia content and determine preset text content and / or other text content to be included in the response and in place of that multimedia content. The preset text content may indicate that the multimedia content could not be rendered and optionally provide the reason why the multimedia content could not be rendered (e.g., indicated by the evaluation output). Additionally, other text content may include text content determined based on the LLM output and / or any additional LLM output generated.

[0063] At box 370, the system causes the response, including preset text content and / or other text content and in place of the multimedia content, to be presented at the client device. For example, the system may cause the response to be visually presented at the client device via the client device's display. The system returns to box 352 and may perform additional iterations of method 300 in response to receiving additional NL-based input associated with the client device (or additional client devices) at an additional iteration of box 352.

[0064] If, at the iteration in box 360, the system determines that multimedia content should be included in the response to be presented at the client device, the system proceeds to box 372. At box 372, the system causes the response including the multimedia content to be presented at the client device. For example, the system may cause the response to be visually presented at the client device via its display. The system returns to box 352 and may perform additional iterations of method 300 in response to receiving additional NL-based input associated with the client device (or an additional client device) at an additional iteration in box 352.

[0065] Turn now Figure 4The diagram depicts a flowchart of another example method 400 for managing multimedia content acquired by a Large Language Model (LLM). For convenience, the operation of method 400 is described with reference to a system performing these operations. This system of method 400 includes a computing device (e.g., Figure 1 Client device 110 Figure 1 Multimodal response system 120 Figure 7 The computing device 710, one or more servers, and / or other computing devices may include one or more processors, memories, and / or other components. Furthermore, although the operations of method 400 are shown in a specific order, this does not imply limitation. One or more operations may be reordered, omitted, and / or added.

[0066] At box 452, the system receives natural language (NL)-based input associated with a client device, which requests multimedia content. At box 454, the system processes the LLM input using a Large Language Model (LLM) to generate an LLM output, the LLM input including at least the NL-based input. At box 456, based on the LLM output, the system determines at least the multimedia content to be included in the response to the NL-based input. Figure 4 The operations in boxes 452-456 of method 400 can be respectively related to the above-mentioned... Figure 3 The same or similar methods are performed as described in boxes 352-356 of method 300.

[0067] At box 458, the system initiates the retrieval of multimedia content to be included in the response to the NL-based input. For example, in an implementation where the multimedia content to be included in the response includes non-generated multimedia content, corresponding multimedia content tags can be used to determine one or more non-generated multimedia content queries (e.g., via multimedia content tagging engine 161), and the system can initiate the retrieval of non-generated multimedia content based on the one or more non-generated multimedia content queries (e.g., by submitting the one or more non-generated multimedia content queries to search system 180 via multimedia content retrieval engine 164). As another example, in an implementation where the multimedia content to be included in the response includes generative multimedia content, a corresponding generative multimedia content prompt can be used to determine one or more generative multimedia content prompts (e.g., via generative multimedia content prompt engine 162), one or more generative systems can be selected to process the one or more generative multimedia content prompts (e.g., via generative multimedia content model selection engine 163), and the system can initiate the retrieval of generative multimedia content based on the one or more generative multimedia content prompts (e.g., by submitting the one or more generative multimedia content prompts to generative system 190 via multimedia content retrieval engine 164).

[0068] At box 460, the system determines whether to continue acquiring multimedia content to be included in the response to the NL-based input. The system may determine whether to continue acquiring multimedia content to be included in the response to the NL-based input based on, for example, one or more signals. These one or more signals may include an NL-based input context relative to multimedia content requested by the NL-based input, a multimedia content context relative to the NL-based input, a response context relative to text content, and / or other signals.

[0069] In an implementation where one or more signals include an NL-based input context relative to multimedia content requested by the NL-based input, the NL-based input context can indicate whether the multimedia content requested by the NL-based input should be presented given the NL-based input. For example, given an NL-based input directed to a client device, there may not be multimedia content suitable for presentation in response to the NL-based input. In an implementation where one or more signals include a multimedia content context relative to the NL-based input, the multimedia content context can indicate whether the multimedia content requested by the NL-based input should be presented given the NL-based input. For example, given a corresponding multimedia content tag and / or corresponding multimedia content prompt included in the LLM output, there may not be multimedia content suitable for presentation in response to the NL-based input. In an implementation where one or more signals include a response context relative to text content given a multimedia content request based on NL-based input, the response context can indicate whether the multimedia content requested by the NL-based input should be presented given the NL-based input. For example, given a corresponding multimedia content tag and / or a corresponding multimedia content prompt included in the LLM output, and relative to other text content determined for inclusion in the response, there may be no multimedia content suitable for presentation in response to the NL-based input.

[0070] It is worth noting that, with Figure 3 Compared to method 300, which determines whether multimedia content has already been acquired, in the system's determination of whether such content is included... Figure 4 In method 400, the system can determine whether to continue acquiring multimedia content before it is acquired by the system. In other words, in Figure 4 In method 400, the system balances latency by initiating the retrieval of multimedia content, but also balances computational resource consumption by suspending any further steps in retrieving multimedia content if one or more signals indicate that any multimedia content retrieved based on the corresponding multimedia content tag and / or the corresponding multimedia content prompt should not be presented at the client device. Therefore, in Figure 4 In method 400, the system may not use the evaluation model after acquiring the multimedia content, but may instead cancel any multimedia content queries used to acquire the multimedia content and / or cancel any processing of generative multimedia content prompts by the generative multimedia content model to generate the multimedia content.

[0071] If, at the iteration in box 460, the system determines that it should not continue fetching multimedia content to be included in the response to the NL-based input, the system proceeds to box 462. At box 462, the system determines whether to fetch alternative multimedia content for the response. Although the system can determine not to continue fetching multimedia content based on one or more signals, it may still attempt to fetch alternative multimedia content if the one or more signals indicate the existence of alternative multimedia content suitable for presentation at the client device.

[0072] If, at the iteration in box 462, the system determines that alternative multimedia content is to be obtained for the response, the system proceeds to box 464. At box 464, the system obtains the alternative multimedia content to be included in the response to the NL-based input. The alternative multimedia content can be related to... Figure 3 The method described in box 364 of method 300 is to obtain the content in the same or similar manner. However, when obtaining alternative multimedia content, the system can begin execution from box 460. Figure 4 Another iteration of method 400 is used to determine whether to continue acquiring alternative multimedia content.

[0073] At box 466, and assuming the system determines to acquire alternative multimedia content, the system causes the response, including and replacing the alternative multimedia content, to be presented at the client device. For example, the system may cause the response to be visually presented at the client device via its display. The system returns to box 452 and may perform additional iterations of method 400 in response to receiving additional NL-based input associated with the client device (or additional client devices) at an additional iteration of box 452.

[0074] If, at the iteration in box 462, the system determines that no alternative multimedia content should be obtained for the response, the system proceeds to box 468. At box 468, the system determines the preset text content and / or other text content to be included in the response and in place of the multimedia content. At box 470, the system causes the response, including the preset text content and / or other text content and in place of the multimedia content, to be presented on the client device. The operations of boxes 468 and 470 can be related to the above descriptions. Figure 3 The method 300 is performed in the same or similar manner as described in blocks 368 and 370. The system returns to block 452 and may perform additional iterations of method 400 in response to receiving additional NL-based input associated with the client device (or additional client device) at an additional iteration in block 452.

[0075] If, at the iteration in box 460, the system determines to continue acquiring multimedia content to be included in the response to the NL-based input, the system proceeds to box 472. At box 472, the system acquires the multimedia content to be included in the response to the NL-based input. At box 474, the system causes the response including the multimedia content to be presented at the client device. For example, the system may cause the response to be visually presented at the client device via its display. The system returns to box 452 and may perform additional iterations of method 400 in response to receiving additional NL-based input associated with the client device (or an additional client device) at an additional iteration in box 452.

[0076] Although for the sake of brevity, it is not included Figure 4 Method 400 is described, but it should be understood that the system may additionally evaluate multimedia content and / or any alternative multimedia content after it has been acquired to determine whether it should be presented at the client device (e.g., as per [reference to...]). Figure 3 (as described in method 300). In these implementations, the system may further determine whether to acquire alternative multimedia content based on an evaluation of any acquired multimedia content.

[0077] Turn now Figure 5 The diagram depicts a flowchart of yet another example method 500 for managing multimedia content acquired by a Large Language Model (LLM). For convenience, the operation of method 500 is described with reference to a system performing these operations. This system of method 500 includes a computing device (e.g., Figure 1 Client device 110 Figure 1 Multimodal response system 120 Figure 7 The computing device 710, one or more servers, and / or other computing devices may include one or more processors, memories, and / or other components. Furthermore, although the operations of method 500 are shown in a specific order, this does not imply limitation. One or more operations may be reordered, omitted, and / or added.

[0078] At box 552, the system receives natural language (NL) based input associated with a client device, which requests multimedia content. Figure 5 The operation of box 552 in method 500 can be related to the above. Figure 3 The same or similar method is used to perform the action described in box 352 of method 300.

[0079] At box 554, the system determines whether to retrieve multimedia content based on the NL-based input. It is worth noting that, compared to... Figure 3 In method 300, the system determines whether to include already acquired multimedia content compared to... Figure 4 Compared to method 400, which determines whether to continue acquiring multimedia content, in... Figure 5 In method 500, the system can determine whether or not to initiate the retrieval of multimedia content. In other words, Figure 5 The system in method 500 may prevent any multimedia content retrieval from being initiated. For example, one or more words based on NL input may request multimedia content that cannot be obtained and / or generated, multimedia content that may have an illegal purpose, multimedia content that may endanger the security and / or privacy of the user and / or other users, and / or meet one or more other criteria that prevent the system from initiating any multimedia content retrieval.

[0080] If, at the iteration point of box 554, the system determines that multimedia content is to be acquired based on the NL-based input, then the system proceeds to... Figure 3 Method 300, box 354 and / or Figure 4 Method 400, box 454. In other words, and assuming the system determines that it needs to acquire multimedia content based on the NL-based input, the system can initiate... Figure 3 Method 300 and / or Figure 4 Method 400 is used to determine whether to continue acquiring multimedia content and / or whether any acquired multimedia content should be presented.

[0081] If, at the iteration in box 554, the system determines that multimedia content is not to be acquired based on the NL-based input, the system proceeds to box 556. In box 556, the system processes the LLM input using a Large Language Model (LLM) to generate an LLM output, the LLM input including at least the NL-based input. The operation in box 556 can be related to... Figure 3 The method described in box 354 of method 300 is performed in the same or similar manner. However, the LLM input may additionally include a prompt indicating that no multimedia content should be retrieved in response to the NL-based input.

[0082] At box 558, the system, based on the LLM output, determines preset text content and / or other text content to be included in the response to the NL-based input and in lieu of any multimedia content. At box 560, the system causes the response, including the preset text content and / or other text content and in lieu of the multimedia content, to be presented at the client device. The operations of boxes 558 and 560 can be related to the above descriptions. Figure 3 Method 300 is performed in the same or similar manner as described in blocks 368 and 370. The system returns to block 552 and may perform additional iterations of method 500 in response to receiving additional NL-based input associated with the client device (or additional client device) at an additional iteration in block 552.

[0083] Turn now Figure 6A , Figure 6B , Figure 6C and Figure 6D This describes various non-limiting examples of managing multimedia content acquired by a large language model (LLM). Client device 110 (e.g., from...) Figure 1 The client device 110 may include various user interface components, including, for example: a microphone for generating audio data based on spoken words and / or other audible input; a speaker for audibly presenting synthesized speech and / or other audible output; and / or a display 112A for visually presenting visual output. Furthermore, the display 112A of the client device 110 may include various system interface elements 112A1, 112A2, and 112A3 (e.g., hardware and / or software interface elements) with which a user of the client device 110 can interact to cause the client device 110 to perform one or more actions. The display 112A of the client device 110 enables the user to interact with the content presented on the display 112A via touch input (e.g., by pointing user input at the display 112A0 or a portion thereof (e.g., pointing to a text input box 112A4, pointing to a keyboard (not depicted), or pointing to another portion of the display 112A)) and / or via verbal input (e.g., by selecting a microphone interface element 112A5—or simply by speaking without having to select a microphone interface element 112A5 (i.e., the automation assistant can monitor one or more words or phrases, gestures, gaze, mouth movements, lip movements, and / or other conditions to activate verbal input at the client device 110)). Although Figure 6A , Figure 6B , Figure 6C and Figure 6D The client device 110 depicted is a mobile phone, but it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, the client device 110 may be a stand-alone speaker with a display, a stand-alone speaker without a display, a home automation device, an in-vehicle system, a laptop computer, a desktop computer, and / or any other device capable of performing an automated assistant to conduct human-computer dialogue sessions with the user of the client device 110.

[0084] Specific reference Figure 6A For the sake of illustration, suppose a user of client device 110 provides NL-based input 652A: "Generate an image of [particular person] riding a unicycle while eating an ice cream cone." Further suppose the system (e.g., from...) Figure 1The multimedia content management system 120) uses LLM (e.g., as described in the context of...) Figure 1 The finely tuned LLM described above processes at least the NL-based input 652A to generate an LLM output for a response including multimedia content (e.g., as described in...). Figures 2 to 5 (As described). For example, suppose the LLM output for the response includes at least text segment 654A that provides an overview of the generative image that will be presented as part of the response. Further suppose the LLM output for the response includes at least generative multimedia content cues 654A1 that will be used to generate the generative image that will be presented as part of the response.

[0085] Despite Figure 6A The generative multimedia content cues 654A1 are depicted in display 112A, but it should be understood that in various implementations, the generative multimedia content cues 654A1 may not be presented (e.g., visually and / or audibly) to the user, making them imperceptible to the user. Instead, the generative multimedia content cues 654A1 act as placeholders for insertion into the response once the generative multimedia content is acquired (and assuming the system determines it should be presented).

[0086] For example, refer to the specific Figure 6B Continuing with the example above, the generative multimedia content prompt 554A1 can be replaced with a generative image of the specific person as indicated by 654B1. It is worth noting that as text segment 654A is acquired by client device 110, and before the generative multimedia content is acquired, this text segment can be visually and / or audibly presented for use in displaying to the user. In other words, client device 110 can stream the text segment 654A as it is acquired, but leave room for inserting the generative multimedia content when it is acquired. This allows for a reduction in latency in presenting the response. Furthermore, an optional pause streaming element 656 can be provided, and when selected, any streaming of the response can be paused to further conserve computational resources if the user decides not to receive further responses.

[0087] Furthermore, in some implementations, generative multimedia content items may be presented together with an indication of the corresponding source of the generative multimedia content (e.g., a Uniform Resource Locator (URL)). Moreover, in some implementations, the generative multimedia content item (or the indication of its corresponding source) may be selectable, and when selected, the client device 110 may navigate (e.g., via a web browser or other applications accessible via application engine 115) to the corresponding generative model used to generate the generative multimedia content item. For example, if the user selects an image of a particular person, the client device 110 may navigate to an image generator for generating a generative image of that particular person (and optionally display a generative multimedia content prompt 654A1 for generating the generative image of that particular person).

[0088] although Figure 6A and 6B This description is relative to the fact that the multimedia content included in the response is generative multimedia content, but it should be understood that this is for illustrative purposes and is not intended to be limiting. In additional or alternative implementations, multimedia content may include non-generative multimedia content obtained via multimedia content tags associated with non-generative multimedia content items. Furthermore, although... Figure 6A and Figure 6B This is described in relation to the acquisition of requested multimedia content in NL-based input 652A, but it should be understood that this is for illustrative purposes and not intended to be limiting. Rather, for illustrative purposes only, in Figure 6A and Figure 6B In this context, we assume the system (e.g., from...) Figure 1 The multimedia content management system 120 determines that the multimedia content obtained based on the generative multimedia content prompt 654A1 should be presented at the client device 110. In making this determination, the system may utilize various techniques described herein (e.g., regarding...). Figures 2 to 5 In other words, in Figure 6A and Figure 6B The multimedia content requested in the example will not jeopardize the data security of the user or other users, will not jeopardize the data privacy of the user or other users, will not be used for illegal purposes, and / or should not be omitted from presentation for any other reason.

[0089] However, and specifically refer to Figure 6CFor the sake of illustration, suppose the user of client device 110 alternatively provides NL-based input 652C: "Show me an image of [particular person]'s face from each of the following angles: front, left side, right side, head tilted up, headtilted down." Further suppose the system (e.g., from...) Figure 1 The multimedia content management system 120) uses LLM (e.g., as described in the context of...) Figure 1 The finely tuned LLM described above processes at least the NL-based input 652C to generate an LLM output for a response that includes multimedia content (e.g., as described in...). Figures 2 to 5 (As described). For example, suppose the LLM output for the response includes at least text segment 654C, which provides an overview of the requested image that will be presented as part of the response. Further suppose the LLM output for the response includes at least non-generative multimedia content tags 654C1, 654C2, 654C3, 654C4, and 654C5 for each of the requested images, which will be used to obtain the non-generative images that will be presented as part of the response.

[0090] Despite Figure 6C Non-generated multimedia content tags 654C1, 654C2, 654C3, 654C4, and 654C5 are depicted in display 112A. However, it should be understood that in various implementations, non-generated multimedia content tags 654C1, 654C2, 654C3, 654C4, and 654C5 may not be presented (e.g., visually and / or audibly) to the user, making them imperceptible to the user. Instead, non-generated multimedia content tags 654C1, 654C2, 654C3, 654C4, and 654C5 act as placeholders for the location where non-generated multimedia content will be inserted into the response once it is acquired (and assuming the system determines it should be presented).

[0091] For example, refer to the specific Figure 6D Continuing with the example above, the system can determine that non-generative multimedia content requested by the user should not be retrieved. For example, based on the evaluation output (e.g., regarding...). Figure 2 and Figure 3 (as described) and / or based on one or more signals (e.g., such as regarding) Figure 4As described, the system can determine that a user is requesting an image for malicious purposes, such as using the requested image to replicate a facial embedding of a specific person. This could potentially be used to facilitate fraudulent attempts to compromise biometric security systems (e.g., facial recognition systems). Therefore, in this example, even if the system is able to determine based on... Figure 6C The non-generative multimedia content tags 654C1, 654C2, 654C3, 654C4, and 654C5 depicted in the diagram obtain the requested image. The system can alternatively determine and present alternative text content as indicated by 654D, and present some alternative multimedia content, such as a single image of the specific person as indicated by 654D1. Therefore, the system can mitigate and / or eliminate the presentation of multimedia content when it is determined that it may jeopardize the user's data security and / or privacy, and / or when it is determined that it may be used to facilitate illegal activities, while doing so in a computationally efficient manner.

[0092] Turn now Figure 7 The diagram depicts a block diagram of an example computing device 710 that can be optionally used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a multimodal response system component, or other cloud-based software application component and / or other components may include one or more components of the example computing device 710.

[0093] Computing device 710 typically includes at least one processor 714 that communicates with a plurality of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724 (including, for example, a memory subsystem 725 and a file storage subsystem 726), a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow users to interact with computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0094] User interface input device 722 may include a keyboard, pointing device (such as a mouse, trackball, touchpad, or graphics tablet), scanner, touchscreen integrated into a display, audio input device (such as a voice recognition system, microphone), and / or other types of input device. Generally, the use of the term "input device" is intended to include all possible types of means and methods for inputting information into computing device 710 or into a communication network.

[0095] User interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may further provide non-visual displays, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of means and methods for outputting information from computing device 710 to a user or another machine or computing device.

[0096] Storage subsystem 724 stores the functional programming and data structures for some or all of the modules described herein. For example, storage subsystem 724 may include selected aspects for performing the methods disclosed herein and for implementing... Figure 1 and Figure 2 The logic of the various components described.

[0097] These software modules are typically executed by processor 714 alone or in conjunction with other processors. The memory 725 used in storage subsystem 724 may include multiple memories, including main random access memory (RAM) 730 for storing instructions and data during program execution and read-only memory (ROM) 732 for storing fixed instructions therein. File storage subsystem 726 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing certain functionalities may be stored by file storage subsystem 726 within storage subsystem 724 or in other machines accessible by processor 714.

[0098] The bus subsystem 712 provides a mechanism for enabling the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative implementations of the bus subsystem 712 may use multiple buses.

[0099] The computing device 710 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, [the following applies]. Figure 7 The description of the computing device 710 is intended only as a specific example for illustrating some implementation methods. Many other configurations of the computing device 710 are possible, and these configurations are related to… Figure 7 The computing device depicted has more or fewer components compared to the one described.

[0100] In situations where the systems described herein collect or otherwise monitor personal information about users, or may utilize personal information and / or monitored information, users may be provided with the opportunity to control whether programs or features collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from content servers. Additionally, certain data may be processed in one or more ways before it is stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed to the point that personally identifiable information about the user cannot be determined, or, in the case of obtaining geographic location information, the user's geographic location may be generalized (e.g., to the city, zip code, or state level) to the point that the user's specific geographic location cannot be determined. Therefore, users can control how information about themselves is collected and / or used.

[0101] In some implementations, a method implemented by one or more processors is provided, comprising: receiving natural language (NL)-based input associated with a user's client device, the NL-based input requesting multimedia content; and generating a response in response to the NL-based input. Generating the response in response to the NL-based input includes: processing the LLM input using a large language model (LLM) to generate an LLM output, the LLM input including at least the NL-based input; determining, based on the LLM output, multimedia content to be included in the response to the NL-based input; acquiring the multimedia content to be included in the response to the NL-based input; determining, based on processing the NL-based input and the multimedia content to be included in the response to the NL-based input, whether to allow a response including the multimedia content to be presented at the client device; and in response to determining that the response should not be presented at the client device: determining alternative multimedia content to be included in the response to the NL-based input instead of the multimedia content. The method further includes: allowing a response including the alternative multimedia content to be presented at the user's client device.

[0102] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0103] In some implementations, processing NL-based input and multimedia content to be included in a response to NL-based input may include: using a Visual Language Model (VLM) to process the NL-based input and multimedia content to be included in a response to NL-based input to generate VLM output; and based on the VLM output, determining whether to allow the response including multimedia content to be presented at the client device.

[0104] In some versions of those implementations, the method may further include: using a VLM to process prompts along with NL-based input and multimedia content to generate VLM output. The prompts may include requests from the VLM to determine whether multimedia content should be presented on a user's client device given NL-based input.

[0105] In some further versions of those implementations, the VLM output may include an indication of whether multimedia content should be presented on the user's client device given NL-based input.

[0106] In some further versions of those implementations, the VLM output may further include reasons regarding whether multimedia content should be presented on the user's client device given NL-based input.

[0107] In some additional or alternative versions of those implementations, determining that the response should not be presented on the client device can be based on VLM output including an indication that multimedia content should not be presented on the user's client device.

[0108] In some, and even further, versions of those implementations, determining the alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content may include: using LLM processing of additional LLM input to generate additional LLM output, the additional LLM input including at least the NL-based input and at least a portion of the VLM output; and determining the alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content based on the additional LLM output.

[0109] In some implementations, processing NL-based input and multimedia content to be included in a response to the NL-based input may include: processing the multimedia content using a caption model to generate one or more corresponding captions for the multimedia content; processing the NL-based input and one or more corresponding captions for the multimedia content using an LLM to generate an evaluation LLM output; and determining, based on the evaluation LLM output, whether a response including the multimedia content should be presented at the client device.

[0110] In some versions of those implementations, the method may further include: using an LLM processing prompt along with NL-based input and one or more corresponding annotations of the multimedia content to generate an evaluation LLM output. The prompt may include a request for the LLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

[0111] In some further versions of those implementations, evaluating LLM output can include an indication of whether multimedia content should be presented on the user's client device given NL-based input.

[0112] In some further versions of those implementations, evaluating LLM output can further include reasons about whether multimedia content should be presented on the user's client device given NL-based input.

[0113] In some additional or alternative versions of those implementations, determining that the response should not be presented on the client device can be based on an assessment of the LLM output, including instructions that multimedia content should not be presented on the user's client device.

[0114] In some, and even further, versions of those implementations, determining the alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content may include: using LLM processing of additional LLM input to generate additional LLM output, the additional LLM input including at least the NL-based input and evaluating at least a portion of the LLM output; and based on the additional LLM output, determining the alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content.

[0115] In additional or alternative versions of those implementations, the LLM output and the evaluation of the LLM output can be generated using a single call to the LLM.

[0116] In some implementations, processing NL-based input and multimedia content to be included in a response to the NL-based input may include: obtaining one or more corresponding multimedia content tags stored in association with the multimedia content; processing the NL-based input and one or more corresponding multimedia content tags using LLM to generate an evaluation LLM output; and determining, based on the evaluation LLM output, whether to allow the response including the multimedia content to be presented at the client device.

[0117] In some versions of those implementations, the method may further include: using an LLM processing prompt, and together with NL-based input and one or more corresponding multimedia content tags for the multimedia content, to generate an evaluation LLM output. The prompt may include a request for the LLM to determine whether multimedia content should be presented on the user's client device given NL-based input.

[0118] In some versions of those implementations, evaluating LLM output can include indications about whether multimedia content should be presented on the user's client device and given NL-based input.

[0119] In some further versions of those implementations, evaluating LLM output can further include reasons regarding whether multimedia content should be presented at the user's client device and given NL-based input.

[0120] In additional or alternative versions of those implementations, determining that the response should not be presented on the client device can be based on an assessment of the LLM output, including indications that multimedia content should not be presented on the user's client device.

[0121] In some further versions of those implementations, determining the alternative multimedia content to be included in the response to the NL-based input and in place of the multimedia content may include: using LLM processing of additional LLM input to generate additional LLM output, the additional LLM input including at least the NL-based input and evaluating at least a portion of the LLM output; and determining, based on the additional LLM output, the alternative multimedia content to be included in the response to the NL-based input and in place of the multimedia content.

[0122] In some implementations, the multimedia content to be included in the response to the NL-based input may be non-generative multimedia content, and determining the multimedia content to be included in the response to the NL-based input may include: determining that the LLM output includes a multimedia content tag indicating that the non-generative multimedia content to be included in the response to the NL-based input.

[0123] In some versions of those implementations, obtaining the multimedia content to be included in the response to an NL-based input may include: determining a multimedia content query based on multimedia content tags; submitting the multimedia content query to a multimedia content search system; and receiving the multimedia content from the multimedia content search system.

[0124] In some implementations, the multimedia content to be included in the response to the NL-based input may be generative multimedia content, and determining the multimedia content to be included in the response to the NL-based input may include: determining that the LLM output includes a multimedia content prompt indicating that the generative multimedia content to be included in the response to the NL-based input.

[0125] In some versions of those implementations, obtaining multimedia content to be included in the response to NL-based input may include: submitting the multimedia content prompt to a generative multimedia content system; and receiving multimedia content from a multimedia content search system.

[0126] In some implementations, the method may further include: determining, based on the LLM output, text content that, in addition to multimedia content and alternative media content, should also be included in the response to the NL-based input.

[0127] In some versions of those implementations, the text context may include multiple text segments, and the presentation of a response including text content and alternative multimedia content at the user's client device may include: having the multimedia content interleaved between a first text segment and a second text segment of the multiple text segments on the display of the client device.

[0128] In additional or alternative implementations, determining whether to allow a response including multimedia content to be presented at the client device can be further based on: processing text content that is also included in the response to NL-based input, together with the NL-based input and multimedia content.

[0129] In some implementations, a method implemented by one or more processors is provided, comprising: receiving natural language (NL)-based input associated with a user's client device, the NL-based input requesting multimedia content; and generating a response in response to the NL-based input. Generating the response in response to the NL-based input includes: processing the LLM input using a large language model (LLM) to generate an LLM output, the LLM input including at least the NL-based input; determining, based on the LLM output, multimedia content to be included in the response to the NL-based input; acquiring the multimedia content to be included in the response to the NL-based input; determining, based on processing the NL-based input and the multimedia content to be included in the response to the NL-based input, whether to allow the response including the multimedia content to be presented at the client device; and in response to determining that the response should not be presented at the client device: determining preset text content and / or other text content to be included in the response to the NL-based input and in lieu of the multimedia content. The method further includes: allowing the response including the preset text content to be presented at the user's client device.

[0130] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0131] In some implementations, the method may further include: determining whether there is alternative multimedia content to be included in the response to the NL-based input and in place of the multimedia content, and in response to determining that there is no alternative multimedia content to be included in the response to the NL-based input and in place of the multimedia content: determining preset text content and / or other text content to be included in the response to the NL-based input and in place of the multimedia content.

[0132] In some implementations, determining whether there exists alternative multimedia content to be included in the response to the NL-based input and in lieu of the multimedia content may include: for N The next iteration, in which... N For positive integers: Use LLM processing to generate corresponding additional LLM inputs to produce corresponding additional LLM outputs, the corresponding additional LLM inputs including at least NL-based inputs and corresponding evaluation outputs; and based on the corresponding additional LLM outputs, determine whether there are any corresponding alternative multimedia content to be included in the response to the NL-based inputs and in place of the multimedia content.

[0133] In some versions of those implementations, the corresponding evaluation output may include an indication of whether multimedia content should be presented on the user's client device given NL-based input, and / or may include reasons for whether multimedia content should be presented on the user's client device given NL-based input.

[0134] In some further versions of those implementations, determining that there is no alternative multimedia content to be included in the response to NL-based input and replacing the multimedia content can be in response to determining that there is no corresponding alternative multimedia content that should be presented at the user's client device given the NL-based input.

[0135] In additional or alternative versions of those implementations, the corresponding evaluation output may be generated based on processing NL-based input using a Visual Language Model (VLM), multimedia content to be included in the response to the NL-based input, and prompts including requests for the VLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

[0136] In additional or alternative versions of those implementations, the corresponding evaluation output may be generated based on LLM processing of NL-based input, multimedia content to be included in the response to the NL-based input, one or more corresponding captions for the multimedia content generated using a caption model, and prompts including requests for LLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

[0137] In additional or alternative versions of those implementations, the corresponding evaluation output may be generated based on LLM processing of NL-based input, multimedia content to be included in the response to the NL-based input, one or more corresponding multimedia content tags stored in association with the multimedia content, and a prompt including a request for LLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

[0138] In those additional or alternative versions of implementation, N It can be a dynamic integer, which is based on one or more of the following: LLM lexical constraints, LLM time constraints, or LLM computational constraints.

[0139] In those additional or alternative versions of implementation methods N It can be a fixed integer.

[0140] In some implementations, preset text content and / or other text content can indicate that the LLM is unable to retrieve and / or generate multimedia content requested by NL-based input.

[0141] In some versions of those implementations, the preset text content and / or other text content can be determined based on the LLM output or the corresponding additional LLM output generated using the LLM.

[0142] In additional or alternative versions of those implementations, the default text content and / or other text content is obtained from one or more databases accessible to the client device.

[0143] In some implementations, a method implemented by one or more processors is provided, comprising: receiving natural language (NL)-based input associated with a user's client device, the NL-based input requesting multimedia content; and generating a response in response to the NL-based input. Generating the response in response to the NL-based input includes: processing the LLM input using a large language model (LLM) to generate an LLM output, the LLM input including at least the NL-based input; determining, based on the LLM output, text content and multimedia content to be included in the response to the NL-based input; initiating retrieval of the multimedia content to be included in the response to the NL-based input; while retrieving the multimedia content to be included in the response to the NL-based input: determining, based on one or more signals, whether to continue retrieving the multimedia content to be included in the response to the NL-based input; and in response to determining that retrieval of the multimedia content to be included in the response to the NL-based input should not continue: stopping the retrieval of the multimedia content; and determining preset text content and / or other text content to be included in the response to the NL-based input and in place of the multimedia content. The method further includes: causing a response, including preset text content and / or other text content, to be presented on the user's client device.

[0144] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0145] In some implementations, the one or more signals may include one or more of the following: an NL-based input context relative to multimedia content requested by the NL-based input, a multimedia content context relative to the NL-based input, or a response context relative to text content, given multimedia content requested by the NL-based input.

[0146] In some versions of those implementations, one or more signals may include an NL-based input context relative to multimedia content requested by the NL-based input, given an NL-based input, and the NL-based input context may indicate whether the multimedia content requested by the NL-based input should be presented given the NL-based input.

[0147] In additional or alternative versions of those implementations, the one or more signals may include a multimedia content context relative to the NL-based input, given a multimedia content request by an NL-based input, and the multimedia content context may indicate whether the multimedia content requested by the NL-based input should be presented, given a generative multimedia content prompt for generating the multimedia content or a non-generative multimedia content query for retrieving the multimedia content.

[0148] In additional or alternative versions of those implementations, the one or more signals may include a response context relative to text content given multimedia content requested by an NL-based input, and the response context may indicate whether the multimedia content requested by the NL-based input should be presented given text content determined to be included in the response.

[0149] In some implementations, the multimedia content can be non-generated multimedia content, and stopping the retrieval of multimedia content can include canceling the non-generated multimedia content query used to retrieve the multimedia content.

[0150] In some implementations, the multimedia content can be generative multimedia content, and stopping the acquisition of multimedia content can include: canceling the generative multimedia content prompts processed by the generative multimedia content model to generate multimedia content.

[0151] In some implementations, a method implemented by one or more processors is provided, comprising: receiving natural language (NL)-based input associated with a user's client device, the NL-based input requesting multimedia content; determining, based on one or more words included in the NL-based input, not to include the multimedia content in a response to the multimedia content; and generating a response to the NL-based input. Generating the response to the NL-based input includes: processing the LLM input using a large language model (LLM) to generate an LLM output, the LLM input including at least the NL-based input; and determining, based on the LLM output, text content to be included in the response to the NL-based input and in lieu of the multimedia content. The method further includes: causing the response including the text content and in lieu of the multimedia content to be presented at the user's client device.

[0152] Furthermore, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associative memory, and wherein the instructions are configured to cause any of the methods described above to be performed. Some implementations also include one or more computer-readable storage media (e.g., transient and / or non-transient) storing computer instructions executable by the one or more processors to perform any of the methods described above. Some implementations also include a computer program product comprising instructions executable by the one or more processors to perform any of the methods described above.

Claims

1. A method implemented by one or more processors, the method comprising: Receive natural language (NL)-based input associated with a user's client device, the NL-based input requesting multimedia content; Generate a response to the NL-based input, wherein generating the response to the NL-based input includes: Large language model LLM is used to process LLM inputs to generate LLM outputs, wherein the LLM inputs include at least the NL-based inputs; Based on the LLM output, determine the multimedia content to be included in the response to the NL-based input; Obtain the multimedia content to be included in the response to the NL-based input; Based on processing the NL-based input and the multimedia content to be included in the response to the NL-based input, determine whether the response including the multimedia content should be presented at the client device; and In response to determining that the response should not be presented at the client device: Determine alternative multimedia content to be included in the response to the NL-based input, replacing the multimedia content; and This causes the response, including the alternative multimedia content, to be presented on the user's client device.

2. The method as described in claim 1, wherein, Processing the NL-based input and the multimedia content to be included in the response to the NL-based input includes: The Visual Language Model (VLM) is used to process the NL-based input and the multimedia content to be included in the response to the NL-based input to generate VLM output; and Based on the VLM output, it is determined whether the response, including the multimedia content, should be presented at the client device.

3. The method of claim 2, further comprising: The VLM process prompts, along with the NL-based input and the multimedia content, are used to generate the VLM output, wherein the prompts include a request from the VLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

4. The method of claim 3, wherein, The VLM output includes an indication of whether the multimedia content should be presented on the user's client device given the NL-based input.

5. The method of claim 4, wherein, The VLM output further includes reasons regarding whether the multimedia content should be presented on the user's client device given the NL-based input.

6. The method as described in claim 4 or 5, wherein, The determination not to present the response on the client device is based on the VLM output including an indication that the multimedia content should not be presented on the user's client device.

7. The method of claim 6, wherein, The alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content includes: The LLM is used to process additional LLM inputs to generate additional LLM outputs, the additional LLM inputs including at least the NL-based inputs and at least a portion of the VLM outputs; and Based on the additional LLM output, the alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content is determined.

8. The method as described in any of the preceding claims, wherein, Processing the NL-based input and the multimedia content to be included in the response to the NL-based input includes: The multimedia content is processed using a caption model to generate one or more corresponding captions for the multimedia content; The LLM is used to process the NL-based input and one or more corresponding annotations of the multimedia content to generate an evaluation LLM output; and Based on the evaluation LLM output, it is determined whether the response, including the multimedia content, should be presented at the client device.

9. The method of claim 8, further comprising: The LLM processing prompt, along with the NL-based input and one or more corresponding illustrations of the multimedia content, is used to generate the evaluation LLM output, wherein the prompt includes a request for the LLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

10. The method of claim 9, wherein, The evaluation LLM output includes an indication of whether the multimedia content should be presented on the user's client device given the NL-based input.

11. The method of claim 10, wherein, The evaluation LLM output further includes reasons regarding whether the multimedia content should be presented on the user's client device given the NL-based input.

12. The method of claim 10 or 11, wherein, The determination to prevent the response from being presented on the client device is based on the evaluation LLM output including an indication that the multimedia content should not be presented on the user's client device.

13. The method of claim 12, wherein, The alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content includes: The LLM process is used to generate additional LLM outputs, the additional LLM inputs including at least the NL-based inputs and at least a portion of the evaluated LLM outputs; and Based on the additional LLM output, the alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content is determined.

14. The method according to any one of claims 8 to 13, wherein, The LLM output and the evaluated LLM output are generated using a single call to the LLM.

15. The method as claimed in any of the preceding claims, wherein, Processing the NL-based input and the multimedia content to be included in the response to the NL-based input includes: Obtain one or more corresponding multimedia content tags stored in association with the multimedia content; The LLM is used to process the NL-based input and one or more of the corresponding multimedia content tags to generate an evaluation LLM output; and Based on the evaluation LLM output, it is determined whether the response, including the multimedia content, should be presented at the client device.

16. The method of claim 15, further comprising: The LLM processing prompt, together with the NL-based input and one or more of the corresponding multimedia content tags for the multimedia content, is used to generate the evaluation LLM output, wherein the prompt includes a request for the LLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

17. The method of claim 16, wherein, The evaluation LLM output includes an indication of whether the multimedia content should be presented on the user's client device given the NL-based input.

18. The method of claim 17, wherein, The evaluation LLM output further includes reasons regarding whether the multimedia content should be presented on the user's client device given the NL-based input.

19. The method of claim 17 or 18, wherein, The determination to prevent the response from being presented on the client device is based on the evaluation LLM output including an indication that the multimedia content should not be presented on the user's client device.

20. The method of claim 19, wherein, The alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content includes: The LLM process is used to generate additional LLM outputs, the additional LLM inputs including at least the NL-based inputs and at least a portion of the evaluated LLM outputs; and Based on the additional LLM output, the alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content is determined.

21. The method as claimed in any of the preceding claims, wherein, The multimedia content to be included in the response to the NL-based input is non-generative multimedia content, and wherein determining the multimedia content to be included in the response to the NL-based input includes: The LLM output is determined to include multimedia content tags indicating that the non-generative multimedia content should be included in the response to the NL-based input.

22. The method of claim 21, wherein, Retrieving the multimedia content to be included in the response to the NL-based input includes: Based on the multimedia content tags, determine the multimedia content query; Submit the multimedia content query to the multimedia content search system; and Receive the multimedia content from the multimedia content search system.

23. The method as described in any of the preceding claims, wherein, The multimedia content to be included in the response to the NL-based input is generative multimedia content, and wherein determining the multimedia content to be included in the response to the NL-based input includes: The LLM output is determined to include a multimedia content prompt indicating that the generative multimedia content should be included in the response to the NL-based input.

24. The method of claim 23, wherein, Retrieving the multimedia content to be included in the response to the NL-based input includes: Submit the multimedia content prompt to the generative multimedia content system; and Receive the multimedia content from the multimedia content search system.

25. The method of any of the preceding claims, further comprising: Based on the LLM output, determine the text content that, in addition to the multimedia content and the alternative media content, should also be included in the response to the NL-based input.

26. The method of claim 25, wherein, The text context includes multiple text segments, and wherein, such that the response, including the text content and the alternative multimedia content, is presented at the user's client device includes: This results in the multimedia content being interleaved on the display of the client device between the first text segment and the second text segment of the plurality of text segments.

27. The method of claim 25 or 26, wherein, Determining whether to allow the response including the multimedia content to be presented at the client device is further based on: processing also including the text content in the response to the NL-based input, together with the NL-based input and the multimedia content.

28. A method implemented by one or more processors, the method comprising: Receive natural language (NL)-based input associated with a user's client device, the NL-based input requesting multimedia content; Generate a response to the NL-based input, wherein generating the response to the NL-based input includes: Large language model LLM is used to process LLM inputs to generate LLM outputs, wherein the LLM inputs include at least the NL-based inputs; Based on the LLM output, determine the multimedia content to be included in the response to the NL-based input; Obtain the multimedia content to be included in the response to the NL-based input; Based on processing the NL-based input and the multimedia content to be included in the response to the NL-based input, determine whether the response including the multimedia content should be presented at the client device; and In response to determining that the response should not be presented at the client device: Determine the preset text content and / or other text content to be included in the response to the NL-based input in place of the multimedia content; and This causes the response, including the preset text content, to be presented on the user's client device.

29. The method of claim 28, further comprising: Determine whether there is alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content, and In response to determining that there is no alternative multimedia content to be included in the response to the NL-based input, in place of the multimedia content: The preset text content and / or the other text content to be included in the response to the NL-based input in place of the multimedia content are determined.

30. The method of claim 28 or 29, wherein, Determining whether there is alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content includes: for N The next iteration, in which... N For positive integers: The LLM process is used to generate corresponding additional LLM outputs, wherein the corresponding additional LLM inputs include at least the NL-based inputs and the corresponding evaluation outputs; and Based on the corresponding additional LLM output, determine whether there is any corresponding alternative multimedia content to be included in the response to the NL-based input in place of the multimedia content.

31. The method of claim 30, wherein, The corresponding evaluation output includes an indication of whether the multimedia content should be presented on the user's client device given the NL-based input, and / or includes a reason why the multimedia content should be presented on the user's client device given the NL-based input.

32. The method of claim 31, wherein, Determining that there is no alternative multimedia content to be included in the response to the NL-based input is in response to determining that there is no corresponding alternative multimedia content that should be presented on the user's client device given the NL-based input.

33. The method of claim 31 or 32, wherein, The corresponding evaluation output is generated based on processing the NL-based input using a Visual Language Model (VLM), the multimedia content to be included in the response to the NL-based input, and a prompt, the prompt including a request from the VLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

34. The method according to any one of claims 31 to 33, wherein, The corresponding evaluation output is generated based on the LLM processing of the NL-based input, the multimedia content to be included in the response to the NL-based input, one or more corresponding captions for the multimedia content generated using a caption model, and a prompt, the prompt including a request for the LLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

35. The method according to any one of claims 31 to 34, wherein, The corresponding evaluation output is generated based on one or more of the following: the multimedia content to be included in the response to the NL-based input using the LLM; the corresponding multimedia content tags stored in association with the multimedia content; and a prompt, which includes a request for the LLM to determine whether the multimedia content should be presented on the user's client device given the NL-based input.

36. The method according to any one of claims 30 to 35, wherein, N The variable is a dynamic integer, which is based on one or more of the following: the lexical constraints of the LLM, the time constraints of the LLM, or the computational constraints of the LLM.

37. The method according to any one of claims 30 to 36, wherein, N It is a fixed integer.

38. The method according to any one of claims 28 to 37, wherein, The preset text content and / or the other text content indicate that the LLM is unable to acquire and / or generate the multimedia content requested by the NL-based input.

39. The method of claim 38, wherein, The preset text content and / or the other text content are determined based on the LLM output or the corresponding additional LLM output generated using the LLM.

40. The method of claim 38, wherein, The preset text content and / or the other text content are obtained from one or more databases accessible to the client device.

41. A method implemented by one or more processors, the method comprising: Receive natural language (NL)-based input associated with a user's client device, the NL-based input requesting multimedia content; Generate a response to the NL-based input, wherein generating the response to the NL-based input includes: Large language model LLM is used to process LLM inputs to generate LLM outputs, wherein the LLM inputs include at least the NL-based inputs; Based on the LLM output, determine the text content and multimedia content to be included in the response to the NL-based input; Initiate the retrieval of the multimedia content to be included in the response to the NL-based input; While acquiring the multimedia content to be included in the response to the NL-based input: Based on one or more signals, determine whether to continue acquiring the multimedia content to be included in the response to the NL-based input; and In response to determining that acquisition of the multimedia content to be included in the response to the NL-based input will not continue: Stop acquiring the multimedia content; and Determine the preset text content and / or other text content to be included in the response to the NL-based input in place of the multimedia content; and This causes the response, including the preset text content and / or the other text content, to be presented on the user's client device.

42. The method of claim 41, wherein, The one or more signals include one or more of the following: an NL-based input context relative to the multimedia content requested by the NL-based input, a multimedia content context relative to the NL-based input given the multimedia content requested by the NL-based input, or a response context relative to the text content given the multimedia content requested by the NL-based input.

43. The method of claim 42, wherein, The one or more signals include the NL-based input context relative to the multimedia content requested by the NL-based input, given the NL-based input, and wherein the NL-based input context indicates whether the multimedia content requested by the NL-based input should be presented given the NL-based input.

44. The method of claim 42 or 43, wherein, The one or more signals include a multimedia content context relative to the NL-based input given the multimedia content requested by the NL, and wherein the multimedia content context indicates whether the multimedia content requested by the NL should be presented given a generative multimedia content prompt for generating the multimedia content or a non-generative multimedia content query for retrieving the multimedia content.

45. The method according to any one of claims 42 to 44, wherein, The one or more signals include the response context relative to the text content given the multimedia content requested by the NL-based input, and wherein the response context indicates whether the multimedia content requested by the NL-based input should be presented given the text content determined to be included in the response.

46. ​​The method according to any one of claims 41 to 45, wherein, The multimedia content is non-generated multimedia content, and stopping the acquisition of the multimedia content includes: Cancel the non-generative multimedia content query used to obtain the multimedia content.

47. The method according to any one of claims 41 to 45, wherein, The multimedia content is generative multimedia content, and stopping the acquisition of the multimedia content includes: Cancel the prompt for processing generative multimedia content by the generative multimedia content model to generate the multimedia content.

48. A method implemented by one or more processors, the method comprising: Receive natural language (NL)-based input associated with a user's client device, the NL-based input requesting multimedia content; Based on one or more words included in the NL-based input, it is determined that the multimedia content will not be included in the response to the multimedia content; Generating a response to the NL-based input, wherein generating a response to the NL-based input includes: Large language model (LLM) is used to process LLM inputs to generate LLM outputs, wherein the LLM inputs include at least the NL-based inputs; and Based on the LLM output, determine the text content to be included in the response to the NL-based input in place of the multimedia content; and This causes the response, which includes the text content instead of the multimedia content, to be presented on the user's client device.

49. A system comprising: One or more hardware processors; as well as A memory that stores instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform the method as described in any one of claims 1 to 48.

50. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform the operation of the method as described in any one of claims 1 to 48.

Images