Supplemental content and generative language models

Abstract

The computing system includes a processor and memory that stores instructions, if executed by the processor, that cause the processor to perform certain actions. The actions include prompting a generative language model, which generates output based on the prompt, identifies text in the output that should be associated with supplemental content items, and assigns hyperlinks to the text in the output. The system displays the supplemental content items when the hyperlinks are selected or paused over.

Description

Background Art

[0001] Background Supplementary content often presents the main content on a web page. For example, a search engine results page (SERP) typically includes the received query, knowledge cards showing information about the entities referenced in the query, instant answers that respond to the questions presented in the query, and hyperlinks to web pages (or web applications) identified as related to the supplementary content. In some embodiments, the supplementary content is sponsored search results. In another example, a publisher issues content to a web page. The web page may include supplementary content related to the content of the web page and / or related to the interests of the viewers of the web page.

[0002] More recently, generative models, including generative language models (GLMs) (also referred to as large language models (LLMs)), have been developed. An example of a GLM is Generative Pretrained Transformer 4 (GPT-4). Another example of a GLM is the BigScience Language Open Science Open Access Multilingual (BLOOM) model, which is also a transformer-based model. Briefly, a generative model is configured to generate an output (e.g., text in human language, source code, music, video, etc.) based on a prompt, which typically includes the input presented by the user. The generative model generates the output almost in real time (e.g., within seconds after receiving the prompt). Further, the generative model generates the output based on the training data on which the generative model has been trained.

[0003] Currently, the practical applications of generative models are quite limited. For example, users of generative models often request that they generate poetry, provide biographical information about celebrities, or provide users with information on topics of interest. Generative models are not used in relation to improving the provision of supplemental content to search engine users and / or web page visitors. Furthermore, technologies configured to provide supplemental content to end users have not been meaningfully integrated with generative models. [Overview of the project] [Means for solving the problem]

[0004] overview The following is an overview of the subject matter, which is described in more detail here. This overview is not intended to limit the scope of the claims.

[0005] This document describes various techniques for using generative models in relation to providing supplemental content to viewers of web pages (or web applications). As used herein, the term “web page” is intended to include both traditional web pages and web applications. The computing environment described herein includes a computing system and client computing devices communicating with the computing system via a network. The computing system runs a generative model and a supplemental content delivery system. Furthermore, the computing system optionally runs a search engine. The supplemental content delivery system identifies supplemental content that should be provided to viewers of a web page (the web page can be a search engine results page (SERP) or several other suitable web pages). A generative model is used to improve the delivery of supplemental content to viewers of a web page, as will be described in more detail here.

[0006] In a non-existent example, a search engine receives queries from client computing devices and generates search engine results pages (SERPs) based on those queries. A generative model can be integrated with the search engine, and an interface for the generative model can be included in the SERP. Thus, users of client computing devices can provide input to the generative model through the aforementioned interface in the SERP. Furthermore, at least some of the information in the SERP can be provided to the generative model as input (as part of a prompt), and as a result, the output generated by the generative model is based on the content of the SERP.

[0007] The input supplied to the generative model (for example, by a user or from information in a SERP) may also relate to a specific entity (e.g., a company) that provides goods or services for acquisition, such as products being available through a company and services being available through a company. The generative model may be able to identify that the input relates to a company and identify a second generative model specific to that company (trained to supply output about the company, including information about products available from the company, the company's offers, etc.). The second generative model may then supply output to the user about the entity. In this example, the output of the second generative model may be supplementary content. Various other examples of providing supplementary content based on the output of a generative model are described here.

[0008] The above summary provides a brief overview to give a basic understanding of some aspects of the systems and / or methods described herein. This summary is not a comprehensive overview of the systems and / or methods described herein. The summary is not intended to identify key / important elements or to describe the scope of such systems and / or methods. The sole purpose of the summary is to present some concepts in a brief form as a prelude to the more detailed explanations that will be presented later. [Brief explanation of the drawing]

[0009] Brief explanation of the drawing [Figure 1] This is a functional block diagram of a computing environment that facilitates the provision of supplementary content based on the output of a generative model. [Figure 2] This is a schematic diagram showing a graphical user interface (GUI) corresponding to a company-specific generative model. [Figure 3] This flowchart illustrates how to select a generative model from several available models and how to supply the output generated by that model to the user. [Figure 4] This flowchart illustrates how a generative model corresponds to a website, and how the output generated by the generative model is supplied to the user. [Figure 5] This is a schematic diagram showing a GUI that identifies a second generative model during a conversation between the user and the generative model, and then allows the second generative model to continue the conversation with the user. [Figure 6] This flowchart illustrates how to present the output of a generative model to the user. [Figure 7] This is a schematic diagram illustrating a GUI that assigns hyperlinks to some of the output generated by a generative model, and further presents supplemental content when the pointer is paused over a hyperlink. [Figure 8] This flowchart illustrates how to assign hyperlinks to text in the output of a generative model, where the hyperlinks correspond to supplemental content items. [Figure 9] This is a schematic diagram illustrating supplementary content about a company, which includes information generated by a generative model. [Figure 10] This flowchart illustrates how to update supplemental content items to include content generated by the generative model. [Figure 11] This is a schematic diagram illustrating the GUI of a SERP, where the GUI contains supplementary content, and the supplementary content further contains an interface for interacting with the generative model. [Figure 12] This is a schematic diagram showing a GUI corresponding to a generative model, which inserts supplementary content into the conversation between the user and the generative model. [Figure 13] An example of a computing device is shown. [Modes for carrying out the invention]

[0010] Detailed explanation Now, in connection with providing supplemental content on web pages, various techniques for using generative models (e.g., GLMs) will be described with reference to drawings that use the same reference numerals throughout to mean the same elements. In the following specification, for illustrative purposes, numerous specific details are shown to give a complete understanding of one or more embodiments. However, it may be apparent that such embodiments can be carried out without these specific details. In other cases, well-known structures and devices are shown in block diagram form to facilitate the description of one or more embodiments. Furthermore, functions described as being performed by specific system components are understood to be performable by numerous components. Similarly, components may be configured to perform functions described as being performable by numerous components, for example.

[0011] Furthermore, the term “or” is intended to mean an inclusive “or,” not an exclusive “or.” That is, unless otherwise specified or it is clear from the context, the phrase “X uses A or B” is intended to mean either of the natural inclusive substitutions. That is, the phrase “X uses A or B” is satisfied by any of the following cases: X uses A, X uses B, or X uses both A and B. Furthermore, the articles “a” and “an,” as used in this application and the attached claims, should generally be interpreted as meaning “one or more,” unless otherwise specified or it is clear from the context relating to the singular form.

[0012] Furthermore, as used herein, the terms “component,” “system,” “engine,” and “module” are intended to include computer-readable data storage devices consisting of computer-executable instructions that, when executed by a processor, cause to perform a particular function. Computer-executable instructions may include routines, functions, etc. Furthermore, it is understood that a component or system may be localized to a single device or distributed across several devices.

[0013] The technology described herein relates to providing supplemental content to web page visitors based on the output of a generative model. For example, the generative model could be a "sponsored" model assigned to a specific company, which is trained to converse with users about products and / or services offered for acquisition by the company. In such a case, the generative model could, for example, access a database corresponding to the company and access the company's web pages. Thus, in this example, the output generated by the generative model is supplemental content. In another example, the generative model is trained to generate text about products or services referenced in the supplemental content. Such text could be based on content from web pages on the website of a company providing products or services for acquisition. Various other examples are described in more detail here.

[0014] Now, let's explain Figure 1. It illustrates a functional block diagram of the computing environment 100. The computing environment 100 includes a computing system 102 and a client computing device 104, which communicate with each other via a network 106. The computing system 102 includes a processor 108 and a memory 110, the memory 110 which stores instructions executed by the processor 108. More specifically, the memory 110 includes a supplemental content serving system 112 configured to identify supplemental content to be presented to the user on a web page (e.g., a SERP or other appropriate web page). Furthermore, the memory 110 includes a generative model 114 that generates output based on prompts. In contrast to conventional technology, prompts supplied to the generative model 114 can include information in addition to user input. For example, a prompt can include information obtained by a search engine based on user input, content displayed on a web page, etc. This additional information can be automatically supplied to the generative model 114 (e.g., without user intervention). Furthermore, memory 110 includes a search engine 116 that performs searches based on queries. Queries can be generated by a user or by a generative model 114 based on user input. For example, the generative model 114 receives user input, constructs a query, and supplies the query to the search engine 116, where the search engine 116 performs a search based on the query. At least a portion of the search results identified by the search engine 116 can be supplied to the generative model 114, where the generative model 114 can generate output based on such information.Therefore, the prompt supplied to the generation model 114 can include 1) user input shown in the generation model 114 and / or the search engine 116, 2) a query generated by the generation model 114 based on the user input, 3) information extracted from search results identified by the search engine 116 based on the user input and / or the query, 4) information in the previous prompt used by the generation model 114 to generate the output, and 5) previous outputs of the generation model 114 (but not limited to including these).

[0015] Furthermore, the computing system 102 includes a data storage device 118. The data storage device 118 stores supplementary content 120 and user history data 122. For example, the supplementary content providing system 112 can identify supplementary content from the supplementary content 120 based at least in part on the information in the user history 122. Furthermore, as will be described in more detail later, the generation model 114 can generate an output based on the user history 122. Such an output can be used by the supplementary content providing system 112 to identify and supply supplementary content for presentation on a web page to the user.

[0016] Various graphical user interfaces related to the generation model and / or the web page will be described later. Refer to the computing environment 100 for these GUIs as needed.

[0017] Now, let's explain Figure 2. It illustrates a GUI 200 for a company-specific generative model (for example, the generative model is trained with training data identified by the company and / or has access to the company's database). The GUI 200 can be presented on a SERP and can also be displayed on a webpage of the company's website. For example, a client computing device 104 can receive user input from a user (not shown) and send the user input to a search engine 116. A supplemental content delivery system 112 identifies supplemental content based on the user input, and the supplemental content corresponds to the company. The supplemental content 204 identified by the supplemental content delivery system 112 is included in the GUI 200. That is, the supplemental content delivery system 112 can wrap the supplemental content 201 with company-specific GUI features, and the GUI features include an input field 202 where the user can indicate input to a company-specific generative model. Therefore, as illustrated, the GUI 200 includes supplemental content 204 identified by the supplemental content delivery system 112, and input fields 202 that allow the user to interact with a company-specific generative model (for example, the generative model can access the company's database and, in detail, is trained to interact with customers such as the company).

[0018] Furthermore, when a browser running on the client computing device 104 loads the company's web page, the GUI 200 can be presented. Thus, even when the browser is not displaying a SERP, the GUI 200 can be presented to the user on the web page. Therefore, the technology described herein provides the same experience to the user when interacting with the generative model, regardless of whether the user is interacting with the generative model via a SERP or via the company's web page. Additionally, the generative model 114 can not only generate insights about the company from the supplementary content 204 identified by the supplementary content providing system 112, but it is also conceivable that the generative model can identify and summarize information from the company's website to personalize the information provided to the user according to the user's preferences. Therefore, the generative model can be integrated into both the SERP and the company's web page, enabling the generative model to generate insights and summarize content for the user.

[0019] Furthermore, the generative model can be a distillation model that can assist the user on the navigation web page of the company's website. For example, when the user is searching for a stopper, the distillation generative model 114 can provide insights, recommendations, complementary products, etc., to assist the user while the user is searching. The distillation model has limited functionality compared to a general-purpose generative model and focuses only on the company's products.

[0020] Referring to FIG. 3, a flowchart illustrating a method 300 for presenting supplementary content to a user is depicted. The method 300 starts at 302 and at 304, receives user input from the client computing device. In an example, the user input is received from a search engine, and the user input is in the form of a query containing keywords.

[0021] In 306, supplemental content from the provider is identified based on user input received in 304 and / or based on search results identified by the search engine based on the query. For example, supplemental content can be identified based on the results of a keyword auction. In another example, supplemental content can be identified based on entities identified in search results returned by the search engine.

[0022] In 308, the generative model assigned to the provider is identified. For example, the generative model can be associated with supplemental content and / or the provider. The generative model is trained based on information identified by the provider (e.g., content on the provider's website, information in the provider's database, etc.). In 310, the GUI for the generative model is presented in the SERP. The SERP includes, for example, a web page identified by the search results as relevant to user input, an information card related to user input, and a link to the generative model's GUI. The generative model's GUI may include supplemental content, or in other words, the supplemental content is wrapped in the generative model's GUI. The generative model's GUI includes input fields where the user can indicate input related to supplemental content and / or the provider. In another example, the generative model's GUI includes a button that, when selected, can capture voice input via the microphone of the client computing device and convert the voice input into text input that can be presented to the generative model.

[0023] In step 312, user input is received via a user input field. In step 314, the output is generated by the generative model and presented on the generative model's GUI. The generative model generates the output based on a prompt that includes user input received via the user field. Furthermore, the prompt may include information from supplemental content presented on the GUI. Furthermore, the prompt may include information from search results identified by the search engine. In addition, the prompt may include default information provided by the provider. Furthermore, the prompt may optionally include information from the user's profile (e.g., user history information, user identification preferences, etc.). Method 300 ends in step 316.

[0024] Referring here to Figure 4, a flowchart illustrating a method for presenting information generated through a generative model is shown. Method 400 begins at 402 and receives at 404 an indication that a user on a client computing device is accessing a webpage on the company's website. In an example, it receives an indication that a web browser running on the client computing device is loading the webpage. In another example, it receives an indication that an application (other than a web browser) is accessing the webpage to search for the webpage's content. For example, the application could be a company application and could be used to view and / or retrieve products or services offered by the company.

[0025] In 406, a GUI for the company's generative model is presented on a web page, and the GUI includes input fields that can receive user input. For example, the computing system runs the generative model in addition to several other generative models from several other providers, and the computing system selects the company's GUI from several possible GUIs based on the website identifier and / or the provider identifier. In 408, user input is received, indicated by the input fields. For example, a web server sends user input to the computing system, and prompts supplied to the generative model include user input. Furthermore, prompts can include web page content. Optionally, prompts can include content from other web pages of the website. In 410, the generative model generates output based on the prompts, and in 412, the output is presented in the generative model's GUI on the web page. For example, the output is sent from the computing system to the web server, and the output is inserted into the generative model's GUI displayed on the web page. In another example, the output is sent from the computing system to a client computing device, where the output is displayed in the generative model's GUI. Method 400 ends in 414.

[0026] Referring to Figure 5, a GUI 500 is shown that corresponds to a company (provider) specific generative model. In the example, the generative model 114 contained in memory 110 is a general-purpose generative model. Furthermore, although not illustrated, memory 110 may contain several other generative models, each corresponding to a different company. GUI 500 includes an input field 202, which allows the user to provide input to the generative model. In the example, first, user input (e.g., the input "I want to book a flight") is supplied to the generative model 114. The generative model 114 can construct output (not presented to the user) based on the user input, the output can identify entities related to the user input, and the output can be a query based on the user input, etc. The output generated by the generative model can be supplied to a supplemental content delivery system 112, which can identify supplemental content and / or supplemental content providers based on the output of the generative model 114. According to the example shown in Figure 5, the output generated by the generative model indicates that the user's purpose is travel, and such output is supplied to the supplemental content delivery system 112. The supplemental content provision system 114 identifies a travel service provider (e.g., an airline) and further identifies a second generative model (different from generative model 114) assigned to the provider. As described above, the second generative model, trained based on information identified by the provider, can access the provider's database, etc. The second generative model generates output based on prompts, which include user input indicated via input field 202. Furthermore, prompts include other information (e.g., preference information in the user profile, search results identified by search engine 110 based on user input, etc.). The output generated by the second generative model can be displayed on the GUI 500 together with the output generated by generative model 114. Subsequently, user input can be supplied to the second generative model, which can generate subsequent output based on subsequent user input.

[0027] As illustrated, the second generative model can generate the output "Hello, I'm an agent for the company. Where are you going?". In this example, the second generative model and / or the content provided by the second generative model can be supplementary content. In the example, the company corresponding to the second generative model is billed for the conversation between the user and the second generative model. After the user provides an input indicating that the conversation has ended (e.g., requesting a new topic), the user input is supplied to the generative model 114.

[0028] Figure 6 is described below. Method 600 is presented, which is executed by a computing system that runs multiple generative models. Method 600 begins at 602, and at 604, user input is received by a first generative model. At 606, the first generative model generates output based on the user input (optionally, other information such as information extracted from search results identified by a search engine based on the user input, and / or based on queries generated by the first generative model based on the user input). At 608, the output is supplied to a supplemental content delivery system, and at 610, the supplemental content delivery system identifies a second generative model based on the output generated by the first generative model. As described above, the second generative model can be assigned to a provider, trained based on information associated with the provider, and the second generative model can access the provider's data. At 612, the second generative model generates a second output based on the user input received at 604, and the supplemental content delivery system supplies the user input to the second generative model. The second output is presented to the user via a GUI. Method 600 ends at 614.

[0029] Referring now to Figure 7, another GUI 700 corresponding to generative model 114 is presented. In the example GUI 700, the user is conversing with generative model 114 about a trip to a Greek city. Generative model 114 generates output based on prompts, which include inputs indicated by client computing device 104. Furthermore, as described above, generative model 114 can generate output based on search results identified by search engine 116, which identifies search results based on incoming user input and / or queries generated by generative model 114 based on incoming user input.

[0030] Furthermore, the output of the generative model 114 can be analyzed to determine whether the content in the output corresponds to a supplemental content item (or a provider of supplemental content items). For example, the generative model 114 (or another example of the generative model 114) can analyze the output and identify the text corresponding to a supplemental content item. In another example, a second generative model is trained to identify the text corresponding to a supplemental content item. In yet another example, the output generated by the generative model 114 is supplied to the supplemental content delivery system 112, which identifies the supplemental content item based on the output. In yet another example, the generative model 114 (or another example of the generative model 114) can generate a summary of the output of the generative model 114 and supply the summary to the supplemental content delivery system 112. The supplemental content delivery system 112 can identify the supplemental content item based on the summary.

[0031] For example, when the generative model 114 determines that text in the output corresponds to a supplemental content item, it can assign a hyperlink to such text (which is highlighted for presentation to the user), and when the user pauses over the hyperlinked text, it presents the identified supplemental content item 902. If the user selects the hyperlinked text in the conversation output, or if the user selects the supplemental content item 902, the web page corresponding to the supplemental content item 902 can be loaded by the web browser.

[0032] Several techniques are available for identifying and highlighting text in the output to correspond to supplemental content items. In the first example, generative model 114 generates output based on input provided by the user of client computing device 104, optionally based on search results identified by search engine 116 presented to the user (based on queries provided to search engine 116 by the user), or based on search results identified by search engine 116 based on input provided to generative model 114 or queries generated by generative model 114. In particular, the output is not based on supplemental content items. The conversational output generated by generative model 114 is then fed to a second generative model that analyzes the output for text that can correspond to supplemental content items.

[0033] In the second example, user input, search results identified by the search engine 116, and supplemental content item information are supplied to the generative model 114. However, the prompt supplied to the generative model 114 itself labels the supplemental content item information, and the generative model 114 does not generate conversational output based on the information about the supplemental content items. Once the generative model 114 generates output, it can use the information about the supplemental content items in relation to identifying the text in the conversational output that corresponds to one or more supplemental content items. This technique eliminates the need to make calls to other generative models because the generative model 114 already has the information about the supplemental content items.

[0034] Referring now to Figure 8, a flowchart illustrating method 800 for assigning hyperlinks to text in the output of a generative model is shown. Method 800 begins at 802 and, at 804, receives user input in the generative model. Optionally, the generative model generates a query based on the user input, feeds the query to a search engine, and the search engine identifies search results based on the query.

[0035] In 806, the generative model generates output based on user input (optionally, based on at least a portion of the search results identified by the search engine). In 808, the generative model identifies the text corresponding to the supplemental content items in the output generated. As described above, the generative model can analyze the output and identify the text in the output corresponding to the supplemental content items. In another example, a second generative model receives the output and identifies the text in the output corresponding to the supplemental content items. In yet another example, the output is supplied to a supplemental content providing system 112, which identifies the text in the output corresponding to the supplemental content items.

[0036] In step 810, the hyperlink is assigned to text in the output, and the hyperlink text is presented to the client computing device. If the hyperlink text is paused, supplemental content items can be presented (for example, as a pop-up window). If the hyperlink text (or supplemental content item) is selected, the web browser loads the web page (landing page) corresponding to the supplemental content item and presents it on the client computing device's display. Method 800 ends in step 812.

[0037] In addition to the functions described above, the generation model 114 can also generate content that can be included in supplemental content items (for example, the generation model 114 can construct supplemental content items at least partially). Referring here to Figure 9, a GUI 900 for supplemental content is shown, which includes at least a portion of the supplemental content generated by the generation model 114. For example, based on a query presented to the search engine 116 by a user of the client computing device 104, or based on a webpage viewed by a user of the client computing device 104, or other appropriate information, the supplemental content delivery system 112 identifies supplemental content items that should be presented to the user of the client computing device 104. For example, a supplemental content item may relate to a company that wants to show that at least one product or service is of interest to the user of the client computing device 104. The supplemental content items identified by the supplemental content delivery system 112 and / or the identity of the company corresponding to the supplemental content items are supplied to the generation model 114. When supplying the company identity to the generation model 114, webpages of the company's website can also be supplied to the generation model 114. For example, the generative model 114 can construct a query and feed such a query to the search engine 116. The search engine 116 can search for search results based on the query, and the search results include web pages belonging to the company's website. Information extracted from the search results can be provided to the generative model 114. Based on supplemental content items identified by the supplemental content delivery system 112 and web pages of the company's website, the generative model 114 can generate insights about the company, products offered for acquisition by the company, services offered for acquisition by the company, and so on.

[0038] The insights described above are, or may include, interesting offerings, indicated by the company, the relevance of supplemental content items about the company to the query, and the highlighting of information about what the user has searched for. In a non-limited example, a supplemental content item identified by the supplemental content delivery system 112 may be about a product. A webpage on the website of a company that offers products for acquisition may indicate that free shipping is available for products purchased online (the supplemental content item may not include information about free shipping). Despite a supplemental content item that does not mention free shipping, the generation model 114 may generate an insight indicating that free shipping is available from the company. Accordingly, the GUI 900 includes an insight field 902 that contains insights about the company and / or products and / or services related to the supplemental content item identified by the supplemental content delivery system 112.

[0039] In yet another example, generative model 114 is well suited for use in relation to mobile computing devices. For example, if a user of a mobile computing device has been inactive for a certain threshold time, a voice agent can be activated on the mobile computing device. Thus, generative model 114 can assist a user with parsed search results by using voice as the dialogue mode through voice interaction. For example, a voice-based agent (generative model 114) might encourage supplementary content items, search results identified by the search engine 116, etc. The voice agent can be activated based on one or more metrics (e.g., the time the user has been browsing a web page, or other indications that the user may need additional assistance when reviewing content on the web page). Generative model 114 can generate voice-based conversational output based, for example, on search results presented in a SERP, the content of a web page presented on a mobile device, etc. Generative model 114 can summarize the content on a web page in a meaningful way, rather than simply providing a reading result.

[0040] Generative model 114 can also be used to provide users with additional information about companies that offer products or services for acquisition. Traditional search engines include shopping-related verticals, and when a user issues a query to such a vertical, the search engine displays supplemental content items from various retailers that offer such products or services for acquisition. Some of the companies corresponding to these supplemental content items are typically well-known, but others may be smaller, less well-known retailers, and users may need to conduct their own research before deciding to purchase from such retailers. Generative model 114 can be used to provide users with additional information about these types of retailers, thereby enabling users to make smarter decisions about purchasing products. For example, generative model 114 can retrieve information from social media sites about company reconsideration, information from company websites, and information from knowledge graphs used by search engine 116. Furthermore, generative model 114 can retrieve information about recent news events, social media posts, etc., and generate semantically meaningful information about the company. For example, search engine 116 searches for such information based on queries generated by generative model 114.

[0041] Referring to Figure 10, Method 1000 is shown as performed by the generative model. The method begins at 1002 and at 1004 receives as input a supplemental content item selected to be presented to a user of a client computing device, and / or information about the supplemental content item. The information may include the identity of a product, the identity of a service, the identity of a product provider, the identity of a web page corresponding to a product or service, etc. At 1006, the generative model generates an output based on the supplemental content item and / or the information received at 1002. At 1006, the supplemental content item is updated to include the output generated by the generative model. Method 1000 ends at 1008.

[0042] Referring here to Figure 11, the GUI 1100 of the SERP is shown. The SERP includes a query field 1102 where a user of a client computing device 104 can submit a query. The search engine 116 searches various different data sources based on such queries and generates search results. For example, the search results may include a knowledge card 1104 that shows information about the entity referenced in the query, a field 1106 containing a link to a webpage identified by the search engine 116 as relevant to the query, a widget 1108 that the search engine 116 identifies as relevant to the query, and a supplemental content item 1110 identified by the supplemental content delivery system 112 (for example, based on the query). In contrast to conventional methods, the supplemental content item 1110 includes features that can interact with a generative model 114 (for example, assigning the generative model 114 to the company corresponding to the supplemental content item 1110). For example, a supplemental content item 1110 includes suggestion chips 1112 and 1114, and the selection of a suggestion chip can supply conversational input represented by the suggestion chip to the generative model 114 (thus initiating a conversation with the generative model 114 about products, services, companies, etc., related to the supplemental content item 1110). Furthermore, the supplemental content item 1110 may include an input field 1116 in which a user of the client computing device 104 can indicate conversational input to the generative model 114, or may have an input field 1116 graphically associated with the supplemental content item 1110. In other words, the supplemental content item 1110 can be visually wrapped with features related to the generative model 114. For example, the generative model 114 generates suggestion chips 1112 and 1114 based on information about the supplemental content item 1110 and / or information about a provider associated with such item 1110. The aforementioned information can be obtained directly from a web page associated with the supplemental content item 1110 and / or the provider.

[0043] In this case, the supplemental content item 1110 and / or graph features can indicate to the user of the client computing device 104 that the generative model 114 can be assigned to the company. The generative model 114 can be customized by the company (to enable gentle sales, aggressive sales, access to company inventory, etc.). Although the supplemental content item 1110 is exemplified as being on a SERP, it is understood that the supplemental content item wrapped in features relating to the generative model 114 can appear on any appropriate web page or web application that presents the supplemental content item.

[0044] For example, the application programming interface of generative model 114 is exposed to a company, allowing such a company to connect to content for generative model 114 to use in conversations (e.g., price feeds, availability, deep web data, data-background databases, product support databases, specifications, features, sales decks for products, etc.). Furthermore, meta HTML tags can be exposed on the page to provide generative model 114 with additional information about how to converse with the user, what to say to the user, conversational rules, politeness, etc. In another example, generative model 114 is customized for a company, allowing the company to control the content output by generative model 114.

[0045] Referring to Figure 12, a GUI 1200 for the generative model 114 is presented. The GUI 1200 shows that supplemental content items 1202 can be presented in a conversation between the user of the client computing device 104 and the generative model 114. In the example shown in Figure 12, the user of the client computing device 104 is conversing with the generative model 114 about a particular type of car seen in a movie of interest to the user of the client computing device 104. Information identifying the brand and year of the car is provided to the generative model 114. As illustrated, the generative model 114 receives the input "What was the car from 'movie'?", and the generative model 114 can construct a query (e.g., "car from the film 'movie'") and feed the query to the search engine 116. The search engine retrieves search results based on the query and feeds at least a portion of the search results to the generative model 114.

[0046] Furthermore, the generation model 114 can verify that the user of the client computing device 104 is interested in such a car based on the content of the conversation (optionally, part of the user history 122), and can indicate to the supplemental content delivery system 112 that the user is interested in the car. For example, the generation model 114 supplies a generation query to the supplemental content delivery system 112, and the supplemental content delivery system 112 identifies a supplemental content item based on the query. The supplemental content delivery system 112 then presents the supplemental content item 1202 within the flow of the conversation. For example, the supplemental content delivery system 112 supplies the supplemental content item 1202 to the generation model 114, and the generation model 114 inserts the supplemental content item 1202 into the flow of the conversation between the user of the client computing device 104 and the generation model 114. For example, a pricing structure can be set up so that, for instance, access to the company corresponding to the supplemental content item 1202 can be made for the placement of the supplemental content item 1202 in the conversation, and access to the pricing structure can be made for each conversational action related to the car, and access to the pricing structure can be made when the supplemental content item 1202 is selected.

[0047] While some examples of functions that can be performed by the supplemental content delivery system 112, the generative model 114, and the search engine 116 have been described above, various other functions are also conceivable. In one example, the generative model 114 generates a story about multi-click supplemental content items. Specifically, the site currently generates a “Top Deals for X” list consisting of a set of supplemental content items. These supplemental content items form a broad class of two-click experiences. In addition to the supplemental content items themselves that can be supplemented by the output of the generative model 114, unique content generated by the generative model 114 can be placed, which is configured to influence user behavior with respect to the supplemental content items. For example, the generative model 114 could generate an equivalent of an inspiring story about how a particular product was used at a key moment. To do so, the generative model 114 can be given the content of existing supplemental content items that form the core of the unique experience, and the generative model 114 can generate a story based on the set of supplemental content items. In such an example, the prompts used by the generative model 114 include information from the supplemental content items and instructions to generate a story about such items.

[0048] In yet another example, GLM114 generates a semantic representation of a user's interests based on the user's interaction history with web pages, supplemental content items, search results, etc. For example, a user of client computing device 104 can supply content to the user history to be analyzed so that the user history 122 contains information about the user. The generative model 114 receives such user history and generates a semantic representation of the user's interests based on it. The generative model 114 offers advantages over conventional methods because it can explain the logic behind the user's user history. Thus, given user history, the generative model 114 can summarize the history into a few words / sentences that summarize the user's interests. This summary can be supplied to the supplemental content supply system 112, which can use this information if it selects and supplies supplemental content items to the user of client computing device 104.

[0049] Furthermore, the generative model 114 can be configured to assign labels to supplemental content items in supplemental content 120. Traditionally, obtaining data on the relevance of supplemental content items to queries, the relevance of supplemental content items to keywords, the relevance of supplemental content items to users or sets of users requires considerable time and resources. Supplemental content items in supplemental content 120 can be supplied to the generative model 114, which can then assign labels to supplemental content items that indicate relevance to specific queries, keywords, sets of users, etc. Furthermore, the generative model 114 can translate supplemental content items for each language so that it can assign labels to supplemental content items in different languages ​​using a single prompt. In such cases, the prompt supplied to the generative model 114 can be designed to cause the generative model 114 to perform these tasks.

[0050] Furthermore, the generation model 114 can be configured to generate supplemental content items entirely, or to generate only a portion of them (title, description, etc.). Traditionally, titles for supplemental content items are automatically generated for various products and / or services. However, traditionally, the generated content is not personalized (to the end user to whom the supplemental content item is provided, or to the company to which the supplemental content item corresponds). The generation model 114 can generate content for supplemental content items (e.g., title, description, site links, etc.), and such content can be generated based on user information (e.g., a set of users corresponding to a specific segment), company information, information extracted from web pages (e.g., describing a product), etc. For example, supplemental content items can be generated offline, using content from user history 122 to identify appropriate supplemental content items that have a portion of the supplemental content items generated by the generation model 114. To generate such portions of supplemental content items, the generation model 114 can access user history 122 (e.g., landing pages for products, company information, etc.), and the generation model 114 can generate portions of supplemental content items based on such information. Furthermore, if the user consents to the analysis of their input, the input provided by the user of the client computing device 104 to the generative model 114 can be leveraged in relation to building a user profile. For example, based on excellent contextual information from a dialogue between the user and the generative model 114, the generative model 114 can build a user-specific embedding that enables more targeted, personalized searches of supplemental content for the user. For example, the context from a dialogue between the user and the generative model 114 can be incorporated into the user representation as a near real-time signal in relation to identifying supplemental content items for presentation to the user.

[0051] In relation to supplying supplemental content items to users, company agents often indicate the valuation of the space on the web page where the supplemental content items should be displayed. Currently, in relation to estimating the demand for supplemental content items, the revenue corresponding to supplemental content items, etc., ad networks and advertisers use valuation estimates before launching ad campaigns. For example, generative model 114 can be used to provide predictive modeling in relation to estimating the valuation corresponding to supplemental content items. That is, the features used in relation to estimating the valuation can be increased with better feedback from conversational content between generative model 114 and the user. Thus, the contextual embedding vectors generated by generative model 114 can be features used to represent user engagement, interest, and click feedback, and as a result can be used to estimate the valuation.

[0052] In relation to generating images for supplemental content items, a generative model 114 can also be used. Images tend to be quite important in relation to certain types of electronic advertising. However, a company does not need to associate an image with every supplemental content item that should be used by the company. Therefore, a generative model 114 can be used to generate queries to be used by the search engine 116, and to find images that can be used to generate images that can be assigned to and / or should be assigned to text supplemental content items.

[0053] Various other features are also conceivable. For example, if a user of a client computing device 104 refers to a specific word or entity (or something related to a word or entity) during a conversation with a generative model 114, a supplemental content item can be triggered. For example, the supplemental content item can be presented during the conversation as an arbitrary widget (image) or a sharper conversational response (e.g., "Have you ever thought of car Y?"). The generative model 114 can be configured to perform intent detection, trigger the identification of the supplemental content item, and generate a response to the specific content. The conversational output generated by the generative model 114 associated with the supplemental content item can be marked to indicate that the supplemental content item is related to a company (e.g., sponsored). In such cases, the supplemental content item can appear in the flow of the conversation or be displayed in other services on a web page.

[0054] Similarity-based ad auctions are also conceivable. Advertisers can write perfect text descriptions of the types of users and / or search and conversational activities they are trying to target. Such text descriptions (paragraphs, sentences, etc.) can be embedded into vectors by the generative model 114. Advertisers can assign values ​​to placements based on the vector similarity between text descriptions and user activities (e.g., dialogue history, search history, etc.). Embeddings can resolve ambiguity in search intent (e.g., does the term "tiger" refer to the animal or the sports team?). The generative model 114 can appropriately consider content in user-generated dialogue changes in response to dialogue changes generated by the generative model 114, and can train embeddings in diverse combinations, embedding existing text, audio, visual content, and / or non-visual metadata. Such contextual embeddings can be sent directly to the search engine 116 to further improve search relevance.

[0055] Generative model 114 can represent the meaning of most text fairly accurately. When users are having interactive conversations, individual keywords are not particularly useful in enabling auctions from advertisers who want to promise users interested in a particular topic. Instead, recent conversation history embeddings capture many attributes of the conversation (e.g., user interests and preferences). Text and training / validation preprocessing of the embeddings can be performed to ensure that certain biases are removed from the embeddings and invalid targeting criteria are eliminated. Generative model 114 can perform these tasks, generate training data, and adjust the embeddings so that they are not biased against such constraints.

[0056] Advertisers can submit bids for any block of embedded text. Nearest-neighbor similarity searches can be performed on user content in real-time or near real-time, and advertisers can be charged proportionally to the similarity. This model can be integrated into existing keyword auction mechanisms. Non-linear pricing can be applied based on the number of similarities and competitive bids.

[0057] Furthermore, generative model 114 can be used in connection with ad optimization to assist advertisers in generating content that enables them to sell products better than competitors' products. For example, a second generative model can be trained to mimic / simulate human users. User personalities can be extracted from actual user conversations (on other topics) and incorporated into customer profiles. User categories can be learned from conversation logs. The results of such simulated conversations can be monitored and used to modify supplementary content items. For example, the simulated user promised in the ad, the sentiment of the simulated user's responses, etc., can be tracked and used in connection with ad modification. According to the example, the ad can be automatically modified until the desired results are achieved.

[0058] Referring here to Figure 13, a high-level explanatory diagram of an exemplary computing device 1300 that can be used by the systems and methods disclosed herein is shown. For example, computing device 1300 may be used in a system configured to supply content displayed in a web browser as at least part of a prompt to a GLM. As another example, computing device 1300 may be used in a system configured to present supplemental content items to a user. Computing device 1300 includes at least one processor 1302 that executes instructions stored in memory 1304. Instructions may be, for example, instructions that perform the functions described to be performed by one or more of the components described above, or instructions that perform one or more of the methods described above. Processor 1302 may access memory 1304 via a system bus 1306. In addition to storing executable instructions, memory 1304 may store prompts, images, supplemental content items, user history, dialogs, and the like.

[0059] Furthermore, the computing device 1300 includes a data storage device 1308 accessible by the processor 1302 via the system bus 1306. The data storage device 1308 may include executable instructions, immediate responses, web indexes, etc. Furthermore, the computing device 1300 includes an input interface 1310 that allows external devices to communicate with the computing device 1300. For example, the input interface 1310 may be used to receive instructions from external computer devices, users, etc. Furthermore, the computing device 1300 includes an output interface 1312 that interfaces the computing device 1300 with one or more external devices. For example, the computing device 1300 may display text, images, etc. via the output interface 1312.

[0060] External devices communicating with the computing device 1300 via the input interface 1310 and output interface 1312 may be included in the environment to provide substantially any type of user interface with which the user can interact. Examples of user interface types include graphical user interfaces and natural user interfaces. For example, a graphical user interface may accept input from the user using an input device (e.g., a keyboard, mouse, remote control, etc.) and supply output to an output device (e.g., a display). Furthermore, a natural user interface may allow the user to interact with the computing device 1300 in a way that is not constrained by input devices (e.g., a keyboard, mouse, remote control, etc.). Rather, a natural user interface may rely on speech recognition, touch and stylus recognition, gesture recognition both on and near the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, artificial intelligence, etc. Furthermore, although exemplified as a single system, it is understood that the computing device 1300 can be a distributed system. Therefore, for example, several devices may communicate via a network connection and perform the tasks described above in a batch, as performed by the computing device 1300.

[0061] The various functions described herein can be implemented in hardware, software, or any combination of hardware and software. When implemented in software, the functions can be stored in one or more instructions or codes on a computer-readable medium, or transmitted as one or more instructions or codes. Computer-readable medium includes computer-readable storage media. Computer-readable storage media can be any available storage medium accessible by a computer. Such computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures, and any other medium accessible by a computer. Disks and discs as used herein include compact discs (CDs), laser discs, optical discs, digital-purpose discs (DVDs), floppy disks, and Blu-ray discs (BDs), where a disk typically reproduces data magnetically, and a disc typically reproduces data optically using a laser. Furthermore, propagating signals are not included within the scope of computer-readable storage media. Computer-readable media further includes communication media, which include any medium that facilitates the transmission of computer programs from location to location. Connections can be, for example, communication media. For example, when transmitting software from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (e.g., infrared, radio, and microwave), coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (e.g., infrared, radio, and microwave) are included in the definition of communication media. The above combinations should also be included within the scope of computer-readable media.

[0062] Alternatively, or furthermore, the functions described herein can be performed at least partially by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), integrated circuits for specific programs (ASICs), standard products for specific programs (ASSPs), system-on-chip systems (SOCs), and composite programmable logic devices (CPLDs).

[0063] Various methods are disclosed herein with respect to generative models and supplemental content items. Although the methods are shown and described as a series of actions performed sequentially, it should be understood that the methods are not limited by the order of the sequence. For example, some actions can be performed in an order different from that described herein. Furthermore, actions can be performed concurrently with other actions. Moreover, in some cases, not all actions are necessary to perform the methods described herein.

[0064] Furthermore, the operations described herein may be computer-executable instructions that can be executed by one or more processors and / or stored in computer-readable media or multiple media. Computer-executable instructions may include routines, subroutines, programs, execution threads, and the like. Furthermore, the results of the operation of the method may be stored, for example, in computer-readable media and displayed on a display device.

[0065] Furthermore, various technologies are described here regarding the presentation of supplementary content, at least by the examples below.

[0066] (A1) In an embodiment, a method is disclosed herein, the method comprising supplying a prompt to a generating model; configuring the generating model to produce an output based on the prompt; further configuring the generating model to identify that text in the output should be associated with a supplemental content item; further configuring the generating model to assign hyperlinks to text in the output; further, the method comprises displaying the output on a display of a client computing device, wherein the text in the display output has hyperlinks assigned to the text, and further, displaying the supplemental content item on the display simultaneously with the output while pausing over the hyperlinks.

[0067] In some embodiments of the method described in (A2)(A1), when a hyperlink is selected, a web page relating to supplemental content is loaded by the web browser of the client computing device and presented on the display.

[0068] (A3) In some embodiments of the method according to any one of claims (A1) to (A2), the method further includes receiving user input which is included in a prompt, and further receiving user input before supplying a prompt to a generative model.

[0069] In some embodiments of the method described in (A4)(A3), the generative model generates a query based on user input. The method further comprises supplying the query to a search engine that identifies search results based on the query. The method further comprises including at least a portion of the search results in the prompt.

[0070] In some embodiments of the method described in any one of (A5)(A1) to (A4), the method further includes several actions performed after causing the output to be displayed on the display of a client computing device. The actions include supplying user input to a generative model configured to generate a query based on user input. The actions further include supplying the query to a supplemental content provider system that identifies a second supplemental content item based on the query. The actions further include supplying the second supplemental content item to the generative model. The actions further include causing the second supplemental content item to be displayed on the display as part of a conversation between the user and the chatbot.

[0071] In some embodiments of the method described in (A6)(A5), the method further includes supplying a query to a search engine that identifies an image based on the query. The method further includes inserting the image into a second supplemental content item.

[0072] In some embodiments of the method described in any one of (A7)(A1) to (A6), the prompt includes a command to disregard information about a supplemental content item if it identifies that text in the output should be associated with a supplemental content item.

[0073] In some embodiments of the method described in any one of (A8)(A1) to (A6), the prompt includes user history information and further identifies the text in the output as being associated with a supplemental content item based on the user history information.

[0074] (B1) In another embodiment, a method is disclosed which is performed by a computing system, the method comprising receiving a prompt in a generative model; the method further comprises generating an output containing text by the generative model based on the prompt; the method further comprises the generative model identifying that at least one word in the text should be associated with supplemental content; the method further comprises obtaining a hyperlink pointing to the supplemental content; the method further comprises assigning a hyperlink to at least one word in the output text by the generative language model that generated the output; the method further comprises displaying the output on a display, displaying at least one word in the text having a hyperlink assigned to the text, and further displays the supplemental content item in a graphical user interface (GUI) associated with the generative model together with the output of the generative model, depending on the hyperlink that is paused above.

[0075] In some embodiments of the method described in (B2)(B1), the method further includes receiving an indication that a supplemental content item has been selected. The method further includes updating a prompt in the generation model to include the content of the web page pointed to by the supplemental content item, updating the prompt in response to receiving an indication that a supplemental content item has been selected.

[0076] In some embodiments of the method described in any one of (B3)(B1)~(B2), the method further includes receiving user input contained in a prompt, and further, receiving user input before the generative model generates an output.

[0077] In some embodiments of the method described in (B4)(B3), the method further includes generating a query by a generative model based on user input. The method further includes supplying the query to a search engine that identifies search results based on the query. The method further includes including at least a portion of the search results in a prompt.

[0078] (C1) In yet another embodiment, the method disclosed herein comprises supplying a prompt to a generating model, the prompt comprising instructions to the generating model, the instructions to 1) reconsider the output to be generated by the generating model based on a prompt for text to be associated with a supplemental content item, and 2) assign hyperlinks to the text in the output. The method further comprises receiving the output from the generating model, the output comprising text and hyperlinks assigned to the text. The method further comprises displaying the output in a graphical user interface (GUI), the text having hyperlinks assigned to the text, and further displaying the supplemental content item simultaneously with the output, while pausing over the hyperlinks.

[0079] In some embodiments of the method described in (C2)(C1), when a hyperlink is selected, a web page relating to supplemental content is loaded by the web browser and presented on the display of the client computing device.

[0080] In some embodiments of the method described in any one of (C3)(C1) to (C2), the method includes several actions performed before supplying a prompt to a generative model. These actions include receiving user input and supplying the user input to the generative model. The actions further include receiving a query generated by the generative model based on the user input. The actions further include supplying the query to a search engine that identifies search results based on the query. The method further includes including at least a portion of the search results in the prompt.

[0081] In some embodiments of the method described in any one of (C4)(C1) to (C3), the method further includes including user input and a query generated by a generative model in the prompt.

[0082] In some embodiments of the method described in any one of (C5)(C1) to (C4), the method further includes supplying user input to the generative model after receiving output from the generative model. The method further includes receiving a query generated by the generative model based on the user input. The method further includes supplying the query to a supplemental content provider system that identifies a second supplemental content item based on the query. The method further includes receiving the second supplemental content item. The method further includes displaying the second supplemental content item in a GUI as part of a conversation between the user and the chatbot.

[0083] In some embodiments of the method described in any one of (C6)(C1) to (C5), the supplemental content item is an electronic advertisement.

[0084] In some embodiments of the method described in any one of (C7)(C1) to (C6), a supplemental content item is linked to a web page via a hyperlink. The method further includes receiving user input after causing the output to be displayed. The method further includes retrieving content from the web page. The method further includes supplying a second prompt to a generative model, the second prompt including user input and content from the web page, and the generative model further generates a second output based on the second prompt.

[0085] In some embodiments of the method described in any one of (C8)(C1) to (C7), the prompt further includes information about a supplemental content item. The prompt further includes instructions to disregard the information about the supplemental content item if an output is generated, and the generation model assigns hyperlinks to the text in the output based on the information about the supplemental content item.

[0086] (D1) In yet another embodiment, the system described herein includes a processor and memory, the memory storing instructions that, when executed by the processor, cause the processor to execute any of the methods disclosed herein (for example, any of (A1) to (A8), (B1) to (B4), or (C1) to (C8)).

[0087] (E1) In yet another embodiment, a computer-readable storage medium, when executed by a processor, includes instructions causing the processor to execute any of the methods disclosed herein (e.g., any of (A1) to (A8), (B1) to (B4), or (C1) to (C8)).

[0088] The above descriptions include examples of one or more embodiments. Naturally, for the purpose of describing the above embodiments, it is not possible to describe all possible modifications and changes to the above-described devices or methods, but those skilled in the art will see that many further modifications and substitutions are possible for various embodiments. Accordingly, the embodiments described are intended to encompass all such modifications, changes, and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in the detailed description or claims, such term is intended to encompass in the same manner as the term “comprising” when “comprising” is used as a provisional term in the claims.

Claims

1. Processor and When executed by the aforementioned processor, Generate output based on the prompt, Identify that the text in the output should be associated with a supplementary content item, Assign a hyperlink to the text in the output. Providing prompts to a generative model configured in such a way, To display the output on the display of the client computing device. A memory that stores instructions causing the processor to perform an operation including the above, Includes, The text in the display output has a hyperlink assigned to the text, and furthermore, when the hyperlink is paused, the supplemental content item is displayed simultaneously with the output on the display. Computation system.

2. The computing system according to claim 1, wherein when the hyperlink is selected, the web page relating to the supplemental content is loaded by the web browser of the client computing device and presented on the display.

3. The aforementioned operation is, Before supplying the prompt to the generation model, the user input included in the prompt is received. The calculation system according to claim 1 or 2, further comprising:

4. The generation model generates a query based on the user input, and the operation is as follows: The query is supplied to a search engine that identifies search results based on the query, The prompt includes at least a portion of the search results. The calculation system according to claim 3, further comprising:

5. The aforementioned operation is, After displaying the output on the display of the client computing device, The user input is supplied to the generation model, which is configured to generate a query based on the user input. The query is supplied to a supplemental content provision system that identifies a second supplemental content item based on the query, To supply the chatbot with the second supplementary content item, To display the second supplemental content item on the display as part of the conversation between the user and the chatbot. A calculation system according to any one of claims 1 to 4, further comprising:

6. The aforementioned operation is, The query is supplied to a search engine that identifies an image based on the query, Insert the image into the second supplemental content item mentioned above. The calculation system according to claim 5, further comprising:

7. The calculation system according to any one of claims 1 to 6, wherein the prompt includes an instruction to disregard information regarding the supplemental content item when it identifies that the text in the output should be associated with the supplemental content item.

8. The calculation system according to any one of claims 1 to 7, wherein the prompt includes user history information, and the text in the output is identified as being associated with the supplemental content item based on the user history information.

9. A method performed by a computing system, In generative models, receiving a prompt and Based on the prompt, the generative model generates output including text, The generative model identifies that at least one word in the text should be associated with supplemental content, Obtaining a hyperlink that points to the aforementioned supplementary content, Assigning the hyperlink to at least one word in the output text by the generating language model that generated the output, Displaying the output on a display, including displaying at least one word in the text having a hyperlink assigned to the text, and further displaying the supplemental content item in a graphical user interface (GUI) associated with the generative model together with the output of the generative model in accordance with the hyperlink that is paused above. A method that includes this.

10. Receiving a message indicating that the aforementioned supplemental content item is selected, In response to receiving the indication that the supplemental content item has been selected, update the prompt in the generation model to include the content of the web page pointed to by the supplemental content item. The method according to claim 9, further comprising:

11. The method according to claim 9 or 10, further comprising receiving user input included in the prompt before the generation model generates the output.

12. The user input is used to generate a query using the generative model, The query is supplied to a search engine that identifies search results based on the query, The prompt includes at least a portion of the search results. The method according to claim 11, further comprising:

Images