Integrating Generative Models into Computer-Executable Applications

Abstract

The computing system described herein includes a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform several actions. The actions include generating a prompt that is input to a generative language model, the prompt including content of a web page being presented to a user. The actions also include providing the prompt as input to the generative language model. The actions further include receiving output from the generative language model, the generative language model generating the output based on the prompt. The actions additionally include causing the client computing device to present the output to the user in conjunction with the web page being presented to the user.

Description

[Background technology]

[0001] background Relatively recently, generative models such as generative language models (GLMs) (also known as large-scale language models (LLMs)) have been developed. One example of a GLM is the Generative Pre-trained Transformer 3 (GPT-3). Another example of a GLM is the BigScience Language Open-science Open-access Multilingual (BLOOM) model, which is also a transformer-based model. Briefly, a generative model is configured to generate output (human language text, source code, music, video, etc.) in near real time (e.g., within a few seconds of receiving the prompt) based on a prompt set by a user. A generative model generates content based on the training data on which the generative model is trained. Thus, in response to receiving the prompt, "How many home runs did Babe Ruth hit before she turned 30?", a GLM can output, "Babe Ruth hit 94 home runs before she turned 30." In another example, in response to receiving the prompt "Please provide a list of famous people born in Seattle and Chicago," a GLM may output two separate lists of people (one in Seattle and one in Chicago), where the list of people born in Chicago includes Barrack Obama. However, in both of these examples, the GLM outputs incorrect information, such as that Babe Ruth hit over 94 home runs by the time she turned 30 and that Barrack Obama was born in Hawaii (not Chicago). Thus, both traditional search engines and generative models are deficient with regard to identifying and / or generating appropriate information in response to certain types of user input.

[0002] Additionally, GLMs are not well-suited to providing accurate answers to user inputs about recent information. For example, a GLM cannot provide an accurate output when it receives the input "What is the current weather in Chicago?" because it generates text output based on training data that tends to be at least somewhat outdated (i.e., it is impractical to retrain a GLM every minute with updated information about the weather, sporting events, stock market, news events, etc.). Summary of the Invention [Means for solving the problem]

[0003] overview The following is a brief summary of what is described more fully herein. This summary is not intended to limit the scope of the claims.

[0004] Described herein are various techniques related to integrating generative models (such as GLMs) with applications so that the generative model is provided with input from the application and generates output based on that input. This integration addresses some of the shortcomings of generative models mentioned above, and in one example, a web browser can be configured with a side panel, which includes functionality that can be used to interact with the generative model. A published page provides the content of the page and / or metadata about the page to the generative model, and this information can be used as part of prompts and / or by the generative model to provide suggestions. For example, if a web page loaded by a web browser is a relatively long news article, the generative model is provided with the news article by the web browser and outputs suggestions such as "Would you like me to summarize this article?" that are presented on the web browser by a side panel. Upon receiving an indication that a user wishes to have the article summarized, the generative model generates a summary of the article and presents the summary by the side panel.

[0005] In another example, a web browser may have loaded thereon a web page containing private information, such as an email page, a social media page, a company page, etc. In such an example, the web browser may receive an indication that a user wishes to interact with the Generative Model via a side panel. The web browser may determine that the page is a “private” page (e.g., based on the page not being indexed by a search engine) and may present the user with a consent request, such that the content of the private page is not provided to the Generative Model unless explicit user consent is received. Upon receiving such consent, the Generative Model may be provided with access to the content of the web page and may generate output based on the content of the web page (e.g., the output is conversational in nature).

[0006] In yet another example, a web browser may have several open tabs, each with a web page loaded in it. The generative model may be provided with, among other information, the URL of the web page loaded in the tab and the time the web page was last viewed by the user. The generative model may receive an input such as "Activate the browser tab you were viewing approximately 30 minutes ago" and be able to identify which tab was active in the web browser approximately 30 minutes prior to receiving the input. This functionality may be particularly useful for voice input, allowing a user to quickly switch tabs based on spoken input.

[0007] In yet another example, an operating system may be configured with a pinnable side panel, thereby providing persistent access to a Generative Model regardless of the application the user is using. Thus, a user may be working with a word processing document containing content and can relatively quickly copy and paste the content from the word processing document into an input field in the side panel, providing that content to a Generative Model that can summarize the content, rewrite the content, generate images based on the content, generate videos based on the content, etc. Additionally, the operating system may provide information about open applications (e.g., application identification, application type, etc.) to the Generative Model, which may use that information to generate output (e.g., conversational output, images, etc.). In other words, information about the application is included in prompts provided to the Generative Model, and the Generative Model generates output based on the prompts.

[0008] The foregoing summary presents a simplified overview in order to provide a basic understanding of some aspects of the systems and / or methods discussed herein. This summary is not an extensive overview of the systems and / or methods discussed herein. It is not intended to identify key / critical elements or delineate the scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a functional block diagram of a computing environment in which a web browser includes a side panel through which a user can interact with a Generative Model. [Figure 2] FIG. 1 is a communication diagram showing the flow of communication between a browser, a web server, an interface module, a generative model, and a search engine. [Figure 3]A web page graphical user interface having a side panel by which a user can interact with a Generative Model. [Figure 4] A GUI for a web page with a side panel that allows users to interact with the Generative Model. [Figure 5] FIG. 1 is a communication diagram illustrating the flow of communication between a browser, a web server, an interface module, and a generative model. [Figure 6] A GUI for a web page with a side panel that allows users to interact with the Generative Model. [Figure 7] A GUI for a web page with a side panel that allows users to interact with the Generative Model. [Figure 8] FIG. 1 is a functional block diagram of a computing environment that supports user interaction with a Generative Model through a side panel presented by the operating system of a client computing device. [Figure 9] FIG. 1 is a communication diagram illustrating the flow of communication between an operating system, an interface module, a generative language model, and a search engine. [Figure 10] A GUI for the operating system homepage presents a side panel through which the user can interact with the generative model. [Figure 11] 1 shows a computing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Detailed Description Various techniques relating to the integration of generative models (such as GLMs) with applications and / or operating systems are described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident, however, that such aspects can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Furthermore, it should be understood that functionality described as being performed by a particular system component may be performed by multiple components. Similarly, a component may be configured to perform functions described as being performed by multiple components, for example.

[0011] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, the phrase "X uses A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X uses A or B" is satisfied by any of the instances where X uses A, X uses B, or X uses both A and B. Additionally, as used in this application and the appended claims, the articles "a" and "an" should be interpreted broadly to mean "one or more" unless otherwise specified or clear from the context that the reference is to the singular.

[0012] Furthermore, as used herein, the terms "component," "system," "engine," and "module" are intended to encompass computer-readable data storage comprised of computer-executable instructions that, when executed by a processor, cause the processor to perform particular functions. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be localized on a single device or distributed across several devices.

[0013] When referring to web pages herein, such term is intended to cover traditional web pages as well as web applications.

[0014] The technology described herein relates to integrating a Generative Model with an application (e.g., a web browser) and / or an operating system (OS). In one example, a client computing device executes a web browser, and the web browser receives user input to retrieve a web page. The web browser is configured to communicate with a Generative Model (e.g., a GLM). For example, upon receiving an indication that a user desires to interact with the Generative Model, a side panel is presented by the web browser. The side panel can overlay a portion of the web page being displayed by the web browser. In another example, the content of the web page is resized to accommodate the screen area consumed by the side panel. If the web page is a public page (e.g., indexed by a search engine), the web browser can cause the content of the web page to be provided to the Generative Model. Additionally, the web browser can cause the Generative Model to provide other information about the web browser, such as the Uniform Resource Locator (URL) of a web page loaded in a browser tab, the title of such web page, the time the web page was accessed, etc.

[0015] In one example, based on such information, the generative model generates suggestions and presents them as selectable "chips." An example of a suggestion is, "Shall I summarize this?" In another example, a user types input into an input field in a side panel, which is provided to the generative model. The generative model generates output based on the conversational input provided to the generative model and the content of the web page. Thus, the prompts provided to the generative model include, among other information, not only the user-generated input but also content from the web page, information from web pages loaded in other tabs of the web browser. Thus, the output of the generative model is contextually "geared" to the content of the web page.

[0016] In another example, a web browser is loading a private page, such as a web page containing the user's email, a social media page, etc. While the web browser is loading such a web page, if the web browser receives an indication that the user wishes to interact with the Generative Model, it determines whether the user has consented to the content of the web page being provided to the Generative Model. If the user has not previously provided consent, a request is provided to the user to provide consent for the content of the web page to be provided to the Generative Model. If the user does not provide consent, the content of the web page is not provided to the Generative Model. Conversely, if user consent is received, the content of the web page is provided to the Generative Model such that output generated by the Generative Model is based on the content of the web page. The Generative Model may be requested to assist in generating text within fields of the web page, summarizing the text, etc. Additionally, the Generative Model may infer the content (public or private) of the web page and provide output. Thus, if a web page contains content about Babe Ruth's statistics and for a conversational input of "How many home runs did Babe Ruth hit before she turned 30?", a generative model can infer the content of the web page and generate an output based on the content of the web page, such as "Babe Ruth hit 284 home runs before she turned 30."

[0017] In yet another example, an operating system of a computing device may be configured to present a pinnable side panel on the operating system's GUI, allowing a user of the computing device to interact with the Generative Model through the side panel. Applications such as a word processing application, a web browser, a slideshow application, etc. may thus be executed by the computing device. The applications may present content, allowing a user to easily copy and paste the content displayed by the application into input fields within the side panel. Such content may be provided to a Generative Model, which may summarize the content, rewrite the content, etc., in response to a user's request.

[0018] 1, there is shown a functional block diagram of a computing environment 100. Computing environment 100 includes computing system 101. While shown as a single system, it should be understood that computing system 101 may include several different server computing devices, may be distributed across a data center, etc. Computing system 101 is configured to facilitate interaction between a user and a Generative Model via a web browser (or other suitable application).

[0019] A client computing device 102 operated by a user (not shown) communicates with computing system 101 over network 104. Client computing device 102 may be any suitable type of client computing device, such as a desktop computer, a laptop computer, a tablet (slate) computing device, a video game system, a virtual reality or augmented reality computing system, a mobile phone, a smart speaker, or other suitable computing device.

[0020] The computing system 101 includes a processor 106 and a memory 108, where the memory 108 contains instructions executed by the processor 106. More specifically, the memory 108 includes a search engine 110, a generative model 112, and an interface module 113 that serves as an interface between applications executing on the client computing device 102 (and / or the operating system of the client computing device 102), the search engine 110, and the generative model 112, as described in more detail below. The operation of the search engine 110, the generative model 112, and the interface module 113 is described in more detail below. The computing system 106 also includes data stores 114-122, which store data accessed by the search engine 110 and / or the generative model 112. More specifically, the data stores 114-122 include a web index data store 114, an instant answers data store 116, a knowledge graph data store 118, a supplemental content data store 120, and an interaction history data store 122. The web index data store 114 includes a web index that indexes web pages by keywords contained in or associated with the web pages. The instant answer data store 116 includes an index of instant answers indexed by queries, query terms, and / or terms semantically similar or equivalent to the queries and / or query terms. For example, the instant answer "2.16 meters" may be indexed by the query "How tall is Shaquille O'Neal" (and semantically similar or equivalent queries such as "How tall is Shaquille O'Neal").

[0021] The knowledge graph data store 118 includes a knowledge graph, which includes a data structure about entities (people, places, things, etc.) and their relationships to each other, thereby representing the relationships between the entities. The search engine 110 can use the knowledge graph in connection with presenting entity cards on a search engine results page (SERP). The supplemental content data store 120 includes supplemental content, such as electronic advertisements, that the search engine 110 can return based on a query.

[0022] The interaction history data store 122 includes an interaction history, which includes interaction information regarding a user and the Generative Model 112. For example, the interaction history may include, with respect to a user, an identification of a conversation that took place between the user and the Generative Model 112, input provided by the user to the Generative Model 112 for multiple dialogue turns during the conversation, responses during the conversation generated by the Generative Model 112 in response to input from the user, queries generated by the Generative Model 112 during the conversation that are used by the GLM 112 to generate responses, etc. Additionally, the interaction history may include context obtained by the search engine 110 during the conversation; for example, with respect to a conversation, the interaction history 122 may include content from SERPs generated based on queries submitted by the user and / or the Generative Model 112 during the conversation, content from web pages identified by the search engine 110 based on queries submitted by the user and / or the Generative Model 112 during the conversation, etc. It should be understood that data stores 114-122 are presented to represent a representative sample of the types of data accessible to search engine 110 and / or Generative Model 112, and that there are many other data sources accessible to search engine 110 and / or Generative Model 112, such as data stores containing real-time financial information, data stores containing real-time weather information, data stores containing real-time sports information, data stores containing images, data stores containing videos, data stores containing maps, etc. Such information sources may be available to search engine 110 and / or Generative Model 112.

[0023] The search engine 110 includes a web search module 124, an instant answer search module 125, a knowledge module 128, and a supplemental content search module 130. The web search module 124 is configured to search the web index data store 114 based on queries received by a user, queries generated by the search engine 110 based on queries received by a user, and / or queries generated by the generative model 112 based on a user's interaction with the generative model 112. Similarly, the instant answer search module 126 is configured to search the instant answer data store 116 based on queries received by a user, queries generated by the search engine 110 based on queries received by a user, and / or queries generated by the generative model 112 based on a user's interaction with the generative model 112. The knowledge module 128 is configured to search the knowledge graph data store 118 based on queries received by a user, queries generated by the search engine 110 based on queries received by a user, and / or queries generated by the generative model 112 based on a user's interaction with the generative model 112. Similarly, the supplemental content search module 130 is configured to search the supplemental content data store 120 based on a query received by a user, a query generated by the search engine 110 based on a query received by a user, and / or a query generated by the generative model 112 based on a user's interaction with the generative model 112.

[0024] Search engine 110 may generate structured, semi-structured, and / or unstructured data representing content identified by at least one of modules 124-130. For example, search engine 110 generates a JSON document that includes information obtained by search engine 110 based on one or more searches performed on data stores 114-120 (or other data stores). In one example, search engine 110 generates data in a structure / format suitable for inclusion within a prompt provided to generative model 112.

[0025] Client computing device 102 includes processor 132 and memory 134, with memory 134 having web browser 136 loaded therein. Web browser 136 has client interface module 138 embedded therein, and web browser 136 communicates with interface module 113 of computing system 101 such that client interface module 138 and interface module 113 enable web browser 136 to exchange information with search engine 110 and generative model 112. Optionally, web browser 136 includes model 140, which may be a relatively small generative model trained to perform fairly specific functions that are frequently repeated on web browser 136, such as text summarization and / or text completion (e.g., summarizing text displayed in a web page and / or completing a sentence or paragraph based on a set of words or phrases).

[0026] Environment 100 also includes a web server 142 in communication with client computing device 102 over network 104. Web server 142 hosts a website that includes web pages 144. Web browser 136 is configured to retrieve web pages of the website hosted by web server 142. Thus, web browser 136 can retrieve web pages 144 from web server 142 (e.g., based on user input, based on program input, etc.).

[0027] Because Generative Model 112 communicates with Web Browser 136 via modules 138 and 113, Generative Model 112 may be provided with any appropriate information obtainable by Web Browser 136 as part of a prompt (in addition to input indicated by a user of Client Computing Device 102 to Generative Model 112). For example, Web Page 144 retrieved by Web Browser 136 may include an email sent to the user. Assuming the user has provided authorization for Generative Model 112 to obtain such email, in response to Web Browser 136 receiving an indication that a conversation between the user and Generative Model 112 should begin, Web Browser 136 may provide information from Web Page 144 (e.g., the email) to Generative Model 112 via modules 138 and 113. Specifically, at least one of Client Interface Module 138 or Interface Module 113 is configured to structure the content of the web page in a manner that can be consumed by Generative Model 112 (e.g., as at least part of a prompt). Additionally, web browser 136 receives input from the user and provides that input to interface module 113, which in turn provides that input to generative model 112. Thus, the prompts provided to generative model 112 include not only the input indicated by the user, but also the content of web page 144 being viewed by the user. Thus, generative model 112 can generate output based on the content of web page 144.

[0028] Additionally, the generative model 112 may generate queries based on the content of the web page 144 and inputs provided by the user. The generative model 112 provides the queries to the interface module 113, which in turn provides the queries to the search engine 110 (in a format suitable for use by the search engine 110). The search engine 110 searches at least one of the data stores 114-120 based on the queries. The search engine 110 provides at least a portion of the search results identified by the search engine 110 based on the queries (via the interface module 113) to the generative model 112, thereby providing the generative model 112 with additional context for use when generating output to the user (the output may be conversational output, images, video, etc.). More specifically, the prompts used by the generative model 112 to generate the conversational output may additionally include at least a portion of the search results identified by the search engine 110.

[0029] The operation of the generative model 112 is improved by having the generative model 112 generate output based on prompts that include content from the web page 144 (or information about the web page 144 as determined by the search engine 110). In contrast, traditionally, generative models generate output based solely on user input and interaction history. There are many use cases in which the generative model 112 provides functionality that traditional generative models cannot provide, some of which are described below.

[0030] Examples of the operation of the computing environment 100 will now be described. It should be understood that these examples are non-limiting and that permutations of such examples are contemplated. Referring to FIG. 2, a communication diagram 200 is shown illustrating communication between the web browser 136, the web server 142, the interface module 113, the GLMM 112, and the search engine 110. The illustrated communication diagram illustrates exemplary methodologies related to the integration of generative models with computer-executable applications (e.g., a web browser, an email application, a word processing application, etc.) and / or an operating system. While the methodologies are illustrated and described as a series of acts performed sequentially, it should be understood and appreciated that the methodologies are not limited by the order. For example, some acts may occur in a different order than described herein. Additionally, some acts may occur simultaneously with other acts. Furthermore, in some examples, not all acts may be required to implement the methodologies described herein.

[0031] Furthermore, the acts described herein may be computer-executable instructions that may be implemented by one or more processors and / or stored on a computer-readable medium. Computer-executable instructions may include routines, subroutines, programs, threads of execution, etc. Still further, the results of the acts of the methodologies may be stored in a computer-readable medium, displayed on a display device, etc.

[0032] Search engine 110 can obtain information about “public” pages, which are pages represented in the search engine's index. When indexing web pages, search engine 110 can identify, for example, the topics discussed in the web page, the entities referenced in the web page, the sentiment of the text in the web page (e.g., positive, negative, neutral), etc. At 202, public page information is provided by search engine 110 to interface module 113. While search engine 110 is shown in FIG. 2 as providing the public page information before web browser 136 loads the web page, it will be understood that interface module 113 can obtain the web page information in response to receiving an indication that the web page has been loaded by web browser 136.

[0033] At 204, web browser 136 requests web page 144 from web server 142, and at 206, web browser 136 obtains web page 144 from web server 142. At 208, web browser 136 transmits an information request to interface module 113, the information request being for public page information of web page 144 (provided to interface module 113 by search engine 110). At 210, interface module 113 provides such page information to web browser 136. In one example, the page information may be presented to the user by web browser 136 upon receiving the user's selection of a graphical icon corresponding to such page information.

[0034] The web browser 136 receives an indication that a user of the web browser 136 wishes to interact with the Generative Model 112. At 212, the web browser 136 transmits an indication that the user has requested to interact with the Generative Model 112 to the interface module 113. In response to receiving such an indication, at 214, the interface module 113 provides, for example, the content of the web page, page information generated by the search engine 110, and / or input (if any) indicated by the user. The Generative Model 112 generates output based on the input provided to the Generative Model 112 by the interface module 113 (the input is in the form of a prompt). In one example, the output includes a query constructed to prompt the search engine 110 to obtain additional information. At 216, the Generative Model 112 provides the output to the interface module 113, and at 218, the interface module 113 transmits the query to the search engine 110. Search engine 110 performs a search on at least one of data stores 114-120 based on the query and identifies search results, which may include web pages, instant answers, knowledge cards, etc., identified by search engine 110 as being relevant to the query. At 220, search engine 110 provides at least a portion of the search results identified by search engine 110 to interface module 113. At 222, interface module 113 provides additional context to generative model 112, where the additional context includes at least a portion of the search results identified by search engine 110 (formatted in a manner that can be consumed by generative model 112).

[0035] Generative model 112 generates model output based on information provided to it by interface module 113. The model output may be conversational output, output contained in a suggestion "chip," an image, etc. At 224, generative model 112 provides the model output to interface module 113, which transmits the model output to browser 136 at 226. The model output may be presented by browser 136 concurrently with web page 144.

[0036] Communication diagram 200 shows that the functionality of Generative Model 112 is integrated into Web Browser 136, and that the output of Generative Model 112 is based on the content of at least one Web page loaded by Web Browser 136, and optionally based on information provided by Search Engine 110. In one example, Web Browser 136 is configured with a side panel in which the model output is displayed and in which the user can further interact with Generative Model 112 via input fields in the side panel.

[0037] Referring now to FIG. 3 , a schematic diagram illustrating a GUI 300 of a web browser 136 is shown. In the example shown in FIG. 3 , the web browser 136 retrieves a web page 144 that includes textual content 302 related to the acid tryptophan. The web page 144 is publicly available, and thus, the search engine 110 can provide information about the web page 144 to the interface module 113, including topics discussed within the web page 144, entities recognized within the web page 144, etc. A user of the web browser 136 can indicate an indication that the user wishes to interact with the generative model 112 via the web browser 136. In response to receiving such an indication, the web browser 136 can cause a side panel 304 to be presented simultaneously with the web page 144. In one example, the side panel 304 is presented after receiving input from the user. In another example, the side panel 304 is presented by default.

[0038] The generative model 112 is provided with at least a portion of the textual content 302, information about the web page 144 provided by the search engine 110, and / or user input. In this example, the user input is "Tell me about tryptophan." The generative model 112 generates a model output based on such input and the content of the web page 144 and / or information about the web page 144 provided by the search engine 110.

[0039] 4 shows another GUI 400 corresponding to a web page 144 displayed by a web browser 136. In this example GUI 400, without receiving any explicit user prompting, the generative model 112 generates a summary of the textual content 302 contained within the web page 144. Thus, a user can fairly quickly read the summary presented by the generative model 112 in the side panel 304 before deciding whether to read the entire textual content 302. The generative model 112 may be configured to automatically provide other information, such as descriptions of images contained within the web page 144, recommendations for filling out forms on the web page 144, etc.

[0040] 5, another communication diagram 500 is shown illustrating the flow of communication between web browser 136, web server 142, interface module 113, and Generative Model 112. At 502, web browser 136 issues a request to web server 142 for web page 144. In this example, web page 144 is a private page, such as a page corresponding to a user's email account, a page corresponding to a user's social media account, or any other page not indexed by search engine 110. At 504, web browser 136 obtains web page 144 from web server 142. Browser 136 receives an indication that the user wishes to interact with Generative Model 112. At 506, web browser 136 transmits the indication that the user wishes to interact with Generative Model 112 to interface module 113. Interface module 113 verifies that web page 144 is a private web page and determines whether the user has consented to providing the content of private web page 144 to Generative Model 112. If it is determined that the user has not provided such consent, the interface module 113 transmits a consent request to the web browser 136, and, for example, graphical data is presented in the browser 136 indicating that the user's consent is being requested.

[0041] The user may consent to the content of the web page 144 being shared with the generative model 112, and at 510, the browser 136 transmits the consent to the interface module 113. The interface module 113 may store a record of the consent so that the user is not prompted for consent each time the browser 136 retrieves the web page 144. In response to receiving the consent, the interface module 113 may request information from the web browser 136. Such information may include the content of the web page 144. At 514, the web browser 136 transmits information to the interface module 113, and such information may include the content of the web page 144. At 516, optionally, when the user indicates user input (such as text input or voice input) to be provided to the generative model 112, the browser 136 may provide the user input to the interface module 113.

[0042] Interface module 113 constructs a prompt at 518 based on the information received by interface module 113 at 514 and 516. Generative model 112 generates a model output based on the prompt received at 518. The model output may be a conversational output, a summary of the content to be displayed on web page 144, etc. At 520, generative model 112 transmits the model output to interface module 113, and at 522, interface module 113 transmits the model output to web browser 136, which presents the model output along with the content of web page 144.

[0043] It is noted that search engine 110 is not represented in communication diagram 500. However, in one example, generative model 112 can generate a query based on the prompt received at 518 and cause the query to be provided to search engine 110 (e.g., by interface module 113). Thus, the prompt used by generative model 112 to generate model output may include information identified by search engine 110. Furthermore, as previously indicated, at least a portion of generative model 112 can be included in web browser 136.

[0044] 6, a GUI 600 of a web page showing a user's email inbox is presented. The GUI 600 may include several selectable buttons 602-610 corresponding to functions related to the email inbox, such as a button related to sending an email, a button related to refreshing the inbox, etc. The GUI 600 may include a pane 612 showing a list of emails in the user's inbox. The GUI 600 may further include a window 614 that may contain the contents of the email being read by the user. A side panel 816 is presented (e.g., as an overlay on or next to the web page shown in FIG. 6) by which the user can indicate inputs to the generative model 112.

[0045] In the example shown in FIG. 6 , the email in window 614 contains information about the user's holidays for the upcoming calendar year. The user can indicate input through side panel 616, and web browser 136 transmits the input to generative model 112 through interface module 113. Additionally, once user consent is received, web browser 136 can provide information from the email shown in window 614 to generative model 112 through interface module 113. Thus, the prompt used by generative model 112 to generate conversational output includes the content of the email. As shown, the user indicates the input, "When are my holidays in May?" When both that input and the content of the email shown in window 614 are provided to generative model 112, generative model 112 can generate an output that accurately corresponds to the input (identifying the user's holidays in May). Traditionally, generative models are unable to respond appropriately to such input because they do not have access to the information requested by the user.

[0046] Referring now to FIG. 7 , another GUI 700 of a web page 144 showing a user's email inbox is presented. The web page includes a text entry field 702 through which a user enters text content. The generative model 112 can be provided with such text content via the interface module 113, and an indication that the user is indicating such content via the text entry field can also be provided. The generative model 112 can output suggestions to the user based on such content, such as "Can I help you complete your email?" When the user indicates that they would like the generative model 112's assistance in completing the email, the generative model 112 can provide a suggested completion that is editable by the user. For example, the user can copy and paste the suggested completion into the text entry field 702 and edit the text to their liking.

[0047] While various examples have been described in which the generative model 112 generates model output based on the content of a web page being presented to a user, it should be noted that the generative model 112 can generate model output based on other information accessible to the web browser 136. For example, the generative model 112 can be provided with a query related to the user's search history (e.g., to find a particular web page within the user's browsing history). In another example, the generative model 112 can be provided with information related to the pages in some threshold number of recent tabs or the current tab as context for use when generating the model output. If the generative model 112 is provided with input and is unable to effectively generate an appropriate answer, the generative model 112 can generate a query to provide to the search engine 110 in connection with obtaining additional context for use when generating the model output. Furthermore, the generative model 112 can be configured to interact with the settings of the web browser 136, such that it can receive input related to the web browser's settings, favorites stored in the web browser, among other information, and update the browser's settings based on the input. Additionally, as previously indicated, the generative model 112 may receive user input regarding an image or video presented within a web page, such as "Who painted that picture?" or "Find the best price on yellow things." In yet another example, the generative model 112 may receive conversational input regarding a web page, such as "Search this website for more things like this," and generate a query to provide to the search engine 110 in connection with identifying similar web pages belonging to the website. According to yet another example, the generative model 112 may receive conversational input, such as "Search for information similar to that in this paragraph," and, as a result, provide an appropriate query to the search engine 110.

[0048] In an even more specific example, for a web page, the Generative Model 112 is provided with the HTML of the web page and / or a rendered image of the web page. The Generative Model 112 may be provided with the main page body as clean text. In another example, the Generative Model 112 is provided with information selected by a user (e.g., when a user highlights a portion of the web page). In another example, entity extraction is performed on the page, and the Generative Model 112 is provided with named entities. In connection with performing such tasks, the HTML (or image) may be converted to text and / or other models, such as object character recognition, object classifiers, image embedding models, etc., may be applied. The Generative Model 112 may be provided with the user interaction history and page content to generate model output. For large documents, techniques may be utilized to summarize portions of the document to form a document index or to deliver one or more chunks / snippets of the document within a limited-size buffer of the Generative Model 112. The Generative Model 112 may output links to other web pages that, when selected, cause the web browser 136 to load the new page. User interactions can be used to navigate within a page or to other pages, or to open pages in new tabs or within the current tab. Answers to questions or passages found in response to a search / exploration query can be highlighted in the main document pane, while a side panel can also have access to a rendered image of the main page to enable conversation, search / exploration, questioning, and answering over non-text elements such as images or drawings. To enable very rich context-sensitive conversations over time, a recent history of all main page content across multiple browser tabs can be accessible to the generative model 112. Reduced representations can be stored to save memory / bandwidth and control the size of prompts. The history can be further compressed by a model that embeds data into a vector representation.More contextual weight can be given to the main page content currently being displayed by scrolling and / or to the parts of the page where the user has spent more time viewing.

[0049] 8, there is shown a functional block diagram of a computing environment 800. The computing environment 800 includes a computing system 101 and a client computing device 102. The memory 134 of the client computing device 102 includes an operating system 802. The memory 134 further includes several applications 804 that run in conjunction with the operating system 802. The applications include a web browser 136, a word processing application, or any other suitable application. The operating system 802 further includes a client interface module 138. Optionally, the operating system 802 may also include at least a portion of a Generative Model.

[0050] The computing environment 800 can operate in a manner similar to the computing environment 100 shown in FIG. 1 . Instead of the client interface module 138 being contained within the web browser 136, the client interface module 138 is contained within the operating system 802. Thus, a pinnable side panel can be displayed within the operating system GUI so that the user is provided with access to the generative model 112 regardless of the application the user is using. Additionally, the client interface module 138 can obtain contextual information about the usage of the client computing device 102 tracked by the operating system 802, such as the identity of the current application being used by the user and the length of time the application has been active. The interface module 113 of the computing system 101 obtains input indicated by the user via the side panel regarding content presented within the application being used by the user and, based on the input, constructs prompts to provide to the generative model 112. The generative model 112 generates model output based on the prompt and provides the model output to the operating system 802 of the client computing device 102 via the interface module 113. Additionally, as described above, the generative model 112 can generate queries based on input received from the interface module 113 and can cause the queries to be submitted to the search engine 110, which performs a search and provides at least a portion of the search results identified based on the search to the generative model 112. The generative model 112 can generate output based on these identified search results.

[0051] 9, a communication diagram illustrating the flow of communication between the operating system 802, the interface module 113, the Generative Model 112, and the search engine 110 is presented. The user of the client computing device 102 indicates an indication that the user intends to interact with the Generative Model 112, and at 902, the operating system 802 transmits the indication to the interface module 113 that the user intends to interact with the Generative Model 112. At 904, the operating system 802 transmits to the interface module 113 the context to be provided to the Generative Model 112. The context may include the identity of the application that is active, the content of such application (assuming the operating system is able to obtain such content), etc. The context may also include information provided by the user, such as conversational input, content from the application that has been selected by the user for presentation to the Generative Model 112 (e.g., copied and pasted by the user), etc. At 906, the interface module 113 provides the context to the Generative Model 112 (formatted in a manner suitable as a prompt that can be provided as input to the Generative Model 112). The generative model 112 generates an output based on the context received at 906. Optionally, the output includes a query generated by the generative model 112 to provide to the search engine 110. At 908, the generative model 112 provides the output to the interface module 113, and if the output includes a query, the interface module 113 provides the query to the search engine 110 at 910. The search engine 110 performs a search based on the query and identifies search results. At 912, the search engine 110 provides at least a portion of the search results to the interface module 113, and at 914, the interface module 113 provides additional context to the generative model 112. The additional context may include at least a portion of the search results identified by the search engine 110 formatted in a manner suitable for consumption by the generative model 112. The generative model 112 generates a model output and provides the model output to the interface module 113 at 916.At 918, the interface module 113 transmits the model output to the operating system 802 for presentation to the user.

[0052] 10 , a GUI 1000 of the operating system 802 is presented. The GUI 1000 has a taskbar 1002 that includes an input field 1004 by which a user can indicate input. The GUI 1000 further includes a display area 1006 that can display GUIs of applications running by the client computing device 102, selectable icons corresponding to the applications, etc. The GUI 1000 also includes a side panel 1008 that can be pinned (opened and closed by the user) within the GUI 1000. A user can interact with the Generative Model 112 through an input field 1010 within the side panel 1008. In one example, the client computing device 102 can run a word processing application, and the contents of the word processing application can be displayed (text). A user can copy and paste text from the application into the input field 1010, causing the text to be provided to the Generative Model 112. This text may be used as at least part of a prompt, for example, a user requesting that the text be summarized, that the text be analyzed for presentation to a particular audience, etc.

[0053] Referring now to FIG. 11 , a high-level illustration of a computing device 1100 usable in accordance with the systems and methodologies disclosed herein is shown. For example, the computing device 1100 may be used in a system configured to provide content displayed in a web browser to a Generative Model as at least a portion of a prompt. As another example, the computing device 1100 may be used in a system configured by an operating system to facilitate user interaction with the Generative Model. The computing device 1100 includes at least one processor 1102 that executes instructions stored in memory 1104. The instructions may, for example, when executed by one or more components discussed above, implement the described functionality or one or more of the methods described above. The processor 1102 can access the memory 1104 via a system bus 1106. In addition to storing executable instructions, the memory 1104 may also store prompts, images, and the like.

[0054] The computing device 1100 additionally includes a data store 1108 accessible by the processor 1102 via the system bus 1106. The data store 1108 may include executable instructions, instant answers, web indexes, etc. The computing device 1100 also includes an input interface 1110 that allows external devices to communicate with the computing device 1100. For example, the input interface 1110 may be used to receive instructions from an external computing device, a user, etc. The computing device 1100 also includes an output interface 1112 that interfaces the computing device 1100 with one or more external devices. For example, the computing device 1100 may display text, images, etc. via the output interface 1112.

[0055] It is contemplated that external devices communicating with computing device 1100 via input interface 1110 and output interface 1112 may be included within an environment that provides virtually any type of user interface with which a user can interact. Examples of types of user interfaces include graphical user interfaces, natural user interfaces, etc. For example, a graphical user interface may accept input from a user using an input device such as a keyboard, mouse, remote control, etc., and provide output on an output device such as a display. Furthermore, a natural user interface may allow a user to interact with computing device 1100 in a manner free from the constraints imposed by input devices such as a keyboard, mouse, remote control, etc. Rather, a natural user interface may rely on speech recognition, touch and stylus recognition on as well as adjacent to a screen, gesture recognition, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, etc.

[0056] Additionally, although illustrated as a single system, it should be understood that computing device 1100 may be a distributed system, such that, for example, several devices may communicate over network connections and collectively perform the tasks described as being performed by computing device 1100.

[0057] The various functions described herein may be implemented by hardware, software, or any combination thereof. If implemented by software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium. A computer-readable medium includes a computer-readable storage medium. A computer-readable storage medium may be any available storage medium accessible by a computer. By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically replicate data magnetically and discs typically replicate data optically with a laser. Furthermore, propagating signals are not included within the scope of computer-readable storage media. Computer-readable media also includes communication media, including any medium that facilitates transfer of a computer program from one place to another. For example, a connection may be a communication medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of communication media. Combinations of the above should also be included within the scope of computer-readable media.

[0058] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0059] Described herein are various features relating to the integration of generative language models into computer-executable applications, by way of at least the following examples.

[0060] (A1) In one aspect, a method performed by a computing system includes generating a prompt for input to a Generative Model, the prompt including content of a web page being presented to a user by a web browser. The method also includes providing the prompt as input to the Generative Model. The method additionally includes receiving output from the Generative Model, the Generative Model generating the output based on the prompt. The method further includes causing a client computing device to present the output to the user in conjunction with the web page being presented to the user.

[0061] (A2) In some embodiments of the method of (A1), the output is displayed in a side panel of the web browser in conjunction with the web browser displaying the contents of the web page.

[0062] (A3) In some embodiments of at least one of the methods of (A1)-(A2), the output is a summary of the content of the web page.

[0063] (A4) In some embodiments of the method of (A3), the summary of the web page is generated automatically by the generative model in response to a web browser loading the web page.

[0064] (A5) In some embodiments of at least one of methods (A1) through (A4), the method also includes receiving input from a user before generating the prompt, the input from the user being indicated by an input field in a sidebar displayed within the web browser, and generating the prompt includes including the input from the user in the prompt.

[0065] (A6) In some embodiments of the method of (A5), the method also includes providing input from a user to a generative model, where the generative model generates a query based on the input from the user. The method further includes providing the query generated by the generative model to a search engine, where the search engine identifies search results based on the query, and the prompt includes at least a portion of the search results identified by the search engine.

[0066] (A7) In some embodiments of at least one of the methods of (A1)-(A6), the method also includes, before generating the prompt, 1) identifying that the web page is not represented in a search engine index; and 2) upon identifying that the web page is not represented in the search engine index, causing a consent request to be presented on the web page simultaneously with the content, wherein the consent request is configured to receive user consent to provide the content on the web page to the generative model, and wherein the prompt is generated after receiving the user consent.

[0067] (B1) In another aspect, a computing system includes a processor and a memory, the memory storing instructions that, when executed by the processor, cause the processor to perform at least one of the methods disclosed herein (e.g., any of methods (A1)-(A7)).

[0068] (C1) In yet another aspect, a computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to perform at least one of the methods disclosed herein.

[0069] What has been described above includes examples of one or more embodiments. Of course, it is not possible to describe every conceivable modification and variation of the above-described apparatus or methodology in order to describe the above-described aspects, but those skilled in the art will recognize that many further modifications and permutations of the various aspects are possible. Accordingly, the described aspects are intended to encompass all such changes, modifications, and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent the term "comprises" is used in the detailed description or in the claims, such term is intended to be inclusive in the same manner as the term "comprises" is interpreted when used as a transitional term in the claims.

Claims

1. 1. A computing system comprising: a processor; When executed by the processor, generating prompts to be input to a generative model, the prompts including content of a web page being presented to a user by a web browser; providing the prompt as an input to the generative model; receiving an output from the generative model, the generative model generating the output based on the prompt; causing a client computing device to present the output to the user in conjunction with the web page being presented to the user; a memory storing instructions that cause the processor to perform actions including A computing system comprising:

2. The computing system of claim 1 , wherein the output is displayed in a side panel of the web browser in conjunction with the web browser displaying the content of the web page.

3. The computing system of claim 1 or 2, wherein the output is a summary of the content of the web page.

4. The computing system of claim 3 , wherein the summary of the web page is automatically generated by the generative model in response to the web browser loading the web page.

5. The said act is receiving input from the user before generating the prompt; 5. The computing system of claim 1, further comprising: a user interface configured to enable a user interface for a user to interact with a web browser; a user interface configured to enable a user interface for a user to interact with a web browser; and a user interface configured to enable a user interface for a user to interact with a web browser.

6. The said act is providing the input from the user to the generative model, the generative model generating a query based on the input from the user; providing the query generated by the generative model to a search engine, the search engine identifying search results based on the query, the prompt including at least a portion of the search results identified by the search engine; and The computing system of claim 5 further comprising:

7. The act, before generating the prompt, Identifying that the web page is not represented in a search engine index; and upon identifying that the web page is not represented in the search engine index, causing a consent request to be simultaneously presented on the web page with the content, the consent request being configured to receive a user's consent to provide the content on the web page to the generative model, the prompt being generated after receiving the user's consent. The computing system of any one of claims 1 to 6, further comprising:

8. 1. A method performed by a computing system, comprising: generating prompts to be input to a generative model, the prompts including content of a web page being presented to a user by a web browser; providing the prompt as an input to the generative model; receiving an output from the generative model, the generative model generating the output based on the prompt; causing a client computing device to present the output to the user in conjunction with the web page being presented to the user. A method comprising:

9. The method of claim 8 , wherein the output is displayed in a side panel of the web browser in conjunction with the web browser displaying the content of the web page.

10. The method of claim 8 or 9, wherein the output is a summary of the content of the web page.

11. The method of claim 10 , wherein the summary of the web page is automatically generated by the generative model in response to the web browser loading the web page.

12. 12. The method of claim 8, further comprising receiving input from the user before generating the prompt, the input from the user being indicated by an input field in a sidebar displayed within the web browser, and generating the prompt comprises including the input from the user in the prompt.

13. providing the input from the user to the generative model, the generative model generating a query based on the input from the user; providing the query generated by the generative model to a search engine, the search engine identifying search results based on the query, the prompt including at least a portion of the search results identified by the search engine; and The method of claim 12 further comprising:

14. Before generating the prompt, Identifying that the web page is not represented in a search engine index; and upon identifying that the web page is not represented in the search engine index, causing a consent request to be simultaneously presented on the web page with the content, the consent request being configured to receive a user's consent to provide the content on the web page to the generative model, the prompt being generated after receiving the user's consent. The method of any one of claims 8 to 13, further comprising:

Images