Chat stream in SERP

Abstract

A computing system including a processor and a memory is disclosed. The memory stores instructions that, when executed by the processor, cause the processor to perform a number of actions. The actions include generating cue words to be input to the generative language model. The cue word comprises session input proposed by the user. The actions also include providing the cue word as input to the generative language model, and receiving a session output from the generative language model, where the generative language model generates the session output based on the cue word. Additionally, the action includes streaming the session output on one of the SERP or the web page to which the user has navigated from the SERP.

Description

Background Art

[0001] Conventional computer-implemented search engines are configured to receive a search query and infer the information retrieval intent of the user who issued the query (e.g., determining whether the user wants to navigate to a specific page, whether the user intends to purchase an item or service, whether the user is looking for facts, whether the user is searching for images or videos, etc.). The search engine identifies results based on the inferred information retrieval intent and returns a search engine results page (SERP) to the computing device employed by the user. The SERP may include links to web pages, text snippets extracted from web pages, images, videos, knowledge cards (graphic items including information about entities such as people, places, companies, etc.), instant answers (graphic items depicting answers to questions posed in the query), widgets (such as a graphical calculator with which the user can interact), supplemental content (e.g., advertisements relevant to the query), etc.

[0002] While search engines are frequently updated with features designed to improve the user experience (and provide users with increasingly relevant results), search engines are not well equipped to provide certain types of information. For example, search engines are not configured to provide outputs that require reasoning about the content of a web page or outputs that are based on several different sources of information. For example, upon receiving a query from a user, "How many home runs did Babe Ruth hit before he turned 30?", conventional search engines return information such as a knowledge card about Babe Ruth (which might depict an image of Babe Ruth, his date of birth, etc.), suggested alternative queries (such as "How many hits did Babe Ruth have in his career?"), and a link to a web page containing statistics. To obtain the answer, the user must visit the web page containing the statistics and calculate the answer themselves.

[0003] In another example, when receiving the query "Give me a list of famous people born in Seattle and Chicago," a conventional search engine returns knowledge cards about the cities of Chicago and Seattle, a link to a first webpage that includes a list of people from Chicago, and a link to a second webpage that includes a list of people from Seattle. However, the search engine cannot reason about the contents of these two webpages to produce a list that includes the identities of people from Chicago and Seattle.

[0004] Recently, generative language models (GLMs) (also known as large language models (LLMs)) have been developed. An example of a GLM is the Generative Pre-Trained Transformer 3 (GPT-3). Another example of a GLM is the BigScience Language Open Science Open Access Multilingual (BLOOM) model, which is also a transformer-based model. In short, a GLM is configured to generate output (such as text, source code, music, video, etc. in a human language) based on a prompt word proposed by a user and in near real time (e.g., within a few seconds of receiving the prompt word). The GLM generates content based on the training data used to train the GLM. Thus, in response to receiving the prompt word "how many home runs did Babe Ruth hit before he was 30 years old", the GLM can output "Babe Ruth hit 94 home runs before he was 30 years old". In another example, in response to receiving the prompt word "provide me a list of celebrities born in Seattle and Chicago", the GLM can output two separate lists of people (one list for Seattle and one list for Chicago), where the list of people born in Chicago includes Barack Obama. However, in these examples, the information output by GLM is incorrect—e.g., Babe Ruth hit more than 94 home runs before he was 30, and Barack Obama was born in Hawaii (not Chicago). Thus, both conventional search engines and GLM are deficient in identifying and / or generating appropriate information in response to certain types of user input. Summary of the Invention

[0005] The following is a brief summary of the subject matter that is described in greater detail herein. This summary is not intended to limit the scope of the claims.

[0006] This document describes various techniques related to providing streaming content by a GLM within a SERP and / or on a web page served from a SERP. Information provided as input to the GLM and used by the GLM to generate output is referred to as a prompt. According to the techniques described herein, the prompt used by the GLM to generate output can include: 1) user input, such as a query; and 2) information from the web page the user is viewing or information retrieved by a search engine. The prompt can also include information from a previous conversation turn, as described in greater detail herein.

[0007] In an example, a browser on a client computing device loads a search engine webpage, and the browser receives a query posed by a user of the client computing device. The browser sends the query to a computing system executing the search engine, and the search engine identifies search results and generates a search engine results page (SERP) based on the query. The search results may include webpages, knowledge cards, instant answers, entity descriptions, supplemental content, etc. related to the query. The search engine returns the SERP to the browser, which is then displayed on a display of the client computing device when the client computing device is accessed.

[0008] In another example, the system organizes and summarizes information from a classic retrieval-based search engine into a semantically meaningful format, making the information more understandable and navigable for search engine users. It does this by first creating a summary that provides an overview of the information from, for example, the top N (e.g., 10 or some other number) search results, and then creating disambiguated subsections about different aspects of the original search query based on their intent. These subsections attribute the summarized information to its source using reference links to provide credibility. The goal of the system is to help users quickly find and understand the information they are looking for by providing a well-curated and structured view of the search engine results page.

[0009] The system retrieves relevant information from a search engine based on the user's search query. The system then uses a GLM to summarize the content based on the intent detected from the query. In some cases, the system can generate a direct answer to the query and provide relevant references to support the information. Additionally, the system uses information from reference documents to provide a brief summary of the main facts or aspects related to the user's query. The model accesses data for the query, such as date and location, as well as the top N web results and surrounding information for each result. The user enters a search query, and the system will use the search engine to retrieve relevant web pages. It then uses a large language model to detect the user's intent, summarize the content from the retrieved documents, generate a direct answer, format the generated content (bold text, cluster content under different headings), cite reference documents, and provide a brief summary of the main facts, events, or aspects of the user's query based on the information from the reference documents. The model is provided with: date and location information and the top N web results (with relevant paragraphs within these results).

[0010] The system described herein goes beyond the capabilities of classic search engines by summarizing and generating answers to user input and providing a concise summary of key facts, aspects, or other disambiguations related to the query. Classic search engines typically only retrieve and rank relevant content based on the user's query without providing additional information or analysis. The described system and method achieve this new capability by leveraging large language models.

[0011] In an example, a search engine receives the query "How many home runs did Babe Ruth hit before he turned 30?" and the search results identified by the search engine include Babe Ruth's date of birth and statistics for Babe Ruth by season. GLM obtains this information along with the query as part of the prompt term. Because the prompt term includes Babe Ruth's season-by-season home run totals, GLM infers this data and provides an output based on the information identified by the search engine as relevant to the query. Thus, GLM may output "Babe Ruth hit 284 home runs before he turned 30." This information may be streamed into a chat window presented on or next to the SERP the user is viewing. In another example, the chat window is presented next to a webpage the user has navigated to from the SERP.

[0012] The technology described herein exhibits various advantages over conventional search engine and / or GLM technology. Specifically, by integrating with a GLM, a search engine can provide end users with information that conventional search engines cannot. Additionally, the GLM described herein is provided with information obtained by the search engine for use in generating output, thereby reducing the likelihood that the GLM will produce factually incorrect or irrelevant output.

[0013] The above summary presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the systems and / or methods discussed herein. This summary is not an extensive overview of the systems and / or methods discussed herein. It is not intended to identify key / critical elements or to delineate the scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that will be presented later. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A functional block diagram of a computing system in accordance with various aspects described herein is shown.

[0015] Figure 2 A computing system with additional elements for providing streaming chat within a SERP or web page is illustrated.

[0016] Figure 3 Illustrated is a GUI of an operating system installed on a client computing device in accordance with aspects described herein.

[0017] Figure 4 Depicted is a GUI displaying a streaming semantic SERP according to various aspects described herein.

[0018] Figure 5 A GUI on a communication device (such as a tablet computer, cellular phone, smartphone, etc.) is shown according to one or more aspects described herein.

[0019] Figure 6 A GUI is shown displaying a web page with a streaming SERP overlaid thereon according to one or more aspects described herein.

[0020] Figure 7 Illustrated is a flow chart depicting a method 700 for providing a streaming experience in a SERP in a computing system according to one or more aspects described herein.

[0021] Figure 8 A high-level diagram of an exemplary computing device that can be used in accordance with the systems and methods disclosed herein is shown. DETAILED DESCRIPTION

[0022] Various techniques for streaming information from a GLM to a SERP on a computing device will now be described with reference to the accompanying drawings, wherein like reference numerals are used throughout to refer to like elements. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it may be apparent that such aspect(s) may be practiced without these specific details. In other examples, well-known structures and devices are shown in block diagram form to facilitate description of one or more aspects. Further, it should be understood that functionality described as being performed by certain system components may be performed by multiple components. Similarly, for example, a component may be configured to perform functionality described as being performed by multiple components.

[0023] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise or clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following instances: X employs A; X employs B; or X employs A and B. Additionally, the articles "a" and "an" used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from the context to be directed to a singular form.

[0024] Further, as used herein, the terms "component" and "system" are intended to encompass a computer-readable data storage device that is configured with computer-executable instructions that, when executed by a processor, cause certain functionality to be performed. Computer-executable instructions may include routines, functions, and the like. It should also be understood that a component or system may be located on a single device or distributed across several devices. Further, as used herein, the term "exemplary" is intended to mean serving as an illustration or example of something and is not intended to indicate a preference.

[0025] Various techniques are described herein for providing streaming chat in SERPs and / or related web pages using a search engine and a generative language model (GLM), also known as a large language model (LLM). The described systems and methods allow search engine results pages (SERPs) to implement streaming content, particularly streaming result summaries, streaming layout and composition, and streaming chat / conversation from a generative language model (GLM), and more generally, from any model or content source, such as images, video generators, voice, and other chained or pipelined models. In another example, real-time conditional (adaptive) text generation is provided that allows for responsive adaptation to dynamic events, as well as other model-level streaming optimizations.

[0026] GLM can take on the order of several minutes to generate a response. Typical GPU load time for a prompt word is on the order of 1000 tokens per second, and output generation time is on the order of 5 to 10 tokens per second. A typical token consists of about 3 English Latin characters on average. A prompt word can be 2000 tokens, and the response can be 1000 tokens. These times are too slow to provide a response on a search engine, which traditionally returns a response in a few seconds at best. In addition, many prompt words are chained in that they call sub-prompt words, make API calls, call search engines, etc. to collect and compose information, which adds more end-to-end latency. According to aspects of this document, many user interface, user experience, and system-level innovations are defined that mitigate most of this latency.

[0027] Now refer to Figure 1 , illustrates a functional block diagram of a computing system 100 according to various aspects described herein. Although illustrated as a single system, it should be understood that computing system 100 may include several different server computing devices, may be distributed across data centers, etc. Computing system 100 is configured to obtain information based on a query posed by a user, and is further configured to provide the obtained information to the GLM as part of a prompt word.

[0028] A client computing device 102, operated by a user (not shown), communicates with the computing system 100 via a network 104. The client computing device 102 may be any suitable type of client computing device, such as a desktop computer, a laptop computer, a tablet computing device, a video gaming system, a virtual reality or augmented reality computing system, a mobile phone, a smart speaker, or other suitable computing device.

[0029] Computing system 100 includes a processor 106 and a memory 108, wherein memory 108 includes instructions for execution by processor 106. More specifically, memory 108 includes a search engine 110 and a GLM 112, wherein the operation of search engine 110 and GLM 112 is described in more detail below. Computing system 106 also includes data repositories 114 through 122, wherein data repositories 114 through 122 store data accessed by search engine 110 and / or GLM 112. More specifically, data repositories 114 through 122 include a web index data repository 114, an instant answer data repository 116, a knowledge graph data repository 118, a supplemental content data repository 120, and a conversation history data repository 122. Web index data repository 114 includes a web index that indexes web pages by keywords included in or associated with the web pages. Instant answer data repository 116 includes an instant answer index indexed by queries, query terms, and / or terms that are semantically similar or equivalent to queries and / or query terms. For example, the instant answer "2.16 meters" may be indexed by the query "how tall is Shaquille O'Neal" (and semantically similar or equivalent queries such as "how tall is Shaquille O'Neal").

[0030] The knowledge graph data store 118 includes a knowledge graph, which includes a data structure about entities (people, places, things, etc.) and their relationships to each other, thereby representing the relationships between entities. The search engine 110 can use the knowledge graph in conjunction with presenting entity cards on search engine results pages (SERPs). The supplemental content data store 120 includes supplemental content that can be returned by the search engine 110 based on a query.

[0031] The conversation history data store 122 includes a conversation history, wherein the conversation history includes conversation information with respect to the user and the GLM 112. For example, with respect to a user, the conversation history may include the identity of the conversation conducted between the user and the GLM 112, input provided by the user to the GLM 112 for multiple conversation turns during the conversation, responses generated by the GLM 112 in the conversation in response to input from the user, queries generated by the GLM during the conversation that were used by the GLM 112 to generate responses, etc. Additionally, the conversation history may include context obtained by the search engine 110 during the conversation; for example, with respect to the conversation, the conversation history 122 may include content from SERPs generated based on queries posed by the user and / or the GLM 112 during the conversation, content from web pages identified by the search engine 110 based on queries posed by the user and / or the GLM 112 during the conversation, etc. Data repositories 114 through 122 are presented to illustrate a representative sample of the types of data accessible to search engine 110 and / or GLM 112 ; it should be understood that there are numerous other data sources accessible to search engine 110 and / or GLM 112 , such as data repositories including real-time financial information, data repositories including real-time weather information, data repositories including real-time sports information, data repositories including images, data repositories including videos, data repositories including maps, etc. Such sources of information may be made available to search engine 110 and / or GLM 112 .

[0032] The search engine 110 includes a web search module 124, an instant answer search module 126, a knowledge module 128, a supplemental content search module 130, and a SERP builder module 132. The web search module 124 is configured to search the web index data repository 114 based on queries received by users, queries generated by the search engine 110 based on queries received by users, and / or queries generated by the GLM 112 based on user interactions with the GLM 112. Similarly, the instant answer search module 126 is configured to search the instant answer data repository 116 based on queries received by users, queries generated by the search engine 110 based on queries received by users, and / or queries generated by the GLM 112 based on user interactions with the GLM 112. The knowledge module 128 is configured to search the knowledge graph data repository 118 based on queries received by users, queries generated by the search engine 110 based on queries received by users, and / or queries generated by the GLM 112 based on user interactions with the GLM 112. Likewise, the supplemental content search module 130 is configured to search the supplemental content data store 120 based on queries received by a user, queries generated by the search engine 110 based on queries received by a user, and / or queries generated by the GLM 112 based on user interactions with the GLM 112 .

[0033] SERP builder module 132 is configured to construct a SERP based on information identified by searches performed by modules 124 through 130. For example, a SERP may include links to web pages identified by web search module 124, instant answers identified by instant answer search module 126, entity cards (including information about the entities) identified by knowledge module 128, and supplementary content identified by supplementary content search module 130. Furthermore, a SERP may include widgets, cards depicting the current weather, and the like. SERP builder module 132 may also generate structured, semi-structured, and / or unstructured data representing SERP content or portions of SERP content. For example, SERP builder module 132 generates a JSON document that includes information obtained by search engine 110 based on one or more searches performed on data repositories 114 through 120 (or other data repositories). In this example, SERP builder module 132 generates data in a structured / formatted manner that will be used by GLM 112 as part of a prompt term.

[0034] As discussed above, the operation of the search engine 110 is improved based on the GLM 112, and the operation of the GLM 112 is improved based on the search engine 110. For example, the search engine 110 is able to provide output that the search engine 110 was previously unable to provide (e.g., based on output generated by the GLM 112), and the GLM 112 is improved by using information obtained by the search engine 110 to generate the output (e.g., information identified by the search engine 110 may be included as part of the prompt words used by the GLM 112 to generate the output). Specifically, the GLM 112 generates results based on the information obtained by the search engine 110 that have a higher likelihood of being accurate than results generated by the GLM 112 that are not based on such information because the search engine 110 is associated with years of design to carefully curate information sources to ensure their accuracy.

[0035] Continue to refer Figure 1 , Figure 2 The computing system 100 is shown as Figure 1 As depicted, computing system 100 includes processor 106 and memory 108, wherein memory 108 includes instructions executed by processor 106. More specifically, memory 108 includes search engine 110 and GLM 112, wherein the above description of Figure 1 The operation of the search engine 110 and the GLM 112 are described in greater detail. The computing system 106 also includes data repositories 114-122, where the data repositories 114-122 store data accessed by the search engine 110 and / or the GLM 112 as described above.

[0036] Search engine 110 includes a web search module 124 , an instant answer search module 126 , a knowledge module 128 , a supplemental content search module 130 , and a SERP builder module 132 .

[0037] In addition to about Figure 1 In addition to the proposed elements, Figure 2 The computing system also includes one or more chat buffers 202 that store response data (e.g., text, images, videos, graphics, etc.) received from GLM 112 for transmission to the client computing device 102 in response to the user query. The SERP builder module 132 uses the response data to generate a SERP, including summary information, instant answers, entity descriptions, search results, supplemental content, etc., as well as a streaming conversation with the user. Before sending the response data to the client device 102, the response data is parsed or chunked into data subsets for streaming. A chunk analyzer module 204 analyzes each data chunk to identify offensive content (e.g., language, images, etc.). If offensive content is identified, the offending content may be removed before the chunk is sent to the client device 102. In another embodiment, the entire chunk may be removed, or the entire response data set to which the chunk belongs may be removed. In these cases, the search system may re-initiate the search in response to the user's query.

[0038] The search system also includes one or more caches 206 that cache user input and responses generated by GLM 112, making responses to relatively frequently requested user input readily available. This feature improves response time. Additionally, the search system includes a synchronization module 208 that synchronizes multiple data streams being rendered using metadata and the like.

[0039] It should be understood that this article is about Figure 1 and Figure 2 The various databases described store cached information, such as, for example, cached web pages or other data sources, for use in responding to chat queries and / or providing search results or other information in SERPs provided to client devices. The cached web pages are periodically updated and / or invalidated to maintain the latest source data.

[0040] Now refer to Figure 3 , illustrates a schematic diagram of a GUI 300 depicting a SERP displayed on a client computing device according to various aspects described herein. The computing device can be any suitable computing device, including but not limited to a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone, a wearable computing device (including a watch or headwear), etc.

[0041] Pointer 310 may be employed in connection with selecting graphical elements displayed in GUI 300 , where the position of pointer 310 is based on a corresponding position of a mouse, a user's gaze position relative to the display, a digital position on a touch-sensitive display, or the like.

[0042] The SERP includes an input field 314 in which a user can provide user input. The user input is submitted to a search engine (not shown), which returns an instant answer 316 (if applicable), an entity description 318 (if applicable), one or more search results 320, and supplemental content 322. An instant answer 316 is, for example, an answer supplied by a search engine in response to a query without the user having to navigate away from the search results 320. The instant answer can be an answer that was previously validated and cached in response to the same user input.

[0043] It should be understood that the illustrated orientation of the input field 314, instant answers 316, entity descriptions 318, search results 320, and supplemental content 322 relative to one another is presented by way of example only and is not intended to limit a particular arrangement of these elements within a SERP. For example, the input field 314 can be presented below the results 320 and entity descriptions 318. In another example, the position of the instant answers 316 can be swapped with that of the entity descriptions 318. In another example, the query field 314, instant answers 316, entity descriptions 318, results 320, and supplemental content 322 can be presented in any order as a vertical stack of fields.

[0044] In one embodiment, a user can hover a pointer 310 over a specific result in the result field 320, and the system will retrieve and present additional content related to the result that the pointer is hovering over in the supplemental content field 322. In another embodiment, when the user hovers the pointer over a specific result, a pop-up window is displayed that shows the source of the information and / or supplemental information, such as an advertisement (image or video) presented on the source page. The hover content can be generated based on the content of the paragraph and the context around the link, as well as the query. The content can be generated in a parallel call that flows asynchronously from the main content.

[0045] exist Figure 3 In the example of , the user hovers the pointer over search result C, and the system retrieves supplementary content C' and C" related to search result C.

[0046] The SERP 312 also includes a conversation field 336 where the user can have a conversation with the GLM 112 ( Figure 1 and Figure 2). The conversation field 336 shows the user input and natural language responses provided to the user. The conversation field 336 also includes a slider 338, via which the user can scroll up and down within the conversation field. Additionally, the conversation field 336 may include one or more chat images 340, wherein the chat images 340 are associated with various natural language responses provided by the system.

[0047] For example, user input is submitted to GLM 112 ( Figure 1 ), the GLM 112 returns a natural language response, image, etc. as an answer to the user's query. The user is then allowed to respond to the responses provided by the GLM 112, just like having a conversation with another person. The GLM 112 then provides the user with a second natural language response, and the conversation continues. At the same time, the user can hover the pointer 310 over one or more of the natural language responses provided by the GLM 112 (see Figure 1 ), and the system will form and submit a query to return supplemental content 322 for presentation on the GUI 300. The supplemental content 322 typically includes one or more selectable links to articles or web pages related to the natural language response over which the pointer 310 is hovered, and may also include one or more supplemental images 334. In another embodiment, the supplemental images 334 are shown as chat images 340 in the conversation field 336. A slider 328 is provided within the supplemental content field 322 and allows a user to scroll through the supplemental content and / or images 334 in the supplemental content field.

[0048] In another embodiment, where the client computing device includes a touch screen, the user can simply swipe up or down using a finger, stylus, or other device. In this embodiment, sliders 324, 326, 328, and 330 become optional. Alternatively, the sliders can be retained or displayed when the user's finger or stylus makes contact with the screen in one of the result field 320, entity description field 318, supplemental content field 322, or main display area 304, respectively. When the user releases the touch screen, the corresponding slider disappears.

[0049] In another embodiment, the client computing device 302 includes a microphone and one or more speakers (not shown) for communicating with the search engine and / or the GLM.

[0050] Now refer to Figure 4, illustrates another GUI 400 for a web page, in which, in an example, a web page is presented when a search result corresponding to the web page is selected. Information 404 is displayed on the web page, including, for example, text, images, selectable graphical icons, etc. Additionally, overlaid on the web page 402 is a semantic SERP 406 constructed by the GLM 112.

[0051] Semantic SERP 406 includes several areas, including a summary field 408 and a summary area 410, which optionally includes one or more images 412, a text description 414, and additional facts 416. Semantic SERP 406 also includes suggestion chips 418, 420 and a text field 422, into which a user can enter text input manually or by voice. Additionally, a graphical icon for a microphone 424 is provided. When a user selects microphone icon 424, the user can speak a query into the text field. Semantic SERP 406 is described in more detail below using examples.

[0052] According to the example, a user enters input regarding Rome, and the search engine 110 returns search results corresponding to web pages that can be selected by the user. A summary field 418 can be generated to say: "It looks like you are interested in Rome. Let me share with you how the Eternal City shaped history:". In one example, the text presented to the user is streamed to the user's display so that it appears as if it is being typed in real time. In the summary area 410, one or more images 412 of Rome and some description 414 of Roman history can be presented. For example, the description 414 can be generated to read: "An overview of Roman history: a fascinating and complex subject spanning thousands of years and covering the rise and fall of one of the most influential civilizations... See more". The phrase "See more" at the end of the description text is clickable by the user to display additional description to the user.

[0053] Additional facts or insights may be displayed in the additional facts field 416. For example, the GLM 112 may generate and display text such as, "Rome was founded by the twins Romulus and Remus in 753 BC. Rome became the capital of Italy in 1870 and has remained so ever since." These additional facts may be listed, numbered, itemized, etc. Additionally, source information may be provided in the additional facts field 416. For example, "Source: rome.net, wikipedia.org." Additionally or alternatively, the source information may be presented in a pop-up or text bubble that appears when the user hovers the pointer 310 over the fact(s).

[0054] Semantic SERP 406 also provides suggestion chips 418 and 420 that users can click to obtain additional information. For example, suggestion chip 418 may have text on it that reads "Roman historical events," while suggestion snippet 420 displays text that reads "Roman historical sites." GLM 112 generates suggestion chips based on user input, web page content, and the like.

[0055] Summary 406 is the conversational response of GLM 112 to the input entered by the user. This response is brief, informative, and illustrates the reasoning capabilities of the system. When rendered, the "Pole" section 426 on the left side of the screen displays a row of "suggestion pills" 428, which are conversation starters generated by GLM 110. When the user is turned around, clicking on any of the suggestion pills takes the user into conversational mode with the selected user input and displays the chatbot's response.

[0056] Suggestion pills represent creative suggestions and help continue a conversation. These suggestions are typically related to ideas and concepts that closely follow the current concept. Therefore, they are highly personalized using the suggestions the user has consumed so far. Unlike PAAs or RS, conversation enablers are somewhat informal and carry context for a longer timeframe. They can also contain referential references to the current input. For example, if the input is "When is the best time to visit the Grand Canyon?", a pill could be "How do I get to the Grand Canyon Visitor Center?" Current technology offers the following types of suggestions: query autocomplete, people also ask questions, and related searches. The primary goal is to help users quickly find what they are looking for. The goal of conversation enablers is to engage users in longer conversations by suggesting highly interesting recommendations that motivate them to explore more content. Highly interesting recommendations engage users in a conversation with the GLM, allowing them to learn more in the process. In one embodiment, conversation enablers can be long suggestions, compared to the typically short current suggestions. In some aspects, the idea is to generate a path of ideas for various concepts. Based on the user's current state on the idea path, the next idea along that path is suggested. For example, a user may have searched and read web pages about concepts such as "artificial neurons" and "multilayer perceptrons." Focusing on the concept of deep learning, it may be desirable to present the user with concepts such as "convolutional neural networks" or "recurrent neural networks." The next pill is generated, maintaining the previously proposed pill, the consumed pill, and the previous user query. Thus, the pill generator provides richer context. Conversational enabler pills should not be bland responses such as "sounds good," "looks good," and so on.

[0057] The SERP session is based on the information in the pole 426 (e.g., stories, creative content, semantic SERPs, etc.). A semantic SERP "peek" can be provided for certain inputs to illustrate the reasoning capabilities of the GLM 112. For example, a peek (a small collection) of semantic SERPs 406 is shown to the user. This is a highly visual storytelling experience in which classic SERP data is reasoned, reorganized, and streamed into a visual storytelling experience with an overview 410 that entices the user to participate in viewing the analysis and links in each category. The semantic SERP is literally "streamed," drawing the user's attention to the active events happening on the page. In one embodiment, the semantic SERP can maintain a constant size after the page load time without user input to reduce page jitter.

[0058] The system generates content that visually flows in character by character, token by token, word by word, or segment by segment. The system backend can segment chunks for speaking or display while streaming. This feature reduces TTS latency. Results can flow in at a random rate as they arrive. In another embodiment, controlled generation using a content buffer is performed. The system can return text word by word, or the response can be spoken word by word.

[0059] The streaming output from one model can be fed into another model. For example, the text produced by a model can flow into the Responsible AI (RAI) component, then into the speech generation model, and so on.

[0060] When the bot is reading a story or generating content, there can be dynamic panes that update with the stream. For example, if it's reading a story, there can be a side pane that displays images generated by the generative model for the current paragraph being streamed. Similarly, an ads pane can update ads based on content currently streaming in another stream (such as a main topic summary). Multiple streams can be synchronized in time as they are rendered.

[0061] SERPs don't require a one-time GET request, but instead contain dynamic content that is continuously fetched and updated by GLM. The connection to the server is maintained until the user navigates away. Images / ads can be rotated as they are generated. Streaming supports both left-to-right and right-to-left languages.

[0062] Supports speculative prefetching of the next input for a given user session. For example, "I thought you might ask." Precomputed / preaggregated results are also available in the semantically matched cache(s). Provides retrospective fact checking, for example, to let users know that the bot's opinion has changed based on new information.

[0063] Other features include: streaming of results for smart lookups on the current page content; streaming results for long documents that are shown in a web browser for a side pane to send chunks of the current page to the model; Map Reduce paradigm using LLM; iterative refinement of (multiple) summary answers to the user as results emerge; different parts of the page that match the query criteria become highlighted as answers flow in; summaries for different parts of the document that can be seen can flow in; the system can send parallel requests to summarize different parts of the document and have them all streamed back in parallel.

[0064] Other features include pop-up text and hidden content that can be generated by the GLM as needed in subsequent on-demand passes. Interactions with certain parts of the page can trigger additional streaming requests; special tokens can be injected into the streaming response for a single prompt word to instruct the model to stop processing the last segment / request and start streaming tokens for a new segment of a large response. For example, the response may have several search results and their summaries, so that the first pass may only generate 100 tokens for each of the 10 summaries. If the user hovers over summary 2 (or clicks on summary 2), the generation can be updated with ACTION-HOVER-2 to start generating more tokens for summary 2. This is similar to the UI metadata conversation of the page (streaming model results) behind the scenes with the user. This metadata can include tokens or soft tokens (embedded (e.g., user context changes)) injected as the streaming progresses. Other actions can include the user clicking on some other part of the page because the robot is generating text / sound that helps refine the user's intent.

[0065] In another embodiment, the GLM 112 is provided with dynamic, real-time response changes based on user feedback. When the results are streamed back as text or voice, the user can insert "Tell me more about X" or "Make it more about Y" to guide the subsequent response in real time. The user can use text, voice, or other methods (gaze tracking, user nodding or shaking head, visual user emotion tracking (smiling, frowning, etc.), EEG, the user performing certain actions in Windows / Edge / Page (e.g., the user asks for help with something, the robot starts explaining how to do it, the user then figures it out, the robot says, "Oh, I see you figured it out now," etc.) to convey updated intent to the system. This feature saves further computation of earlier responses.

[0066] Based on the example dialogue:

[0067] Robot: “There are many breeds of cattle that are raised for various purposes, including milk production, meat production, and heavy work. Some of the main breeds of cattle include:

[0068] 1. Holstein Cow: This breed is known for its high milk production and is the most common dairy cow breed.

[0069] 2. Jersey Cow: This breed is known for the high butterfat content in its milk and is a 3 popular breed used for dairy production.

[0070] Angus cattle: This breed is well known"

[0071] User: "Tell me more about Holstein cows"

[0072] The voice and text models have adaptive ways to gracefully shorten speech or text so that new requests can be processed without abruptly stopping. Smaller, faster models / prompt words can be used to generate fast-turnaround responses while the model searches and composes new responses, such as "Holstein cows? Ah, yes, a great breed..." ... "Wikipedia says they are usually black and white..." Additionally, the robot's responses can be broken down into multiple smaller bubbles to reduce TTS latency.

[0073] Other features include the ability to rephrase Responsible Artificial Intelligence (RAI). Similar adaptive responses can be used when the system regenerates a more desirable response in the event that RAI has been triggered. If the system detects that something already displayed needs to be restated, it can also stream in reverse—i.e., using the backspace key. This can be challenging with voice, in which case the system can say, "Oops, I meant...; this is a better answer..." or some other phrase to correct its error. The voice buffer on the client side can be large enough so that these corrections can occur before the user hears / sees them. Instead of rephrasing, the robot can decide not to answer, or to answer with a more common / safe response that can be predefined.

[0074] With adaptive aggregation, if GLM 112 determines it has sufficient evidence to support a fact, it can stop searching for more information. In database aggregation, or generally in streaming statistics, once the model has sampled sufficient evidence from an IID distribution, it can formulate an answer with specific confidence bounds. Once certain bounds are reached, the model can stop processing. Similarly, once the active information agent has convincing evidence for an answer to a specific sub-question, it can stop.

[0075] Dynamic updates to the ranked list of results provide for asynchronous chatbot requests, such as returning answers later via a conversation or task list and its progress. In another example, the system can stop processing requests after the user has scrolled out of view. If the user is likely to scroll to results outside the current viewport, the system can speculatively start streaming those results.

[0076] According to another feature, eye tracking is used to decide where to focus the production budget. For example, the system determines answers to questions such as: Does this paragraph need more text tokens? What is the best production rate? How fast can the user read / absorb? Information retention can be improved by using appropriate speed, pauses, etc.

[0077] In another embodiment, as the user types, the input is streamed to GLM 112. The model can already figure out what the user wants and give answers more quickly. It can also autocomplete user sentences (autocomplete the question and, if the user accepts the completion, present a tentative answer). For example, "How tall is Barack Obama? Answer: Barack Obama is 6'1". He served as the 44th President of the United States..."

[0078] The bot can actively stream summaries of k real-time streams (conferences, videos, audio, chats, other event streams (sports events, TV shows, etc.)) in less than real time. Thus, a user can participate in the activities of k conferences happening at the same time. A dashboard of real-time semantic summaries can also be provided. The summary can include text and various visual elements. The summary can include a compressed clip of the most interesting moments in the recent basketball game. This feature can allow a user to watch 10 games at a time and not miss any interesting games, or support 10 customers simultaneously via text-based chat.

[0079] The model can start looking up its own output as it generates output to initiate additional background processes: for example, RAI (responsible for the AI classifier), fact-checking processors, etc. This can happen token by token at sentence boundaries, etc. This feature is suitable for multimodal generation (text, images, video, speech, etc.). A buffer is also provided for calling downstream models with output and getting feedback back from downstream models before the user sees it. If RAI detects a problem, it can dynamically send that feedback to the streaming output / input of the first model, causing the response to change in a more desired way. There can be many processes generated simultaneously that merge these near-real-time feedback signals.

[0080] There can also be complex workflows where data is streamed into (multiple) models. An ad generated by one expensive model can be streamed into a second, expensive model that is also streaming. This page can visually show multiple streams from different models and model calls. Similar to the above, some of these streams can be internal and only seen by other models.

[0081] Pipelining multiple prompts enables a chain of n prompts to be rendered in time t instead of time t*n, where t is the complete generation time. At the hardware and system level, we allow multiple standing queries to communicate with each other. Given two prompts a and b, where b depends on the output of a, both prompts can start loading in parallel. As output from a arrives, it can be appended / loaded into b, and once enough output has been received, b can begin generation.

[0082] There can be hotspots where certain experts in a mixture of experts (MoE) model become overloaded. Parallel copies of the experts can be made to improve throughput. Furthermore, distilled models can be employed that are specialized in their functionality while requiring fewer computational resources (and sometimes even time) than a general-purpose GLM. For example, a distilled GLM can be configured to perform text summarization (but not query generation). Another distilled GLM can be configured to generate queries (but not text summarization).

[0083] For iteratively generated content, such as diffusion models, intermediate steps in the computation can be streamed. For example, during image streaming, a single image can start as white noise, but then as the model iterates, it sharpens into a picture of a cat.

[0084] The fill-in method can also flow. If the user pans the image, the model can begin to fill in the missing areas, generating new details that are consistent with the original image. Similarly, if the image is scaled, the scaled image can also flow in.

[0085] Generated text for writing assistance can also be streamed in. This streaming can occur within sentences or between existing paragraphs, for example. This is an example of text "repair," where the user wants the model to fill in areas. Users can initiate multiple simultaneous requests, for example, "Add a paragraph about pit bull health issues after the paragraph about dog breed costs" or "Add an introductory paragraph to this buying guide for dog breeds." These can all be streamed in simultaneously.

[0086] There can be many UI elements to indicate when a bot or page is doing something. "..." or "I was wondering" or "Search for x" or "Find prices" or "Find the best deals for you". The description of what the system is doing can be generated by the model and depends on the query / task.

[0087] There can be more than two participants in a session, and possibly more than one bot. This feature can address situations where there are long prompts and long responses, but instead of timing out and discarding all partially generated tokens, the system can use the partial output and complete the job.

[0088] In another embodiment, a "home" SERP is provided, offering an experience similar to watching a television news program, where users can interact with the news anchor. For example, the news anchor can be represented as a dynamically generated avatar generated in near real time. Transitions between queries can be animated, such as "Now switching to the weather forecast" when a user asks about the weather. The system can ask users questions to help guide the interactive broadcast. In this regard, the system acts as a virtual operating system, where users can request different information to be displayed, summarized, and so on. Some jobs may require a long time to process; in this case, the system can say, "Now back to the question about cows..." and transition. This stream can include metadata for synchronization with other streams. For example, voice tone, facial expressions, style, and so on can be input to other widgets, models, avatars, or renderers that process or combine one or more streams. For example, when rendering a specific sentence, a hint may appear to flip to an advertisement. The page can also have multiple open connections to pull different streams. Some streaming can occur on the client, while some stream processing can occur on the server.

[0089] Now refer to Figure 5 , according to one or more aspects described herein, a GUI 500 is illustrated on a communication device 502 (such as a tablet computer, cellular phone, smartphone, etc.). GUI 500 includes a SERP interface 504, which includes a query field 506, an instant answer field 508 (when applicable), an entity description field 510 (when applicable), and a result field 512. One or more results (labeled A through D) returned in response to the query entered into query field 506 and, optionally, one or more images 514 are displayed in result field 512. A user clicks on one of the returned results A through D, and the device displays the received information associated with the selected result. Additionally, the system retrieves supplemental content 516 (e.g., additional articles, hyperlinks, images, advertisements, etc.) related to the selected result and displays the supplemental content on SERP interface 504.

[0090] It should be understood that the specific order of the query field 506, instant answer 508, entity description 510, result field 512, and supplemental content field 514 is not limited to Figure 5 Rather, the elements may be arranged in any order. Furthermore, the elements depicted in the SERP interface 504 are not limited to Figure 5 Rather than being arranged in a stacked arrangement as shown, the devices may be arranged side by side, in a grid arrangement, etc.

[0091] The communication device 502 also includes a microphone 518 and one or more speakers 520, through which the user can enter voice commands and receive audio from the communication device. For example, the user can activate the microphone by speaking the word "query" or "question" and then speak a word or phrase that the user can otherwise manually enter into the query field 506 to initiate a query. The result field 512 can be populated with results (e.g., hyperlinks, article titles, images 512, etc.) in response to the user's voice query. In another embodiment, the results can be read out and presented to the user as audio output via the speaker(s) 520.

[0092] In another embodiment, a voice-activated graphical icon (not shown) may be provided in the query field 506 or elsewhere in the SERP interface 504. When the user selects (e.g., clicks or long-presses) the voice-activated graphical icon, the user is prompted to begin speaking and may speak a natural language query into the microphone 518. One or more of the returned instant answers 508, entity descriptions 510, results 512, and / or supplemental content 514 may be presented to the user as audio output via one or more speakers 520.

[0093] The GUI also includes a conversation field 522 that illustrates a conversation between the user and the GLM and / or search engine. Included in the conversation field 522 are the user query and corresponding natural language response, as well as one or more chat images 524 (i.e., images included in the chat or conversation field).

[0094] In another embodiment, the surf 504 displayed on GY500 is a semantic SERP, such as a Figure 4 Semantic SERP 406 of description.

[0095] Now refer to Figure 6 , shown on the GUI 500 of the communication device 502 is a web page 602 that includes displayed information 604 and includes text, images, selectable graphic icons, selectable text, etc. Also shown are supplemental content 606 and a semantic SERP 608. The semantic SERP 608 may be associated with information about Figure 4 The described semantic SERP 406 is similar or identical. Supplemental content 606 can be retrieved based on the user's indicated interest in the information provided in the semantic SERP.

[0096] The user is also allowed to have a text or voice conversation, which is shown in the conversation field 610 of the semantic SERP. The user query and the natural language response generated by the system are displayed to the user as a conversation in the conversation field 604. An example of a conversation dialog that can be displayed in the conversation field 610 (or the conversation field of any of the aforementioned figures) is provided below.

[0097] Query 1: In which state is Ann Arbor located?

[0098] Response 1: Ann Arbor is located in Michigan, United States.

[0099] Query 2: Tell me more.

[0100] Response 2: Ann Arbor is a city in southeastern Michigan, approximately 35 miles (56 km) west of Detroit. It is the county seat of Washtenaw County and is best known as the home of the University of Michigan, one of the oldest and most prestigious public universities in the United States.

[0101] Query 3: What SAT scores do universities require?

[0102] Response 3: The University of Michigan requires students to submit SAT scores as part of their application. The middle 50% range for the Class of 2025 on the SAT is 1340 to 1470.

[0103] As can be seen, the responses generated by the system take into account the context of the conversation. For example, when the user mentioned "university" in query 3, the system inferred that the user was referring to the University of Michigan based on the context of response 2. The communication device 502 also includes a microphone 518 and one or more speakers 520, which allow the user to speak queries during the conversation and listen to responses, as described above with respect to Figure 5 Descriptive.

[0104] Supplemental content 610 is retrieved using the context of the session and may include additional links, images, selectable graphical icons, etc., on which the user can click for additional information. For example, the content may include, but is not limited to, links to one or more hotels in the Ann Arbor area, restaurants in Ann Arbor, links to purchase tickets to a University of Michigan sporting event, etc.

[0105] Figure 7 The diagram illustrates a method related to providing streaming information within a SERP, according to one or more embodiments described herein. Although the method is illustrated and described as a series of actions performed in sequence, it should be understood and appreciated that the method is not limited by the order of the sequence. For example, some actions may occur in an order different from that described herein. In addition, an action may occur concurrently with another action. Furthermore, in some instances, not all actions may be required to implement the method described herein.

[0106] Furthermore, the actions described herein may be computer-executable instructions that can be implemented by one or more processors and / or stored on one or more computer-readable media. Computer-executable instructions may include routines, subroutines, programs, execution threads, etc. Furthermore, the results of the method actions may be stored on a computer-readable medium, displayed on a display device, etc.

[0107] Now go to Figure 7 , depicts a flow chart of a method 700 for providing a streaming experience in a SERP in a computing system according to one or more aspects described herein. At 702, user input is received at a search system. At 704, search results are generated for the user query. At 706, the query and the generated search results are sent to the GLM. At 708, GLM narrative results based on the query and the provided search results are received. At 710, the search results and the GLM narrative results are streamed to a client device for presentation in the SERP.

[0108] At 712, an indication of a user interaction with a SERP is received. The user interaction can be a user clicking a selectable icon, word, or phrase presented in the SERP, or a user voice command, such as "Tell me more about this," during the initial streaming of a particular portion of streaming data into the SERP. At 714, additional search results are retrieved based on the user's interest as determined by the indication of the user interaction with the SERP. At 716, a new query is generated for the GLM based on the user interaction. At 718, an updated GLM response narrative is received. At 720, the updated search results and the updated GLM response narrative are streamed to the SERP on the client device.

[0109] Continue to refer Figures 1 to 7 , various additional contemplated features and aspects are described below. In one embodiment, the streaming HTML may include multimedia (MM) content. The MM content itself may be generated by the GLM and may flow in as it is generated, for example, starting with a white noise image to a fine detail image. The SERP may be a full page with many sections and / or may integrate multiple elements into a dialog response as shown above. Some initial content may be generated that is visible, but as the user scrolls down or otherwise interacts with the UI, additional content may continue to flow in and be displayed.

[0110] Results from multiple paragraphs can flow in in parallel. When the user scrolls a paragraph out of view, it can stop flowing, saving GPU resources. The user can still quickly scan the initial sentences of multiple results. Priority signals for the various streams can be dynamically sent to the model to prioritize GPU power for more important streams. Certain streams can be paused, deprioritized, or canceled.

[0111] Users can switch to different streams using certain widgets, such as "next page" for text and image results. The system can also automatically switch to give users a preview of the first line of each page.

[0112] The system caches results for faster searching and retrieval. While retrieving results, there can be animated icons / avatars showing the system "thinking." Status for subtasks can be streamed in to indicate progress to the user. Some parts of the page can load quickly while others stream in. A full-page optimizer controls the display of streaming results to avoid a jarring UX where results constantly move across the user.

[0113] A classifier can be employed to determine which model to use. Some responses may not be worth the expensive model and can be served by a lower latency model.

[0114] In another embodiment, a workflow engine (not shown) is provided for parallelizing tasks, such as a MapReduce job optimizer (not shown), which determines which tasks or operations can be run in parallel.

[0115] We also consider prompt words that can generate extensions, reach checkpoints, insert newly arrived streaming content, continue generation, etc. This feature allows input to arrive at the prompt word template out of sequence.

[0116] The described systems and methods can also provide adaptive refinement from certain models: for example, iteratively generated images; frame-by-frame generated videos; dynamically generated music; dynamically generated sounds; Excel-like spreadsheets where each aggregate unit is asynchronously populated; dependency graphs between units where certain units can extract, summarize, etc.; Map-Reduce dependency graphs; workflow engines, etc.

[0117] In another example, the model itself can be used to plan / optimize / schedule workflows, including running tasks in parallel and asynchronously, and to decide which sub-prompts to invoke. In another example, the system can generate a voice conversation using k different voices. This feature helps provide users with certain experiences, such as hosting dueling newscasters or podcast interview simulations, providing two perspectives read by different robot speakers, narrating a story with k different characters speaking different languages, and providing multimedia video content with multiple speakers. This feature also helps provide bot2 with the adaptive ability to interrupt bot1 mid-sentence with fully context-sensitive interjections.

[0118] The system can also allow the displayed text to be more complex / detailed, while the voice only speaks a brief summary of the content (which may not appear on the screen). For example, a mixed mode result includes text (full page) + a voice summary of the content.

[0119] In another example, the system provides streaming results in a sidebar as the user navigates, while facilitating zero-shot suggestions, reviews, recommendations, and the like.

[0120] The system provides the ability to attend k virtual meetings simultaneously, and can employ summaries to shorten live content (e.g., text or audio summaries), allowing users to attend multiple meetings in parallel. This feature can be extended to watching k TV channels or k sporting events. Narratives can be seamlessly spliced when switching between running commentaries of different event streams.

[0121] According to another aspect, the system can synchronize the current streaming content from the model with the next instruction being issued by the user at the same time. For example, if the user starts talking about something that is still streaming, when the GLM 112 generates the next response, it can use the synchronization information to infer what is meant by "that one", that is, what stream the user was seeing when he said "that one". This feature can be applied to any streaming content - text, video, etc. In one embodiment, this functionality is provided by the synchronization module 208 ( Figure 2 )supply.

[0122] In another embodiment, the system provides adaptive streaming. For example, if a user hovers over an image, the overview narration can capture this event and then, like a salesperson or teacher, adapt the subsequent commentary of the page in near real time to focus on what the user is paying attention to. The user can focus on things through any human-computer interface method.

[0123] Certain features also apply to augmented reality reviews. SERPs can be overlaid in VR or AR. There can be a continuous stream of "queries," allowing the model to always accumulate context and adapt the content / reviews it generates. The ground edge sidebar can also be extended to ground VR / AR content.

[0124] In another embodiment, the system provides a streaming "summary" of an event stream, such as updating a log graph of a financial stock. The model can perform a streaming log computation of the original input stream. The running generation tokens from the prompt word can be checkpointed to generate a histogram where the last unfilled bucket shows the latest data point. Once the bucket window passes, the latest data point is aggregated and the generation / hint is deferred. In this respect, the operation is similar to reading and writing hints. For example, [1,2,3,4],5,6=>[1,2,3,4],[5,6,7,8],9,10=>, etc. The checkpoint can be stored on the continuation of the prompt word / generation.

[0125] In another example, when listening to k conferences, the user can select which streams the user wants to actively listen to in audio or text form and which streams the user does not want to participate in. The combined summary stream can tell the user more about the subset that the user is most interested in, and occasionally give the user a higher-level summary of what is happening in other streams. This feature can also be applied to asynchronous information requests that the system is processing. For example, when the user asks the system to research k things, and the system has generated k subtasks, the system can give the user a summary of the k things the system is doing. That is, the system can tell the user how things are going, what the system wants to do next, and so on. The user can guide where the system directs its attention and search / processing.

[0126] When generating responses, the system knows how much time the user has to review the response (e.g., whether the user is busy, driving, what device the user is using, etc.). The system can generate responses of appropriate length, along with a high-level summary, asking the user where they would prefer more details, etc. For those places where the system anticipates the user might request more details, it can already start pre-generating responses.

[0127] This system can provide running commentary on basketball or football games. It can listen to AM radio stations, perform speech-to-text conversion, and then generate more interesting, stylized, and shortened commentary. It can listen to k different radio stations discussing the same event and integrate the commentary. It can continuously inform the user about the stock market, stocks of interest, different industries, any other news stories of interest, any other world news or ongoing events, and so on. It can create a personalized Bloomberg TV feed for the user. It can generate a dynamic mix of information types, such as charts, text, and voice. It can dynamically watch and mix video content streaming from k televisions or media sources.

[0128] In another example, feedback is given to the user regarding the quality or usefulness of the search results and / or chat conversation narrative, such as a "thumbs up" or "thumbs down" response, a rating (e.g., on a scale of 1 to 5), etc.

[0129] In another embodiment, a user can submit a URL, etc. as a query and ask the system to summarize its content. The system then analyzes the content of the URL and returns a streaming summary of the content to the user. Similarly, a user can initiate a chat feature while reading an email and ask the system to summarize the content of the email or its associated document.

[0130] Other features include voice-driven search, visual question-answering on visual content, robot tagging comments (icons plus comments) for objects on the page, voice comments on user interactions with page elements, adaptive generation based on user interaction and attention, conversation-driven interaction with content on the SERP page, wrappers or right-rail overlays for interaction with other UIs (email search, SharePoint search, etc.) for other content.

[0131] Additional features may include: conversation-driven interaction with dynamically generated web pages; dynamic layout reflow; dynamic editing of elements into the conversation history; full-page transitions of conversation elements within a conversation; weather elements within a conversation; the ability to click on an answer card to switch back to a full-page portal / detailed / non-mini version of the element; the ability to switch between conversation elements and the full experience mode; and more. For example, a user can switch from a shopping answer to a full-page shopping page. Other features include: answer / exploded view; the ability to set answers aside within a conversation; pin them; place them in a new browser tab; and more. Additionally, an "expand" button may be provided for switching answers to an alternative expanded mode.

[0132] Additionally, options can be provided for multi-page conversations. For example, a given model state can include conversations / interactions across multiple tabs / pages / searches.

[0133] Other features include the ability to display the content of a complete news article as an answer to add context to the conversation. For example, the system may retrieve a news article but only display the headline and image of the news article. In this scenario, the user can then ask a subsequent question that utilizes the entire content of the news article and / or prioritizes the retrieved content to generate the next response.

[0134] In another embodiment, search results in conversation mode may be delivered as web result answer cards and / or semantic summary answer cards.

[0135] The described systems and methods also provide the following capabilities: sharing conversations with others; allowing multi-party conversations; saving conversations for later resumption; bookmarking conversations; saving the complete conversation history for later review; setting timestamps for conversation sessions so that the next response can leverage recent conversation history across multiple windows / sessions; sharing conversation turns widely (e.g., on social media); integrating mixed-mode external content / conversations into enterprise chat applications (e.g., Teams, Skype); providing upsells to Sapphire application experiences, etc.

[0136] In another example, the described systems and methods facilitate providing a "new tab page" that includes one or more of: a "what's new" summary, asynchronous updates about user data, user interests, what's happening in the world, etc. Email can be another experience that provides mini-answer / expand / full-page modes. For email, sub-answers can be personal emails, information about people, people answers / cards, etc. "Expand" mode can launch a new window or tab for composing / sending email replies. The answer can include a short list of related emails or SharePoint items. Expand mode can also transition to a full-page Word document for document results. The option to switch back to a session with mini-answer mode is also contemplated.

[0137] Now refer to Figure 8 , illustrates a high-level diagram of an exemplary computing device 800 that can be used in accordance with the systems and methods disclosed herein. For example, computing device 800 can be a client computing device having an operating system stored thereon, wherein the operating system provides streaming functionality for presenting SERPs to a user on the client device. By way of another example, computing device 800 can be a server computing system that provides streaming search presentation functionality. Computing device 800 includes at least one processor 802 that executes instructions, the instructions being stored in memory 804. For example, the instructions can be instructions for implementing functionality described as being performed by one or more of the components discussed above or instructions for implementing one or more of the methods described above. Processor 802 can access memory 804 via a system bus 806. In addition to storing executable instructions, memory 804 can also store content, graphical icons, profile information, and the like.

[0138] The computing device 800 additionally includes a data repository 808 accessible by the processor 802 via the system bus 806. The data repository 808 may include executable instructions, graphical icons, profile information, content, and the like. The computing device 800 also includes an input interface 810 that allows external devices to communicate with the computing device 800. For example, the input interface 810 may be used to receive instructions from an external computer device, from a user, and the like. The computing device 800 also includes an output interface 812 that interfaces the computing device 800 with one or more external devices. For example, the computing device 800 may display text, images, and the like via the output interface 812.

[0139] It is contemplated that external devices communicating with the computing device 800 via the input interface 810 and the output interface 812 may be included in an environment providing substantially any type of user interface with which a user can interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and the like. For example, a graphical user interface may accept input from a user employing (a plurality of) input devices (such as a keyboard, mouse, remote control, etc.), and output may be provided on an output device (such as a display). In addition, a natural user interface may enable a user to interact with the computing device 800 in a manner that is not constrained by the constraints imposed by the input devices (such as a keyboard, mouse, remote control, etc.). In contrast, a natural user interface may rely on speech recognition, touch and stylus recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, sound and speech, vision, touch, gestures, machine intelligence, and the like.

[0140] Additionally, although illustrated as a single system, it should be understood that computing device 800 may be a distributed system. Thus, for example, several devices may communicate via a network connection and may jointly perform the tasks described as being performed by computing device 800.

[0141] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium. Computer-readable media include computer-readable storage media. Computer-readable storage media can be any available storage medium that can be accessed by a computer. By way of example and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage devices, magnetic disk storage devices, or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks (BDs), wherein disks typically copy data magnetically, while optical disks typically copy data optically using lasers. Further, the propagated signal is not included within the scope of computer-readable storage media. Computer-readable media also include communication media, which include any media that facilitates transferring a computer program from one place to another. For example, a connection can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of communications media. Combinations of the above may also be included within the scope of computer-readable media.

[0142] Alternatively or in addition, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0143] Various techniques are described herein according to at least the following examples.

[0144] (A1) In one aspect, a computing system is described herein. The computing system includes a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform actions. The actions include receiving a query provided by a user and generating prompt words to be input to a generative language model, wherein the prompt words include instructions for the generative language model to generate conversational output based on the query. The actions also include providing the prompt words as input to the generative language model, wherein the generative language model generates conversational output based on the prompt words. The actions also include streaming the conversational output to a search engine results page (SERP) or a webpage to which the user has navigated from the SERP, so that the conversational output appears to the user as if it were being typed in real time.

[0145] (A2) In some embodiments of the computing system of (A1), generating the prompt word further comprises providing a query to a search system, wherein the search system identifies search results based on the query and includes a portion of the search results in the prompt word.

[0146] (A3) In some embodiments of the computing system of (A2), the actions further include receiving an indication of a user interaction with the SERP in response to the conversation output. The actions further include retrieving additional search results in response to the user interaction. Furthermore, the actions include generating and sending updated prompt words to the generative language model based on a portion of the additional search results and the user interaction. The actions further include receiving the updated conversation output from the generative language model. Additionally, the actions include streaming the updated search results and the updated conversation output to the SERP on the computing device.

[0147] (A4) In some embodiments of the computing system of at least one of (A1) to (A3), the actions further include providing supplemental content related to the session output for display on a dynamically updateable display pane on the computing device. The actions further include updating the supplemental content in response to at least one of additional user interaction and the updated session output.

[0148] (A5) In some embodiments of the computing system of (A4), the actions further include synchronizing the streaming of the supplemental content to the session output by comparing metadata associated with the session output and the supplemental content.

[0149] (A6) In some embodiments of the computing system of at least one of (A1) to (A5), the actions further include identifying offensive content in at least one of the session output and the supplemental content. The actions further include, before rendering, performing at least one of: removing the offensive content; and removing the source of the offensive content.

[0150] (A7) In some embodiments of the computing system of at least one of (A1) to (A6), the actions further include caching the generative language model response for retrieval in response to similar queries.

[0151] (B1) In another aspect, a computing device is described herein. The computing device includes a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform an action. The action includes sending a user query to a search system. The action also includes receiving conversation output from a generative language model, wherein the generative language model generates the conversation output based on prompt terms that include the query. The action also includes streaming the conversation output on a search engine results page (SERP) or a webpage to which a user has navigated from the SERP, such that the conversation output is presented to the user as if it were being typed in real time.

[0152] (B2) In some embodiments of the computing device of (B1), the session output includes information related to search results responsive to the query.

[0153] (B3) In some embodiments of the computing device of at least one of (B1) to (B2), the actions further include sending an indication of the user interaction with the SERP in response to the conversation output. The actions further include receiving updated search results and updated conversation output in response to the user interaction.

[0154] (B4) In some embodiments of the computing device of (B3), the actions further include displaying the streamed updated search results and the updated conversation output on the SERP.

[0155] (B5) In some embodiments of the computing device of at least one of (B1) to (B4), the actions further include receiving and displaying, on a dynamically updateable display pane on the SERP, supplemental content related to the conversation output. The actions further include receiving and displaying updated supplemental content in response to at least one of additional user interaction and updated conversation output.

[0156] (B6) In some embodiments of the computing device of (B5), the actions further include synchronizing the streaming of the supplemental content to the session output by comparing metadata associated with the session output and the supplemental content.

[0157] (B7) In some embodiments of the computing device of at least one of (B1) to (B6), the actions further include identifying offensive content in at least one of the session output and the supplemental content. Additionally, the actions include, before rendering, performing at least one of: removing the offensive content; and removing the source of the offensive content.

[0158] (C1) In another aspect, a method performed by a computing device is described herein. The method facilitates providing conversational interaction on a search engine results page presented on an interface of the computing device. The method includes receiving a query provided by a user. The method also includes generating a prompt word to be input to a generative language model, wherein the prompt word includes instructions for the generative language model to generate conversational output based on the query. The method also includes providing the prompt word as input to the generative language model, wherein the generative language model generates the conversational output based on the prompt word. Additionally, the method includes streaming the conversational output to one of the search engine results pages (SERPs) or a webpage to which the user has navigated from the SERP, such that the conversational output appears to the user as if it were being typed in real time.

[0159] (C2) In some embodiments of the method of (C1), generating the prompt word further comprises providing a query to a search system, wherein the search system identifies search results based on the query and includes a portion of the search results in the prompt word.

[0160] (C3) In some embodiments of the method of (C2), the method further includes receiving an indication of a user interaction with the SERP in response to the conversation output. The method further includes retrieving additional search results in response to the user interaction. The method further includes generating and sending updated prompt words to the generative language model based on a portion of the additional search results and the user interaction. Furthermore, the method includes receiving the updated conversation output from the generative language model. Additionally, the method includes streaming the updated search results and the updated conversation output to the SERP on the computing device.

[0161] (C4) In some embodiments of the method of at least one of (C1) to (C3), the method further includes providing supplemental content related to the session output for display on a dynamically updateable display pane on the computing device. The method further includes updating the supplemental content in response to at least one of additional user interaction and the updated session output.

[0162] (C5) In some embodiments of the method of (C4), the method further includes synchronizing the streaming of the supplemental content to the session output by comparing metadata associated with the session output and the supplemental content.

[0163] (C6) In some embodiments of the method of at least one of (C1) to (C5), the method further includes identifying offensive content in at least one of the session output and the supplemental content. The method further includes, before rendering, performing at least one of: removing the offensive content; and removing the source of the offensive content.

[0164] (D1) In another aspect, described herein is a method performed by a computing device, wherein the method includes any of the actions set forth in embodiments (B1) to (B7).

[0165] What has been described above includes examples of one or more embodiments. Of course, for purposes of describing the aforementioned aspects, it is not possible to describe every conceivable modification and alteration of the above apparatus or methods, but one of ordinary skill in the art will recognize that many further modifications and permutations of the various aspects are possible. Accordingly, the described aspects are intended to include all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term "include" is used in the detailed description or the claims, such term is intended to be inclusive in a manner similar to how the term "comprising" is interpreted when employed as a transition word in a claim.

Claims

1. A computing system comprising: processor; as well as a memory storing instructions that, when executed by the processor, cause the processor to perform actions comprising: receiving queries provided by users; generating prompt words to be input to a generative language model, wherein the prompt words include instructions for the generative language model to generate a conversational output based on the query; providing the prompt word as input to the generative language model, wherein the generative language model generates a conversational output based on the prompt word; and The session output is streamed to a search engine results page (SERP) or one of the web pages to which the user has navigated from the SERP, such that the session output appears to the user as if it were being typed in real time.

2. The computing system of claim 1, wherein generating the hint word further comprises providing the query to a search system, wherein the search system identifies search results based on the query and includes portions of the search results in the hint word.

3. The computing system of claim 2, wherein the actions further comprise: receiving, in response to the conversation output, an indication of a user interaction with the SERP; responsive to the user interaction, retrieving additional search results; generating and sending an updated prompt word to the generative language model based on the portion of the additional search results and the user interaction; receiving updated conversation output from the generative language model; as well as The updated search results and the updated conversation output are streamed to the SERP on the computing device.

4. The computing system according to at least one of claims 1 to 3, wherein the actions further comprise: providing supplemental content related to the session output for display on a dynamically updateable display pane on the computing device; as well as The supplemental content is updated in response to at least one of additional user interaction and updated session output.

5. The computing system of claim 4, wherein the actions further comprise: Streaming of the supplemental content is synchronized to the session output by comparing metadata associated with the session output and the supplemental content.

6. The computing system according to at least one of claims 1 to 5, wherein the actions further comprise: identifying offensive content in at least one of the session output and the supplemental content; as well as Before rendering, do at least one of the following: remove said offensive content; as well as Remove the source of the offensive content.

7. The computing system according to at least one of claims 1 to 6, wherein the actions further comprise: Generative language model responses are cached for retrieval in response to similar queries.

8. A method for facilitating providing conversational interactions on a search engine results page presented on an interface of a computing device, the method comprising: receiving queries provided by users; generating prompt words to be input to a generative language model, wherein the prompt words include instructions for the generative language model to generate a conversational output based on the query; providing the prompt word as input to the generative language model, wherein the generative language model generates a conversational output based on the prompt word; as well as The session output is streamed to a search engine results page (SERP) or one of the web pages to which the user has navigated from the SERP, such that the session output appears to the user as if it were being typed in real time.

9. The method of claim 8, wherein generating the cue word further comprises providing the query to a search system, wherein the search system identifies search results based on the query and includes portions of the search results in the cue word.

10. The method according to claim 9, further comprising: receiving, in response to the conversation output, an indication of a user interaction with the SERP; responsive to the user interaction, retrieving additional search results; generating and sending an updated prompt word to the generative language model based on the portion of the additional search results and the user interaction; receiving updated conversation output from the generative language model; as well as The updated search results and updated session output are streamed to the SERP on the computing device.

11. The method according to at least one of claims 8 to 10, further comprising: providing supplemental content related to the session output for display on a dynamically updateable display pane on the computing device; as well as The supplemental content is updated in response to at least one of additional user interaction and updated session output.

12. The method according to claim 11, further comprising: Streaming of the supplemental content is synchronized to the session output by comparing metadata associated with the session output and the supplemental content.

13. The method according to at least one of claims 8 to 12, further comprising: identifying offensive content in at least one of the session output and the supplemental content; as well as Before rendering, do at least one of the following: remove said offensive content; as well as Remove the source of the offensive content.

Images