Contextual search tools in the browser interface

Abstract

A browser-based tool for providing context-based assistance during web browsing is disclosed. An exemplary method includes receiving a contextual search request related to main content displayed in a browser's display area, extracting content from the main content, receiving contextual suggestions based on the extracted content, and displaying the contextual suggestions within a designated contextual search area in the browser. This innovative approach streamlines the search process by providing users with relevant suggestions based on the content they are currently viewing, thereby improving efficiency in navigating online information.

Description

[Technical Field]

[0001] Websites provide useful information or functionality to users, and many users use the Internet to research products, places, companies, services, browse social media or news feeds, and the like. Summary of the Invention

[0002] Embodiments relate to a browser-integrated tool that can provide additional context for main content, suggest content related to main content, summarize main content, etc., without navigating away from the main content. Main content is the content of a resource, such as a web page, document, image, etc., fetched by the browser. Main content is associated with a location (e.g., a URL) and a content provider. Main content may be content that is visible within the browser's viewport. In some examples, main content may also include content of a resource that is not visible within the viewport. A contextual search tool provides contextual, suggested, or summarized content within a browser's search area, allowing a user to explore information about the main content without navigating away from the main content. The search area remains under the control of the browser and does not have access to the main content provider. Embodiments may extract at least some content from the main content. Embodiments may use the extracted content to provide contextual search suggestions. Embodiments may use the extracted content to provide generated content related to the main content. Embodiments may use the extracted content to identify entities to provide additional contextual information. The extracted content may be referred to as core content.

[0003] According to one aspect, the techniques described herein relate to a computer-implemented method, the method including receiving a contextual search request related to main content, the main content being displayed within a display area of ​​a browser, the method further including extracting at least some content from the main content, receiving contextual suggestions associated with the at least some content, and displaying the contextual suggestions in a contextual search area within the browser in response to the contextual search request.

[0004] According to one aspect, the techniques described herein relate to a computer-implemented method that includes receiving a contextual search request for main content, where the main content is displayed within a display area of ​​a browser, the method further including receiving a prompt input, extracting content from the main content, receiving generated content based on the prompt input and the extracted content, and displaying the generated content within the contextual search area of ​​the browser.

[0005] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will become apparent from the description and drawings. [Brief explanation of the drawings]

[0006] [Figure 1A.B] FIG. 1 illustrates initiation of a contextual search tool within a contextual search area for content displayed within a browser's display area, according to one embodiment. [Figure 1C] An example of main content corresponding to FIG. 1A is shown. [Figure 1D] 1C illustrates an example of a contextual search area with example contextual search suggestions corresponding to FIG. 1B, according to an embodiment. [Figure 1E] 1D shows an example of a context search area that is displayed in response to selection of SQ1 in FIG. 1D. [Figure 1F]1D shows an example of a browser interface displayed in response to selection of the area search control of FIG. [Figure 1G] Figure 1F shows an example of a browser interface generated in response to an image search submission using the selected region. [Figure 1H] 1 illustrates an example of a browser interface with a condensed context search tool interface, according to an embodiment. [Figure 1I] 1H shows an example of a contextual search area with example contextual content and contextual search suggestions generated from the prompts provided in FIG. 1H, according to one embodiment. [Figure 2A.B] 10 illustrates another variation of initiating a contextual search tool within a contextual search area for content displayed within a browser's display area, according to an embodiment. [Figure 2C] An example of main content corresponding to FIG. 2A is shown. [Figure 2D] 2C illustrates an example of a contextual search area with example contextual search suggestions corresponding to FIG. 2B, according to an embodiment. [Figure 2E.F] Figure 2E illustrates a contextual search area with results of the contextual search query selected in Figure 2D, and Figure 2F illustrates an updated display area and updated contextual search area generated in response to selecting a search result in the contextual search area of ​​Figure 2E. [Figure 3A.B] 10 illustrates another variation of initiating a contextual search tool within a contextual search area for content displayed within a browser's display area, according to an embodiment. [Figure 3C.D] Figure 3C shows an example of main content corresponding to Figure 3A. Figure 3D shows an example of a contextual search area with generated content, local search queries, and example contextual search suggestions corresponding to Figure 3B, according to an embodiment. [Figure 3E]FIG. 3D illustrates a context search area 130 with results of the context search query selected in FIG. 3D. [Figure 4A] 1 shows a contextual search tool in a contextual search area on a device with a limited display area. [Figure 4B] 1 shows a contextual search tool in a contextual search area on a device with a limited display area. [Figure 4C] 4B illustrates an example of a context search area corresponding to FIG. 4A, according to an embodiment. [Figure 4D] 4C illustrates an example of a context search area corresponding to FIG. 4B, according to an embodiment. [Figure 5] FIG. 1 illustrates a computing system and search server for implementing the concepts described herein. [Figure 6] 1 is a flowchart illustrating an exemplary process for providing a contextual search tool, according to an embodiment. [Figure 7] 1 is a flowchart illustrating an exemplary process for providing a contextual search tool, according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] The present disclosure relates to a browser tool that provides an interface for examining related search suggestions, contextually related content, and / or content associated with a resource (e.g., a web page) displayed in a browser's display area. The contextual search tool can be displayed within a browser's contextual search area, such that the content of the web page is maintained (e.g., persisted) within the browser. The contextual search tool can be interactive, allowing a user to access context-related information about the web page. At least one of the search suggestions within the contextual search area can be selected, and search results for the selected search suggestion can be displayed within the contextual search area while the content within the display area is maintained. Thus, the contextual search tool allows a user to perform context-related search tasks (e.g., checking product prices or reviews, checking restaurant, hotel availability, or points of interest, browsing a movie cast list while reading reviews, etc.) without leaving (navigating away from) the main content. A browser, when associated with an operating system (OS) window, can be referred to as a browser window.

[0008] At least one technical problem with browsing the main content of a resource is that a user may have a question about the content but may not want to leave the content, either by opening a new tab or navigating away from the resource, to find the answer. At least another technical problem is that when a user leaves the content (e.g., by opening a new tab or navigating away from the content), there is no context for the browser to assist the user in identifying additional information that answers the question. At least one technical solution offered by contextual search tools is that the tool helps users discover relevant information by making it easier for users to explore deeper and find answers through intelligent page content understanding and intelligent content generation. In other words, the tool combines multiple functions in one place and uses intelligent understanding of text and / or images in the main content to help users answer questions, understand the content, and discover new information. The content of the resource is maintained (e.g., persisted) in the browser in a separate display area while the contextual search tool, including suggested searches, search results, and contextual information, is displayed. At least one technical effect of the tool is a reduction in the number of times a user must interact with their browser to navigate to different resources and submit different queries, and at least another technical result is the presentation of generative content related to the main content to aid in the understanding of the resource.

[0009] In some embodiments, the contextual search area can be integrated as part of the browser so that it cannot be spoofed (e.g., imitated) by a third party or the owner of the web page content. This can be implemented as a security feature, allowing a user to distinguish between legitimate contextual search content provided by the contextual search tool and search results that may be inserted by a third party or by the provider of the web page or another resource. Thus, the technical problem of content spoofing can be obviated by the technical solution of integrating a contextual search area within the browser. The technical effect of integrating a contextual search area within the browser is that spoofing can be prevented / more easily recognized by the user.

[0010] The implementations described herein enable improved guided human-machine interaction for displaying web pages and other resources, on the one hand, and context-related information, on the other hand. In some implementations, the context-related information changes as a user scrolls through the main content within the display area. The display of a web page and context information that helps a user understand and / or explore the content of the web page occurs simultaneously using only one user interface. This reduces user interaction with the user interface because a user can easily search, find, and view related information about the content while browsing the content.

[0011] Some implementations include a core content extractor that scrapes the main content by, for example, examining the main content's Document Object Model (DOM) tree and / or the main content's accessibility tree to identify core content. Thus, core content may represent at least a portion of the content extracted from the main content, or, in other words, the extracted content. Core content includes text. Core content may include text and / or images represented in the main content. In some implementations, the core content extractor may be a machine-learned extraction model. The core content extractor may be configured to exclude certain types of information from the core content. For example, the excluded content may include user information, confidential information, third-party information (e.g., content provided by entities other than the content provider, such as advertisements), etc. For example, an extraction model may be trained to recognize and exclude user information, confidential information, third-party information, etc.

[0012] Embodiments use the extracted core content to provide contextual suggestions. The contextual suggestions may include contextual search suggestions. The contextual search suggestions may include image searches based on images and / or portions of images identified in the main content. The contextual search suggestions may include searches for products or product categories identified using the core content. The contextual search suggestions may include searches for additional information about an entity identified in the main content. The contextual search suggestions may include the name of the entity identified in the core content. The additional information may include user-generated content (e.g., reviews, ratings). The additional information may include news about the entity. The contextual search suggestions may include other searches that may be related, such as queries related to the provider of the main content (e.g., queries based on a URL or portion of a URL), queries related to content included in the displayed resource but not currently in the viewport, queries to which the main content may be responsive, and / or any other queries generated by the contextual query generator.

[0013] Contextual search suggestions may be actionable. In other words, they may be associated with a selectable control, such as a link or button, that, when selected, is configured to submit (send) a query to a search engine. Thus, when a contextual search suggestion is selected, some embodiments initiate a search in a search engine and provide search results in the contextual search area. Thus, the contextual search tool allows a user to browse information sources related to the main content without navigating away from the main content. In some embodiments, the contextual search tool may be iterative. For example, selecting a search result presented in the contextual search area may change the main content. For example, the browser may navigate to a resource identified in the search results and display the content of the new resource in the content display area. This content then becomes the main content, which may be provided to a content extractor, which identifies core content and displays the contextual search suggestion in the browser's search area.

[0014] Some embodiments may use the extracted core content to provide generated context related to the main content as context suggestions. For example, the extracted core content may be provided to a generative (large-scale) language model. The generative language model may provide one or more summaries for one or more entities identified in the main content. The generative language model may provide summaries of large portions of the text in the main content. The generative language model may provide creative content related to the main content (e.g., math problems generated for a web page explaining math problems, phrases generated for writing explanations in iambic pentameter, suggested dinner menus for an invitation website, etc.). The generated content may be any content generated by the generative language model based on the core content. The generated content may be any content generated by the generative language model based on the core content and a prompt. The generated content may include a degree of creativity, or in other words, content that is not factual content. In some embodiments, the context search tool may include a dialog input area. The dialog input area may allow a user to ask the generative (large-scale) language model questions about the main content or about any topic in the form of a prompt input. The generative language model may use core content, i.e., content extracted from the main content, as part of the dialog context. This may enable the generative model to increase the relevance of generated responses. In some implementations, preference information may be provided as part of the dialog context. In such implementations, the generative language model may use preference information, core content, and / or prompt inputs to provide generated content.

[0015] Embodiments may use the extracted core content to identify entities to provide additional contextual information as part of content suggestions. For example, embodiments may use a model to predict entities identified in the core content and provide information about the entities, such as from an entity repository. Embodiments may use the core content to provide information about entities related to the core content. For example, for main content about a tourist attraction (such as the Eiffel Tower), embodiments may provide generated content describing nearby hotels and / or restaurants. As another example, for main content about a product, embodiments may provide generated content describing related products or suggesting where to purchase the product. In some embodiments, categories of related entities may be provided as suggested searches. For example, a suggested search for "restaurants near me" may be provided for main content related to the Eiffel Tower, or a suggested search for "related products" may be provided for main content related to a product.

[0016] In some embodiments, the contextual search tool combines multiple search tools. For example, the contextual search tool may combine a multimodal search box and an image domain search in a single user interface. Some embodiments may also include one or more of the contextual suggestions in the single user interface. The contextual suggestions may include contextual search suggestions (queries), which are queries related to the core content. Some contextual suggestions may include generative content related to the core content. The contextual suggestions may include page summaries. The page summaries can be generated by a language model based on the core content. Some embodiments may also include a dialog input area in the single user interface.

[0017] The contextual search tool includes a novel user interface and new browser capabilities. The novel user interface helps users organize, understand, and act on internet-based documents. Specifically, the contextual search tool includes intelligence (i.e., one or more machine-learned models) that extracts core content from resources (e.g., websites) and uses that core content in combination with search services to provide contextual answers without navigating away from the original resource. This tool enables users to discover information with fewer resources (fewer provided inputs and less navigation). The contextual search tool can be surfaced in multiple ways and presented at various levels of detail.

[0018] The browsers described herein can be executed within a computing device. For example, the browser can be executed within a laptop device. In some implementations, the browser can be executed within a mobile device or on any other device with limited available screen space. While many of the implementations shown and described herein are shown in portrait mode, any of the implementations described herein can be rendered in landscape mode. Similarly, implementations described herein in portrait mode can be rendered in landscape mode.

[0019] 1A and 1B illustrate initiation of a contextual search tool within a contextual search area for content displayed within a browser's display area, according to one embodiment. FIG. 1A illustrates a web page 110 displaying a resource W1 within a display area 120 of the browser 110. In some embodiments, the display area 120 may be within a tab 112 of the browser 110. The browser 110 includes an address bar area 114. The address of the web page W1 may be displayed in the address bar area 114 (e.g., in the input address area 113). The address bar area may include a user icon 111 representing a profile of a user associated with the browser window. In some embodiments, the browser 110 may include a tool icon 125. The tool icon 125 may be a selectable control configured to open and display a contextual search tool. A request to open and display a contextual search tool may be considered a contextual search request. Although shown within the address bar area 114, the tool icon 125 may be located anywhere within the address bar area 114, including within the input address area 113. Other controls, icons, and / or the like may be included in the address bar area 114. The address bar area 114 may be controlled by and / or associated with the browser 110 (e.g., a browser application). Because the address bar area 114 is controlled by the browser 110, the web page W1 and / or the provider of the web page W1 do not have access to the content displayed in the address bar area 114 or the trigger actions provided by actionable elements in the address bar area 114.

[0020] As shown in FIG. 1B , in response to selecting the tool icon 125, the browser 110 may be configured to display a contextual search area 130 within the browser 110. In the example of FIG. 1B , the address bar area 114 and the contextual search area 130 of the browser 110 may be part of a continuous area. In other words, the browser may render the contextual search area 130 as part of the browser-controlled address bar area 114. This continuous area (e.g., the address bar area 114 and the contextual search area 130 combined) may be referred to as the browser search area. The browser search area is shown in FIG. 1B as a continuous gray color between the address bar area 114 and the contextual search area 130.

[0021] The context search area 130 may be integrated into the browser search area as part of the browser 110 (e.g., the address bar area 114), thereby preventing the context search area 130 from being spoofed (e.g., imitated), for example, by a third party or the owner of the content of the web page W1. Because the context search area is under the control of the browser, the context search area is distinct from the display area. Because the context search area 130 is part of the browser 110 application, the integration of the context search area 130 may be difficult for providers of content shown in the display area or other parties (third parties) unrelated to the browser 110 to imitate. In some implementations, there is no clear separation (e.g., no dividing line) between the context search area 130 and the address bar area 114. Because the context search area 130 and the address bar area 114 are contiguous, any background or theme applied to the address bar area 114 will intersect with (and be contiguous with) the context search area 130 (as shown by the gray area). The continuous background would be difficult for a third party (e.g., the provider of web page W1) to mimic within the application of browser 110. Additionally, the browser search area can be implemented as a security feature, allowing a user to distinguish between legitimate search results provided by a search engine in response to a query and search results that may be inserted or manipulated by a third party or the provider of the web page.

[0022] Although not shown in FIG. 1B , in some implementations, the context search area 130 is separated from the address bar area 114 by a visible separator line between the address bar area and the context search area 130. For security purposes, the line can be removed (e.g., omitted). If provided, this is a line that cannot be removed (e.g., painted over, erased) by a third party (e.g., the provider of the web page W1). In other words, if provided, this is a line that, like the address bar area 114, is controlled (e.g., provided, removed) by the browser 110 (e.g., a browser application).

[0023] Combining the contextual search area 130 and the address bar area 114 can be an indicator of the trustworthiness of the content of the contextual search area 130. When the contextual search area 130 and the address bar area 114 are combined, it means that the browser 110 (or a provider of the browser 110) is providing the contextual search suggestions 138 in the contextual search area 130. In some implementations, a dividing line between the contextual search area 130 and the address bar area 114 indicates that the results in the contextual search area 130 may be provided by an untrusted provider (e.g., a third party).

[0024] In some implementations, the contextual search area 130 can be triggered in response to the selection of a menu option, for example, from a menu displayed in response to the selection of the more options icon 108. The trigger for the contextual search area 130 is a contextual search request. In some implementations, the contextual search area 130 can be triggered in response to the selection of a menu option for a content item, such as the menu options displayed in response to right-clicking or long-pressing a word in the text T1 or image I1. In one example, a user may select the tool icon 125 in FIG. 1A. Thus, in response to interacting with the tool icon 125 in FIG. 1A (i.e., in response to a contextual search request), the web page W1 is displayed within the display area 120′ in FIG. 1B. Also, in response to the selection of the tool icon 125 in FIG. 1A (i.e., in response to a contextual search request), the contextual search area 130 is also rendered within the browser 110 in FIG. 1B. 1A , web page W1 is displayed in display area 120′, and a contextual search tool interface is rendered in browser 110 in contextual search area 130, as shown in FIG. 1B . Display area 120′ differs in size from display area 120 of FIG. 1A . In other words, display area 120′ occupies a smaller area on the display than display area 120 to make room for contextual search area 130, but is otherwise the same as display area 120. Consistent with the above discussion, the rendering of contextual search area 130 of FIG. 1B can also be triggered in other ways, for example, using a menu option.

[0025] 1B , in some implementations, the contextual search tool interface may include a pop-out control 136. The pop-out control 136 may be a selectable control that, in response to being selected, causes the content currently displayed in the contextual search area 130 to replace the main content of W1 in the display area 120 and close the contextual search area 130. In some implementations, the contextual search tool interface may include a close control 135. The close control 135 may be a selectable control that, in response to being selected, causes the browser 110 to remove (e.g., clear) the contextual search area 130. In some implementations, removing (e.g., clearing) the contextual search area 130 automatically returns the display area 120′ to the display area 120. The UI design includes visual means to clearly separate the contextual search area 130 from the display area 120, whereby the contextual search area 130 is provided as part of the browser's inherent functionality and user interface, i.e., by the browser implementation. On the other hand, the content and user interface shown in the content area 120 is obtained and rendered from external content loaded, for example in the form of web pages, from an external web server.

[0026] The contextual search tool interface may also include a search box 132. A user may initiate a query by typing the query into the search box 132. In some implementations, the search box 132 may function as a conversational input box. For example, some implementations may provide input from the search box 132 to a generative language model, which provides a generated response to the input. In some implementations, the contextual search tool interface may include a voice input control 133. The voice input control 133 may be configured, in response to being selected, to cause the browser to receive a voice recording, convert the voice recording to text, and initiate a search using the text as a search query.

[0027] In some implementations, the contextual search tool may include an area search control 134. The area search control 134 may be configured to, in response to being selected, allow a user to select a portion of content within the display area 120′ and initiate a query using the selected portion. In other words, the area search control 134 may allow a user to select an image, a portion of an image, or a frame of video from the web page W1 and initiate a search using that image. In some implementations, the area search control 134 may allow a user to define an initial image search, and the search box 132 may be configured to allow the user to further define the image search. For example, if an area of ​​content in W1 is selected, the search box 132 may show an image of the selected content. In this example, the search box 132 may include refinement controls that allow a user to crop the initial image (e.g., the initially selected area) and submit a subsequent query. Search results from a query submitted (sent to a search engine) in the search box 132 are displayed in the contextual search area 130 and do not replace the main content (the content of W1) in the display area 120′.

[0028] As is well known, a search result page is generated by a search engine in response to a query. The search result page includes search results and may include other content, such as advertisements, knowledge panels, short answers, other types of rich results including generated content, and links that limit the search to specific resource types (e.g., images, travel, shopping, news, videos, etc.). Each search result corresponds to a resource available over a network, for example, via a URL / URI. The resource is determined by the search engine to be responsive to the query. The search result includes a link to the corresponding resource. The search result may include additional information, such as a title, a snippet (text taken from the content of the resource), an image associated with the resource, or other information related to the resource and / or the query, as determined by the search engine. Traditionally, a browser displays a search result page in a display area. When a query is issued from the contextual search area 130, some embodiments may display the search results in the contextual search area 130 instead of the display area 120'. In some embodiments, the content (type, length, etc.) of search results may differ based on the query submitted via the contextual search area 130, rather than from the search engine homepage or input address area 113. In some embodiments, the ranking of search results within the search results page may differ from the core content and / or based on the query issued from the contextual search area 130. In other words, in some embodiments, the search engine may use the core content in ranking results.

[0029] In the example of FIG. 1B , the contextual search area 130 includes contextual search suggestions 138. The contextual search suggestions 138 represent queries related to the content of web page W1. In other words, the contextual search suggestions 138 represent queries selected by the contextual search tool as relevant and / or helpful for understanding or discovering additional information about the main content displayed in the display area 120′. The contextual search suggestions 138 are determined using core content extracted from the main content, i.e., the content of web page W1 of FIG. 1A. In some implementations, the contextual search suggestions 138 are determined before receiving a user-provided query. In other words, in some implementations, the initial contextual search area 130 may include suggestions for a user's search determined based on the core content, without any user-provided input. In some implementations, although not shown in FIG. 1B , additional contextual search suggestions 138 may be displayed by scrolling the contextual search area 130.

[0030] In some implementations, core content is identified by analyzing the Document Object Model (DOM) of the main content. In some implementations, core content is identified by analyzing the accessibility tree of the main content. In some implementations, core content is identified by analyzing the DOM and accessibility tree of the main content. In some implementations, core content is non-third-party content. For example, advertising content may be excluded from the core content. In some implementations, user input may be excluded from the core content. For example, if the main content includes any input controls (e.g., text boxes, drop-down boxes, etc.), content associated with the input controls may be excluded from the main content. In some implementations, sensitive content may be excluded from the core content. For example, content that is adult content or content related to financial information (e.g., websites listing bank account information) may be excluded from the core content. In some implementations, user content may be excluded from the core content. For example, a user's date of birth, name, identifiers, etc. may be excluded from the core content. In some implementations, machine learning models may be used to identify core content. For example, a DOM and / or accessibility tree may be provided to a model, and the model may determine the core content. The model is a model that runs on a client device. Thus, the core content is determined on the client device.

[0031] FIG. 1C illustrates an example of main content corresponding to FIG. 1A. FIG. 1D illustrates an example of a contextual search area 130 with example contextual search suggestions 138 for the example main content of FIG. 1C. Thus, FIG. 1D may correspond to FIG. 1B. In some implementations, the contextual search area 130 of FIG. 1D may be generated by the browser 110 in response to selection of the tool icon 125 of FIG. 1C. In the example of FIG. 1D, various entities have been identified (in the text and images of web page W1), and the contextual search suggestions 138 relate to the entities. Additionally, several searches are suggested based on the images identified in the content of W1. Because the contextual search suggestions 138 and entities are determined without user input, one or more models may be used to determine which images and / or portions of images a user is most likely to use in a search, given the page context (e.g., core content). One or more models may be used to determine which entities a user is most likely to search for, given the page context (e.g., core content). In some implementations, the model(s) may consider one or more of the following factors: These factors include: image size, image content, core content category (e.g. product, hotel, restaurant, company homepage, blog, etc.), resource URL, file type, interchangeable image file format of the image, text associated with the image (e.g. caption and / or accessibility text added by the accessibility tree), file path of the image, relative position of the image in the viewport, relative position of the image in the resource, relative size of the image compared to other images in the page, relative size of the image compared to other images in the page, whether the image is a hotlinked image (hosted on a site other than the main content site), etc.

[0032] In the example of FIG. 1B , the suggested search SQ1 is a suggestion to find a similar sofa. The sofa is identified in image I1 of web page W1. Another suggested search includes a search for user-generated content (e.g., reviews) about a particular product mentioned in the content of web page W1. As another example, the suggested search includes a search for a category of sofas identified in the content of web page W1. As another example, the suggested search includes a search for images similar to I1. As another example, a lamp is identified in image I1, and the suggested search includes a search for lamps similar to the identified lamp. Further examples of suggested searches are shown in FIG. 1D , all related to the main content in display area 120′. As described in more detail with respect to FIG. 5 , queries can be suggested by a query suggestion service in response to receiving core content. In some implementations, the suggested searches can be ranked (ordered) based on characteristics of the core content. For example, the suggested searches can be ranked based on categories determined for the core content. As another example, suggested searches may be ranked based on the expected user action(s) on the core content. As another example, suggested searches may be ranked based on the category of entities identified in the core content. As another example, suggested searches may be ranked based on the placement or prominence of entities in the core content.

[0033] FIG. 1E shows an example of a contextual search area displayed in response to selection of SQ1 in FIG. 1D. Accordingly, FIG. 1E illustrates that a search initiated in the contextual search area 130 does not replace the content of the display area 120′. Instead, the contextual search suggestions 138 are replaced with search results 158. A back control 156 may cause the contextual search suggestions 138 to be rendered. Selection of one of the search results 158 may cause the web page W1 to be replaced with content represented by the resource associated with the selected search result. The content of the search results in the search results 158 may be based on the category of the resource.

[0034] FIG. 1F shows an example of a browser interface displayed in response to selection of region search control 134 in FIG. 1D. In response to selection of region search control 134, the browser may allow the user to select a region of the main content (of web page W1) via region selector 160. Region selector 160 may be any method of identifying a region of the main content. While shown as a rectangle, region selector 160 may be any shape, e.g., circular, oval, triangular, or free-form. In some implementations, the user may use predetermined gestures to indicate which portion of the image to search. In one example, the user may place an input device (e.g., a cursor, finger, stylus, etc.) within the main content and drag diagonally or draw a circle, oval, etc. to identify the selected region. In one example, the browser may place a default region selector 160 over a portion of the main content, and region selector 160 may have a size control 162 to change the size of region selector 160. Region selector 160 may be moved from one position to another. Any portion of the main content within the boundaries of the region selector 160 can be used as an image search.

[0035] In some implementations, the browser 110 may perform object segmentation on an image in the main content. In some implementations, the object segmentation may be performed in response to selection of the region search control 134. In some implementations, the object segmentation may be performed in response to presentation of the context search area 130, for example, in response to selection of the tool icon 125. In object segmentation, pixels in an image are assigned to a class. Each image may have multiple classes. A class may correspond to a particular entity. Although a class may not correspond to a particular entity, pixels may be understood to belong to the same type (class). In such implementations, the browser may enable selection of segments (different classes), e.g., segments may function as region selector 160. For example, the object segmentation model may be used to analyze image I1 of FIG. 1F. As a result, several object segments may be recognized, such as a lamp, a sofa, a floor, a curtain, and a table. Rather than needing to click and drag region selector 160 to select the table shown in FIG. 1F, a user may hover over the table, and browser 110 may change the appearance of the region of image I1 corresponding to the table in response to detecting the hover input. For example, browser 110 may change the appearance of the region so that a dashed line outlines the table. This outline may indicate that the portion of image I1 represented by the table is selectable, for example, as region selector 160. In such an implementation, if a user selects (e.g., clicks / tap) the portion of the image outlined by the dashed line, that portion of the image may be submitted as a query. FIG. 1G shows an example browser interface generated in response to submitting an image search using the region selected in FIG. 1F. In some implementations (not shown), query suggestions related to region selector 160 may be provided in contextual search suggestions 138.

[0036] FIG. 1H illustrates an example browser interface with a condensed context search tool interface (i.e., a condensed context search area) generated in response to initiation of a context search tool, as disclosed herein. For example, the condensed context search tool interface may be generated in response to selection of the tool icon 125 of FIG. 1C , in response to a menu option, from a menu selection generated, for example, in response to selection of the more options icon 108, etc. Accordingly, under any condition in which the context search tool is initiated, a dialog input area 170 may be generated. The condensed context search area may include the dialog input area 170. The dialog input area 170 provides the user with an opportunity to provide a question regarding the main content of the web page W1 before the context search area 130 is displayed. Thus, the dialog input area 170 is similar to the search box 132 described above in that the user can type a query into the dialog input area 170. In some implementations, the query may provide a prompt in the dialog input area 170. In some implementations, input into the dialog input area 170 may be considered a query and a prompt. In such embodiments, search results may be determined for the input as a query, and generated content may be returned for the input as a prompt. Dialog input area 170′ shows an example of a prompt provided by a user as input. In some embodiments, dialog input area 170 may include speech input control 133, as described above, to provide the prompt. In some embodiments (not shown), dialog input area 170 may include area search control 134, as described above. In some embodiments, the condensed context search area may include area search instructions 172. Area search instructions 172 may provide an indication to the user that the user can select an area of ​​the displayed main content of web page W1 for search. Area selection may be performed in any of the manners described with respect to FIG. 1F.This selected content may be provided to the contextual content engine. In some embodiments, the selected content may be provided as prompt input. In some embodiments, this selected content may be provided in addition to the prompt input provided in dialog input area 170. In some embodiments, there are no region search instructions 172, and it is implied that the user can select a region of the displayed main content of web page W1. In some embodiments, the contextual search tool may cause the browser to display a transparent overlay over the main content of web page W1 as part of the condensed contextual search area. The transparent overlay may be removed when a region is selected to indicate which portion of the main content has been selected for search.

[0037] FIG. 1I illustrates an example of a contextual search area with example contextual content and contextual search suggestions generated from the prompts provided in FIG. 1H, e.g., as shown in dialog input area 170′, according to one embodiment. In the example of FIG. 1I, the contextual search area 130 may be an overlay interface. In an overlay interface, the contextual search area 130 may even be integrated as part of the browser 110, thereby preventing the contextual search area 130 from being spoofed (e.g., imitated) by, for example, a third party or the owner of the content of web page W1. However, with an overlay interface, some of the content of web page W1 is obscured by the contextual search area 130. It is clear that the contextual search area 130 and address bar area 114 in FIG. 1I are not contiguous, and any background or theme applied to the address bar area 114 also interrupts the contextual search area 130, thereby associating the contextual search area 130 with the contextual search tools provided by the browser, not the content provider.

[0038] In the example of FIG. 1I, the contextual search tool provides two contextual search suggestions 138. Additional contextual search suggestions 138 may be displayed, for example, by scrolling through the content in the contextual search area 130. In the example of FIG. 1I, the contextual search tool provides search suggestions 175. The search results 175 may include generated content. The generated content may be considered a contextual suggestion. For example, the prompt in the dialog input area 170′ may be provided to a generative language model with core content extracted from web page W1. The prompt in the dialog input area 170′ may be denormalized before being processed by the generative language model or as part of being processed by the generative language model. Thus, for example, “this couch” in the prompt (or query) may be denormalized to refer to a relaxer two-piece couch offered for sale by the content provider of web page W1. In some implementations, the generative language model may return generated content as a contextual suggestion that reflects (summarizes) reviews for the relaxer two-piece couch offered by the content provider of web page W1. In some implementations, the content of the actual reviews (eg, top, most useful, most recent reviews) may be provided as search results 175 .

[0039] Although described in the context of web page W1, in some implementations, the content rendered in display area 120′ may not be a web page. As described herein, the content may be associated with any resource accessible over a network or stored on the user's device. Thus, in some implementations, the content displayed in display area 120′ may be an image, a link, a video, text, a PDF file, and / or the like.

[0040] 2A and 2B illustrate another variation of initiating a contextual search tool within the contextual search area 130 for content displayed within the display area 120 of the browser 110, according to one embodiment. In the example of FIG. 2A, website W2 is the source of the main content displayed within the display area 120 of the browser 110. Website W2 is a news source, e.g., a social media site, a news aggregation site, etc. Website W2 includes post P1 and post P2 within the visible area (e.g., the display area 120), although only a portion of post P2 is visible. Website W2 may have many additional posts that are not included in the display area 120. In some examples, website W2 may be an infinite scrolling site. In other words, as a user scrolls down website W2, additional posts may be fetched, preventing the user from reaching the end of the page. Website W2 may also include non-core content C1 and non-core content C2. Non-core content C1 and C2 may represent advertisements, offers, or other content that may or may not be related to post P1 or post P2. Non-core content C1 and C2 may be examples of content excluded from the core content extracted for website W2.

[0041] In response to a user selecting tool icon 125, the browser may render the main content in display area 120′ and the contextual search tool in contextual search area 130, as described above. The controls for the contextual search tool are similar to those described with respect to FIG. 1B. Contextual search suggestions 238 may be similar to contextual search suggestions 138. However, because the context of the content has changed, contextual search suggestions 238 may differ in nature from contextual search suggestions 138.

[0042] In some implementations, although not shown in FIG. 2B , the additional contextual search suggestions 238 can be triggered by a scroll input. For example, by scrolling or otherwise navigating the main content within the display area 120′ of FIG. 2B . In other words, in response to a user action (scroll input), the browser 110 can be configured to determine when the main content rendered within the display area 120′ (viewport) has changed (e.g., when it's 60% new, 75% new, when a new post is fully rendered on display, etc.) and trigger a contextual search tool, which can re-extract the core content and generate a new set of contextual search suggestions 238. In other words, when a user scrolls through the posts of web page W2, the browser 110 can detect the scroll input and modify the contextual search suggestions 238 in the contextual search area 130 to match the currently displayed content. In some embodiments, this may include re-ranking and / or re-ordering the contextual search suggestions so that contextual search suggestions not previously provided (or not previously provided within the viewport of the contextual search area 130) are visible. In some embodiments, this may include redetermining (re-extracting) the core content and submitting a new request to the contextual content engine to receive new contextual search suggestions 238. In some embodiments, this may include re-submitting a request to the contextual content engine with an indication of the core content that corresponds to the current main content within the viewport to receive new contextual search suggestions 238.

[0043] Figure 2C shows an example of main content for web page W2 corresponding to Figure 2A. In the example of Figure 2C, content C1 is recommended content generated by the provider of website W2, and content C2 may be advertisements not generated by the provider of website W2. Both C1 and C2 may be excluded from the core content used to generate contextual search suggestions 238. Figure 2D shows an example of contextual search area 130 with example contextual search suggestions 238 corresponding to Figure 2B, according to one embodiment. Contextual search suggestions 238 differ from the type of contextual search suggestions 138 because the underlying core content is different.

[0044] FIG. 2E illustrates a contextual search area with results of a contextual search query selected in FIG. 2D. For example, a user may select suggested search SQ2 in FIG. 2D, and the browser may submit a query to a search engine for "news about protecting the Great Barrier Reef." In response to receiving search results for the query, the browser may display a search results page 236 in the contextual search area 130, as shown in FIG. 2E. Thus, the contextual search suggestion page 236 replaces the contextual search suggestions 238 with the search results page 236. In some implementations, because the contextual search area 130 has less display space than the display area 120, the items in the search results page 236 may differ from those returned for the same query when the results are rendered in the display area 120 instead of the contextual search area 130.

[0045] FIG. 2F illustrates an updated display area and an updated contextual search area generated in response to selection of a search result in the contextual search area of ​​FIG. 2E. In the example of FIG. 2E, web page W3 replaces web page W2 in display area 120 because web page W3 is the resource associated with the selected search result. In the example of FIG. 2F, contextual search area 130 is not closed but is updated with the content of web page W3. In other words, in some embodiments, browser 110 may retain contextual search area 130, extract the core content of web page W3, and provide that core content to a search service to obtain additional contextual suggestions related to the core content of web page W3. However, in some embodiments, the browser may close contextual search area 130 and restore display area 120′ to display area 120 in response to selection of a search result. In some embodiments, browser 110 may maintain search result page 236 in contextual search area 130.

[0046] In the example of FIG. 2F, the contextual search area 130 includes a page description 242. The page description 242 may follow any contextual search suggestions 238 for web page W3. The page description 242 may be triggered by a control (not shown). In some implementations, the page description may be some or all of the information about the web page provider obtained from a trusted source, such as a WIKIPEDIA page, an "about" section from a directory, etc. In some implementations, the content may be selected using a model. In some implementations, a generative model may be used to generate the page description 242. The page description 242 represents details about the source of web page W2.

[0047] 3A and 3B illustrate another variation of a contextual search tool within a contextual search area relative to content displayed within a browser's display area, according to some implementations. In the examples of FIGS. 3A and 3B, the contextual search tool includes generative content that is generated using the context of the main content (e.g., core content). In FIG. 3A, website W4 is the source of the main content displayed within display area 120 of browser 110. Website W4 may include content that is not currently displayed within display area 120. In some examples, website W4 may be an infinite scroll site that always has additional content that is not currently displayed within display area 120.

[0048] In response to a user selecting tool icon 125, the browser may render the main content in display area 120′ and the contextual search tool in contextual search area 130, as described above. Controls for the contextual search tool are similar to those described with respect to FIGS. 1B and 3B. In the embodiment of FIG. 3B, the contextual search tool includes several examples of contextual suggestions, such as generated content 360, local suggestions 348 (LQ1 and LQ2), and contextual search suggestions 338 (SQ1 and SQ2). Generated content 360 may be a summary of the main content generated by a generative language model. For example, core content may be provided as input to the generative language model, which generates a summary and provides it as generated content 360. In some embodiments, the suggested search may be generated based on other content on web page W4 or based on the source of the generative language model used to generate the response. These are shown as LQ1 and LQ2 in FIG. 3B. As described herein, other contextual search suggestions SQ1 and SQ2 may also be provided within the contextual search area 130. In some implementations, one or more of the contextual search suggestions SQ1 or SQ2 and / or LQ1 or LQ2 may be queries related to facts presented in the generated content 360.

[0049] Figure 3C shows an example of main content corresponding to Figure 3A. Figure 3D shows an example of a contextual search area with generated content 360, example local search queries LQ1 and LQ2, and contextual search suggestions SQ1 and SQ2 corresponding to Figure 3B, according to one implementation.

[0050] FIG. 3E illustrates the context search area 130 with the results of the context search query LQ1 selected in FIG. 3D. LQ1 is a query related to how the movie described in the main content of the display area 120 was received. Like FIG. 3D, the context search area 130 in FIG. 3E includes generated content GC2 and GC3. Additionally, the browser 110 modifies the appearance (e.g., highlights, underlines, etc.) of content from website W4 (e.g., underlined section 340a, underlined section 340b) related to the query (LQ1) and generates a summary of responses from sources available via the Internet.

[0051] In some implementations (not shown), the context search area 130 may include a dialog input area similar to the dialog input area 170 of FIG. 1H. In some implementations, the search box 132 may be a dialog input area. In some implementations, the context search area 130 may include a separate dialog input area. Using the dialog input area, a user may initiate a dialog with a generative language model, for example, by entering a prompt input. An advantage of the disclosed implementations is that the browser 110 may be configured to provide the core content of the display area 120′ to the language model in addition to the text (prompt) provided by the user via the dialog input area. Thus, the core content becomes part of the dialog context. This allows the user to ask questions related to the core content and may also help the language model provide more relevant generated responses. As with other disclosed implementations, the core content may be determined at the client device, such that certain information displayed in the display area 120′ is excluded from the core content, as described herein. The dialog context may include the prompt, the core content, and any previous prompts and generated content preceding the current prompt. This dialog context may be denormalized before being processed by the generative language model or as part of being processed by the generative language model. Thus, for example, the current prompt, "What style is the image on the left?" may be denormalized to formulate a prompt that identifies images in the display area to the left of other images and analyzes the style of those images. Similarly, the prompt, "Where can I buy this?" can be denormalized to identify the main entities mentioned in the core content and / or selection region. In some implementations, with the user's permission, attributes about the user may be included in the context. For example, the user's location may be used to generate a response to the prompt, "Where can I buy this?", to return retailers within a specified range of the user's approximate location.The response generated by the generative language model is provided as a response to the current prompt and becomes part of the dialog context, which in some implementations is limited to a predetermined number of previous prompts and responses.

[0052] FIG. 4A illustrates a contextual search tool within a contextual search area 130 on a device with a limited display area. The contextual search tool may be any of the implementations described above with respect to FIGS. 1A-1I, 2A-2F, and / or 3A-3E. However, because the screen is smaller than the screens shown in these previous figures, the contextual suggestions may include fewer suggestions, may be presented in a different manner, and / or may include fewer elements of different types. The contextual search area 130 in FIG. 4A may initially be partially hidden. In other words, the contextual search area 130 in FIG. 4A may be an overlay window (e.g., similar to the contextual search area 130 in FIG. 1I) that is initially only partially visible. FIG. 4B illustrates the contextual search area 130 with the overlay window expanded. The contextual search area 130 in FIG. 4B is still a separate area from the display area 120′, but within the browser 110. FIG. 4C shows an example of a context search area corresponding to FIG. 4A, and FIG. 4D shows an example of a context search area corresponding to FIG. 4B, according to one embodiment. While the context search area 130 shown in FIGS. 4A-4D is shown as an overlay window at the bottom of the display area 120′, several embodiments include an overlay window on either side or on top of the display area 120′. In some embodiments, the location of the overlay window may depend on the device type and / or device orientation. The context search area 130 may also be an overlay window on devices without display area limitations.

[0053] In some implementations, the context search area 130 of any of the figures described above may include one or more feedback controls. The feedback controls may be used to collect preference information, with the user's permission. The preference information may include information about user interaction with the suggestions. The feedback controls are interactive user interface elements used to collect direct feedback that can be recorded as preference information. The feedback controls may allow the user to reject the suggestions. Rejecting the suggestions may be interpreted as negative feedback on the suggestions (negative user interaction with the suggestions). The feedback controls may be interactive user interface elements that allow the user to indicate support for the suggestions. The interactions may be with any type of suggestions, including additional queries, generated content, search results, or any other content presented in the context search area 130. In some implementations, the user interaction information may include the user's selection of a suggestion, which may be interpreted as positive feedback on the suggestion (positive user interaction with the suggestion). In some implementations, the browser may provide a user interface for the user to provide preference information. Such preference information may include preferred resource providers (e.g., preferred merchants, preferred news sites, preferred social media sites, and / or preferred influencers on social media sites, etc.). Such preference information may include preferred attributes for particular entity categories and / or particular resource categories. For example, a user may provide preferred attributes of shoppable resources, such as price, review ratings, and / or dimensions. A user may provide preferred attributes of celebrities, etc. In some implementations, a user interface for providing preference information may include controls for indicating preferences for particular types of suggestions and / or for completely rejecting particular types of suggestions.

[0054] In some implementations, with user permission, preference information may be recorded, for example, in a user preference file. In some implementations, preference information may be provided with core content and used to determine contextual suggestions. In some implementations, preference information may be used to further train a model; for example, contextual suggestions may be based at least in part on a user preference file. In some implementations, with user permission, one or more models may be further trained (fine-tuned) using preference information. In some implementations, preference information may lack any information that identifies a particular user. For example, preference information may pair resources with selected and / or rejected suggestions. The system may use this user-independent preference information to fine-tune a model, such as a context engine, to favor (retain, uprank) selected contextual suggestions and oppose (filter, downrank) rejected suggestions.

[0055] FIG. 5 illustrates a system 500 including a computing system 502 and a search server 540 configured to implement the concepts and various embodiments shown and described herein. The computing system 502 may be a computing device with a limited screen size, such as a smartphone, a smartwatch, a smart device (e.g., A / R glasses or V / R glasses), or a tablet. The computing system 502 may also be a computing device with a larger screen size, such as a desktop computer, laptop, netbook, notebook, tablet, smart TV, or game console, that runs a browser. In general, the computing system 502 may represent any computing device that runs a browser. As shown in FIG. 5 , the computing system 502 is configured to communicate with a search server 540 and / or a resource provider 510 (e.g., a web server) over a network 550. The computing system 502 includes at least a browser 520 and a search companion manager 523. In some embodiments, the browser 520 is configured to manage resource content, such as web page content, provided by the resource provider 510 (e.g., a web server). In some embodiments, the browser 520 is configured to operate as one of several applications 528 executed using an operating system (O / S) 529. The browser 520 can be configured to implement a user interface, context suggestions, search results, windows, browser areas, and / or other portions as described in connection with the embodiments described herein.

[0056] 5, computing system 502 includes several hardware components, including a communications module 561, one or more cameras 562, memory 563, a central processing unit (CPU) and graphics processing unit (GPU) 564, one or more input devices 567 (e.g., a touchscreen, a mouse, a stylus, a microphone, a keyboard, etc.), and one or more output devices 568 (a screen, a speaker, a vibrator, lights, etc.). The hardware components may be used to facilitate the browser 520, the search companion manager 523, and / or other operations of computing system 502.

[0057] Browser 520 includes a user interface (UI) generator 521 configured to generate and / or manage various user interface elements of a browser, such as browser 110 shown and described herein. For example, UI generator 521 can generate UI elements including various windows of browser 110, such as display area 120, contextual search area 130, etc., shown in at least Figures 1A-1H, 2A-2F, 3A-3E, and 4A-4D.

[0058] Browser 520 includes a tab manager 522 configured to create and / or manage various tabs (e.g., tab 112) of a browser, such as browser 110. Browser 520 may be configured to provide / perform actions associated with actionable controls, such as contextual search suggestions 138, 238, 338, generated content 360, tool icons 125, and links in controls of contextual search area 130 (such as voice input control 133, search area control 134, close control 135, pop-out control 136, search box 132, back control 156, etc.), among others. Tab manager 522 may also be configured to provide a menu of actions and cause performance of an action selected from the menu of actions.

[0059] Browser 520 includes a search companion manager 523 configured to generate and / or manage the rendering of content, such as content in contextual search area 130 (as shown at least in FIGS. 1B, 1D, 1E, 1F, 1I, 2B, 2D, 2D, 2E, 3B, 3D, 3E, and 4A-4D). Search companion manager 523 can also be configured to determine when to trigger the display of contextual search area 130. In other words, search companion manager 523 can be configured to determine which event triggers the rendering of contextual search area 130 and whether a triggering event has occurred. A triggering event can include any of the events described herein, such as selecting a tool icon (tool icon 125), selecting an action from a menu of actions, receiving a voice command, etc. In some implementations, the search companion manager 523 may determine whether to display a condensed-context search tool interface or a full-context search tool interface (e.g., in the context search area 130). In some implementations, the search companion manager 523 may include an object segmenter 525. The object segmenter 525 may be configured to enable selection of different portions of the displayed main content of a resource, for example, by categorizing the portions into different classes. Thus, for example, recognized entities within the main content may be assigned to their respective classes. Portions of the main content that are part of the same object may be assigned to their respective classes even if the entities are not recognized. In such implementations, when a user hovers over an image, the search companion manager 523 may provide an indication of the portions of the image that are included in the class. In such implementations, the user may be able to easily ask questions about that portion of the main content.

[0060] In some embodiments, the search companion manager 523 may include a core content extractor 524. In some embodiments, portions of the core content extractor 524 may be part of the browser process. In some embodiments, the core content extractor 524 is not integrated into the search companion manager 523. The core content extractor 524 may be configured to identify and extract core content from the main content of a displayed resource. In other words, the core content extractor 524 may be configured to identify which content associated with a resource displayed within the browser's display area is core content. As described herein, the core content extractor 524 may receive as input a DOM tree and / or an accessibility tree generated by the browser 520 for a resource and use that input to determine the core content. An advantage of using both a DOM tree and an accessibility tree is that description nodes are added to the accessibility tree for DOM elements, such as images. A resource provider and / or the location of the resource (e.g., a URL) may be considered core content. The core content extractor 524 may be configured to ignore or exclude certain elements from the core content. These elements may include user information, or in other words, elements provided by the user (e.g., associated with an input control), elements describing the user (e.g., username, profile information, account number, etc.), etc. These elements may include sensitive information. Sensitive information may include age-restricted content (e.g., adult content, whether text or images). Sensitive information may include account information (e.g., financial institution pages). Thus, in some implementations, there may be little core content provided to the search server 540 because much of the core content is excluded by the core content extractor 524 based on the type of resource (e.g., the resource is a sensitive resource).In some implementations, once a resource is determined to be a sensitive resource, not all features of the contextual search tool may be enabled. For example, generative content may be disabled for some or all sensitive types of the sensitive resource. In some implementations, the core content extractor 524 may be a machine-learned model running on the computing system 502. The model may be trained to detect the sensitivity of a resource. The model may be trained to determine what to extract based on the sensitivity. The model may be trained to exclude (e.g., ignore) certain types of information, such as user information, sensitive information, etc.

[0061] Browser 520 can be configured to generate and / or manage the rendering of content associated with a resource (e.g., web page W1) in display area 120 (including display area 120′) as shown. Resource content can be provided to computing system 502 by resource provider 510. Browser 520 and / or search companion manager 523 can be configured to perform one or both of the processes, or portions of the processes, described in connection with FIG. 6.

[0062] 5, session data 527, which may be stored in memory 563 (not shown), may be managed by or as one of applications 528. Session data 527 may include data related to one or more browser sessions. Application information 526 may include information related to various applications running within and / or executable by O / S 529.

[0063] 5, communications module 561 can be configured to facilitate communication with resource provider 510 and / or search server 540 over network 550 via one or more communications protocols. Camera 562 can be used to capture one or more images, and memory 563 can be used to store information associated with browser 520 and / or search companion manager 523, other applications 528, O / S 529, etc. CPU / GPU 564 can be used to process information and / or images associated with browser 520 and / or search companion manager 523. Computing system 502 also includes one or more output devices 568, such as a communications port, speakers, a display, and / or others. Functionality described herein can be implemented based on one or more policies 565 and / or preferences 566 stored in memory 563.

[0064] FIG. 5 illustrates some aspects of a search server 540. For example, the search server 540 includes one or more processors 546 (i.e., processors formed within a substrate) and one or more memory devices 548. The search server 540 includes a search engine 544 configured to receive search requests (queries) from the computing system 502. The queries may include search suggestions generated for and rendered in a contextual search area. The search engine 544 may be used to provide a search result page for a given query. The queries may be a word, a phrase, multiple words, an image, a media file, etc., as is well known. The search engine 544 is configured to search one or more data repositories for resources in response to the queries. Specifically, the search engine 544 may use a web page index, an image index, a product index, an entity repository, a news index, etc. to determine the content of the search result page. The search result page includes one or more search results, each corresponding to a resource responsive to the query. Each search result includes a link to its respective resource. In some implementations, the search engine 544 can add additional elements to the search results page, such as advertisements, sponsored links, and rich elements (such as knowledge panels, short answers, mini-apps, carousels, navigation links, etc.).

[0065] The search server 540 may also include a contextual content engine 542. The contextual content engine 542 may be configured to determine contextual suggestions, generative content, etc. for a given resource context, i.e., core content, etc. In some implementations, with user permission, the contextual content engine 542 may be configured to determine contextual suggestions based on preference information stored, for example, in user preferences 566. For example, the preference information (preference data) may be from a user profile and may include user preferences that are considered when making contextual suggestions. In some implementations, the preference information (preference data) may be inferred from browsing history, with user permission. For example, the preference information may include actions taken by the user with respect to previous contextual suggestions. The actions may include selecting a previous contextual suggestion. The actions may include rejecting a previous contextual suggestion. In some implementations, the contextual content engine 542 may include (or have access to) a contextual query generator 543. The contextual query generator 543 may be configured to suggest searches (suggested queries) given inputs such as core content and / or resource locations (URLs). The contextual query generator 543 may be configured to rank suggested searches (suggested queries) differently based on requests originating from the search companion manager 523 rather than from a search engine homepage or from another application 528 executing on the computing system 502. The contextual content engine 542 may be configured to generate a page description, such as, for example, page description 242 of FIG. 2F. In some implementations, the page description may be generated by a content description generator 545.

[0066] The contextual content engine 542 may include a content description generator 545. The content description generator 545 may be or may include a generative language model, also referred to as a large-scale language model. Such a generative language model can generate natural language responses to prompts, such as prompts entered in a search box or dialog input area. In some implementations, the content description generator 545 may include or have access to several different models. The content description generator 545 may be configured to receive core content as input and provide a generated summary for the core content. The generated summary may be a single sentence or several sentences. The content description generator 545 may be configured to generate more than one summary for the core content. The content description generator 545 can be configured to generate additional summaries. The additional summaries may be about one or more entities (topics) identified in the content. The additional summaries may be about one or more entities (topics) related to the entity identified in the content. For example, rather than suggesting searches for related products, the content description generator 545 may generate summaries for related products. As another example, the content description generator 545 may generate summaries of reviews about related products. The content description generator 545 may be configured to generate answers to prompts about the core content, such as answers to prompts entered in a dialog input area. The content description generator 545 may be configured to generate creative content related to the core content. For example, math problems, poems, party menus, party ideas, etc. may be generated and provided by the content description generator 545 in response to prompts. While shown as part of the search server 540, in some implementations, one or more components of the contextual content engine 542 may be implemented in the computing system 502.

[0067] The content description generator 545 may be configured to add annotations (e.g., footnotes) to the generated summary. The annotations may be actionable. For example, selecting a footnote causes the content to scroll in a browser's viewing area or causes resources to be displayed in a contextual search area. In some implementations, the content description generator 545 may be configured to suggest searches. Thus, the content description generator 545 or the contextual query generator 543 may suggest additional searches. The content description generator 545 may be configured to receive as input a dialog context that includes core content and provide a generated response in return. The generated response may also include annotations. The generated response may include additional search suggestions.

[0068] FIG. 6 is a flowchart illustrating an example method 600 for implementing at least some of the concepts described herein in the various figures. Many elements of method 600 can be performed by at least the system illustrated in FIG. 5. Specifically, method 600 can be performed by a browser (e.g., browser 520) of computing system 502. The example method 600 is an example of triggering the display of a contextual search area (contextual search area 130) in response to a contextual search request. The contextual search request can be triggered by a user, for example, via interaction with an icon, a menu option, or the like. In step 602, the system can extract core content from the main content of a resource. The main content can be represented by a DOM tree. The main content can be represented by an accessibility tree. The main content can be represented by a DOM tree and an accessibility tree. Core content can be identified as content provided by an entity associated with the resource, or, in other words, content hosted at a domain associated with the resource, as opposed to content provided from a source not associated with the domain. In other words, the core content may exclude content associated with entities that are not the content provider of the resource (e.g., excluding advertising content). Main content may be identified as the content in the viewport, or in other words, the content that is currently visible to the user. Main content may be identified as any content in the DOM tree and / or accessibility tree, regardless of whether it is in the viewport. In some implementations, content in the viewport may be indicated as such in the core content (extracted content). This may give more weight (higher weight) to visible content when determining contextual suggestions, e.g., contextual search suggestions and / or generated content. As described herein, some content may be excluded from the core content based on the type of information (e.g., user information, confidential information, etc.).In step 604, the core content is provided to a contextual content engine, for example, at search server 540. The contextual content engine uses the core content to generate contextual suggestions.

[0069] In step 606, contextual suggestions are received from the contextual content engine. The contextual suggestions may include a generated summary. The contextual suggestions may include a suggested search. The contextual suggestions may include a page description. The type of suggestions included may be determined by the category of the resource. For example, a resource categorized as a shopping resource may be provided with different types of suggestions than a resource categorized as a news article or a celebrity resource. Similarly, the information included in the suggestions may be based on the category of the resource. The suggestions may be ordered based on the resource category. The suggestions may be ordered based on expected user actions for the resource category. In some implementations, a user may customize the type of suggestions included with respect to the resource category and / or the type of information included in the suggestions. For example, a user may provide preferences for specific merchants, specific news sites, specific social networks, specific attributes of entities (e.g., product price and dimensions, etc.). In step 608, the system may display the contextual suggestions in a contextual search area outside the display area but within the browser. Displaying the contextual search area may, for example, reduce the size of the display area to make space for the contextual search area on the display. In some implementations, the contextual search area may be an overlay window that can overlay the display area. The contextual search area is considered to be outside the display area whether it is displayed simultaneously with the display area or as an overlay window.

[0070] Once the contextual search area is displayed, the user may interact with the content of the contextual search area. For example, suggested searches displayed in the contextual search area may be actionable (selectable). At step 610, the system may receive a selection of the suggested search. At step 612, in response to the selection, the system may provide the selected suggested search to a search engine. The search engine may use the suggested search as a query and provide search results. In response to receiving the search results, the system may render a search results page within the contextual search area. Thus, searching can be accomplished without navigating away from the main content. In some implementations, selection of a search result may trigger the fetching and rendering of content for a new resource (the resource associated with the selected search result) within the display area. In some implementations, method 600 resumes at step 602 with the new resource.

[0071] As another example, the contextual search area may include a region search control. The region search control allows a user to select a portion of the displayed content (including all or part of a displayed image). In step 614, a selection of the region search control is received. In step 616, the system provides a region selector that allows the user to specify the selected portion of the displayed content. In step 618, the selected region is provided to the search engine. In response, the search engine provides a search results page. The search results page is displayed within the contextual search area. In step 612, selection of a search result may trigger fetching and rendering of content for a new resource (the resource associated with the selected search result) within the display area. In some embodiments, method 600 resumes at step 602 with the new resource. In some embodiments, the system may provide a refinement tool. The refinement tool may allow the user to further crop the selected portion and resubmit the query for the cropped portion.

[0072] Some embodiments may include a prompt input in the contextual search area. In step 620, the system may receive a prompt input from a user, for example, from a dialog input area and / or from a search box. In step 622, the system may obtain a generated response to the prompt input by providing a dialog context to the generative language model. The dialog context includes not only the prompt input but also core content. In some embodiments, the dialog context may include previous prompt input and generated responses associated with the displayed resource. The core content provides context for the generative language model in generating a response to the prompt. The system may display the response to the prompt in the contextual search area. In some embodiments, a change in the resource displayed in the display area clears the dialog and signals the start of a new dialog.

[0073] FIG. 7 is a flowchart illustrating an example method 700 for performing at least some of the concepts described herein in the various figures. Many elements of method 700 can be performed by at least the system shown in FIG. 5. Specifically, method 700 can be performed by a browser (e.g., browser 520) of computing system 502. The example method 700 is an example of triggering the display of a contextual search area (e.g., contextual search area 130) in response to a contextual search request. The contextual search request can be triggered by a user, for example, by interacting with an icon, a menu option, or the like. In step 702, the system can receive a prompt input for a contextual search. The prompt input can be any natural language text provided by a user, for example, via a dialog input area. The prompt input can be normalized, for example, to replace pronouns. In some implementations, the normalization can be based at least in part on core content. In some implementations, receiving the prompt input can be a selected portion of main content. The selected portion can be, for example, an object identified within an image of the main content by an object segmenter. In some implementations, the selected portion may be a region selected by a click and drag gesture, or a surround gesture, or some other region selector.

[0074] In step 704, in response to receiving the contextual search request, the system may extract core content from the main content of the resource. The main content may be represented by a DOM tree. The main content may be represented by an accessibility tree. The main content may be represented by a DOM tree and an accessibility tree. Core content may be identified as content provided by an entity associated with the resource, or in other words, content hosted on a domain associated with the resource, as opposed to content provided from a source not associated with the domain. In other words, core content may exclude content associated with an entity that is not the content provider of the resource (e.g., core content may represent non-advertising content). Main content may be identified as content within the viewport, or in other words, content currently visible to the user. Main content may be identified as any content within the DOM tree and / or accessibility tree, regardless of whether it is within the viewport. In some implementations, content within the viewport may be so indicated in the core content. This may give more weight to visible content when determining contextual suggestions, such as contextual search suggestions and / or generated content. As described herein, some content may be excluded from the core content based on the type of information (eg, user information, confidential information, etc.).

[0075] In step 706, the system may provide the prompt (including portions of the selected image, if any) and the core content to a generative language model. The generative language model may be part of a contextual content engine. The generative language model may use the prompt and the core content to generate a response to the prompt. The response may be content generated based on the core content. The core content may be used as dialog context for the prompt. The response may include information about the core content, such as information about entities identified in the core content. The response may include creative content. The response is based on the prompt. The response may include contextual suggestions. In some implementations, the response may include content generated based on the prompt and search results.

[0076] In step 708, the system receives the response and displays the response in a contextual search area. The contextual search area is outside the browser's display area but within the browser's control. The contextual search area may be an overlay interface that partially obscures the display area. The contextual search area may be at the edge of the display area, for example, reducing the size of the display area to make space on the display for the contextual search area. The contextual search area is considered to be outside the display area whether it is displayed simultaneously with the display area or as an overlay window.

[0077] Once the contextual search area is displayed, the user may interact with the content of the contextual search area and / or submit other prompts as described with respect to steps 610-622 of FIG.

[0078] The contextual search tool described above can be used to provide additional context in a variety of settings. In one example, a user may be viewing a news item from a news source, such as a social media site, a news feed, or a publishing website. The contextual search tool may provide the user with a broader context for the news item, allowing them to verify the authenticity of the image, explore contrasting perspectives, and / or access the veracity of the source. In other settings, a traveler reading a travel blog may be able to obtain reviews, search geographic information, and / or make a reservation. In other settings, a shopper may use the contextual search tool to identify products by image, compare multiple products, consolidate user reviews, find deals on items, and so on. In other contexts, a learner doing homework may be able to obtain relevant educational content, view step-by-step instructions on how to solve a problem, ask related questions to explore a subject more deeply, and obtain multiple perspectives. Implementations can support other types of travel.

[0079] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs (e.g., computer-implemented methods) executable and / or readable on a programmable system including at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0080] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural programming language, an object-oriented programming language, and / or an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LED (light emitting diode), or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Similarly, other types of devices can be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic input, speech input, or tactile input.

[0082] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., a data server, etc.), or that includes middleware components (e.g., an application server), or that includes front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or that includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0083] Several embodiments have been described. Of course, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosed embodiments.

[0084] Furthermore, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Furthermore, additional steps may be provided in or removed from the described flows, additional components may be added to, or other components may be removed from, the described systems.

[0085] Clause 1. A computer-implemented method, the method including receiving a contextual search request related to main content, the main content being displayed within a display area of ​​a browser, the method further including extracting at least some content from the main content; receiving contextual suggestions associated with the at least some content; and displaying the contextual suggestions in a contextual search area within the browser in response to the contextual search request.

[0086] Clause 2. The method of clause 1, wherein the context suggestion is based on a category of the main content determined based on the at least some content.

[0087] Clause 3. The method of clause 1 or 2, wherein the context suggestion is one of a plurality of context suggestions, and the plurality of context suggestions are ordered based on a category of the main content determined based on the at least some content.

[0088] Clause 4. The method of clause 1, wherein the context suggestion is one of a plurality of context suggestions, and the plurality of context suggestions are ordered based on a category of an entity identified in the at least some content.

[0089] Clause 5. The method of clause 1, wherein the contextual suggestion is a search suggestion, and the method further includes receiving a selection of the search suggestion, obtaining a search results page using the search suggestion as a query, and displaying the search results page within the contextual search area.

[0090] Clause 6. The method of clause 5, further comprising receiving a selection of a search result from the search results page, fetching second main content of a resource associated with the search result, extracting second content from the second main content, receiving second contextual suggestions associated with the second content, and displaying the second contextual suggestions within the contextual search area.

[0091] Clause 7. The method of clause 1, wherein the context suggestions are determined based at least in part on the content and preference information.

[0092] Clause 8. The method of any of clauses 1 to 7, wherein the at least some content is identified by a model that uses a Document Object Model of the main content as input.

[0093] Clause 9. The method of any of clauses 1 to 7, wherein the at least some content is identified by a model that uses the accessibility tree of the main content as input.

[0094] Clause 10. The method of any of clauses 1-7, wherein the at least some content is identified by a model trained to exclude user information from the at least some content.

[0095] Clause 11. The method of any of clauses 1 to 10, wherein the at least some content is identified by a model trained to filter out from the at least some content information provided by entities not associated with the provider of the main content.

[0096] Clause 12. The method of clause 1, wherein the context suggestions include generated content that describes entities identified in the at least some of the content.

[0097] Clause 13. The method of clause 1, wherein the contextual suggestions are suggested searches related to entities identified in the at least some content.

[0098] Clause 14. The method of clause 1, wherein the context suggestion is a description of the main content.

[0099] Clause 15. The method of any of clauses 1 to 14, further comprising: detecting a scroll input; and, in response to detecting the scroll input, modifying the contextual suggestions based on the main content within a viewport of the display area.

[0100] Clause 16. The method of clause 15, wherein modifying the contextual suggestions includes re-extracting content from the main content currently in the viewport and receiving contextual suggestions associated with the re-extracted content.

[0101] Clause 17. The method of clause 15, wherein the context suggestion is a first context suggestion of a plurality of context suggestions, and modifying the context suggestion includes reordering the plurality of context suggestions based on the scrolling input, and replacing the first context suggestion with a second context suggestion of the plurality of context suggestions based on the reordering.

[0102] Clause 18. The method of any of clauses 1-17, wherein the at least some content includes an indication of content that is visible in a viewport.

[0103] Clause 19. The method of clause 18, wherein the content that is visible in the viewport has a higher weight in determining the context suggestions than content that is not visible in the viewport.

[0104] Clause 20. A computer-implemented method, the method including receiving a contextual search request for main content, the main content being displayed within a display area of ​​a browser, the method further including receiving a prompt input; extracting content from the main content; receiving generated content based on the prompt input and the extracted content; and displaying the generated content within a contextual search area within the browser.

[0105] Clause 21. The method of clause 20, further comprising receiving a selected area of ​​the main content, the selected area being used as the prompt input.

[0106] Clause 22. The method of clause 21, wherein the selection area is an area determined to correspond to an object, and receiving the selection area includes detecting a hover input over the area, and in response to detecting the hover input, changing an appearance of the area during the hover input, and receiving a selection of the area.

[0107] Clause 23. The method of any of clauses 20-22, further comprising providing a dialog input area in a condensed context search area, wherein the prompt input is received in the dialog input area.

[0108] Clause 24. A method according to any of clauses 20 to 23, wherein the generated context includes a search suggestion, the method further comprising receiving a selection of the search suggestion, sending a query associated with the search suggestion to a search engine, receiving a search results page for the query, and displaying the search results page within the context search area.

[0109] Clause 25. The method of clause 24, further comprising receiving a selection of a search result from the search result page, fetching second main content of a resource associated with the search result, extracting the second content from the second main content, receiving contextual suggestions associated with the second content in response to providing the content to the search engine, and displaying the contextual suggestions within the contextual search area.

[0110] Clause 26. The method of any of clauses 20 to 25, wherein the generated content is determined based on the extracted content, the prompt input, and preference information.

[0111] Clause 27. The method of any of clauses 20 to 26, wherein the extracted content is extracted by a model that uses a Document Object Model of the main content as input.

[0112] Clause 28. The method of any of clauses 20 to 26, wherein the extracted content is extracted by a model that uses the accessibility tree of the main content as input.

[0113] Clause 29. The method of any of clauses 20 to 28, wherein the extracted content is identified by a model trained to exclude user information from the extracted content.

[0114] Clause 30. The method of any of clauses 20 to 29, wherein the extracted content is identified by a model trained to filter out from the extracted content information provided by entities not associated with the provider of the main content.

[0115] Clause 31. The method of any of clauses 20 to 30, wherein the prompt input identifies an entity and the generated content includes information about the entity.

[0116] Clause 32. The method of any of clauses 20 to 31, further comprising receiving contextual search suggestions based on the extracted content, and displaying the contextual suggestions together with the generated content in the contextual search area.

[0117] Clause 33. The method of any of clauses 20 to 32, wherein the generated content relates to entities identified in the extracted content.

[0118] Clause 34. A computing device comprising: a processor formed in a substrate; and a memory storing instructions that, when executed by the processor, cause the computing device to perform a method according to any one of clauses 1 to 33.

Claims

1. 1. A computer-implemented method comprising: receiving a contextual search request related to main content, the main content being displayed within a display area of ​​a browser, the method further comprising: extracting at least a portion of the content from the main content; receiving context suggestions associated with the at least some of the content; displaying the context suggestions in a context search area within the browser in response to the context search request; A method comprising:

2. The method of claim 1 , wherein the context suggestions are based on a category of the main content that is determined based on the at least some content.

3. The method of claim 1 , wherein the context suggestion is one of a plurality of context suggestions, the plurality of context suggestions being ordered based on a category of the main content determined based on the at least some content.

4. The method of claim 1 , wherein the context suggestion is one of a plurality of context suggestions, the plurality of context suggestions being ordered based on a category of an entity identified in the at least some of the content.

5. The context suggestions are search suggestions, and the method further comprises: receiving a selection of the search suggestions; obtaining a search result page using the search suggestions as a query; and displaying the search results page within the contextual search area; The method of claim 1 , comprising:

6. receiving a selection of search results from the search results page; fetching second main content of a resource associated with the search result; and Extracting second content from the second main content; receiving a second context suggestion associated with the second content; displaying the second context suggestion in the context search area; The method of claim 5 further comprising:

7. The method of claim 1 , wherein the context suggestions are determined based at least in part on the content and preference information.

8. The method of claim 1 , wherein the at least some content is identified by a model that uses a document object model of the main content as input.

9. The method of claim 1 , wherein the at least some content is identified by a model that uses an accessibility tree of the main content as input.

10. The method of claim 1 , wherein the at least some content is identified by a model trained to exclude user information from the at least some content.

11. The method of claim 1 , wherein the at least some content is identified by a model trained to filter out from the at least some content information provided by entities not associated with a provider of the main content.

12. The method of claim 1 , wherein the context suggestions include generated content that describes entities identified in the at least some of the content.

13. The method of claim 1 , wherein the contextual suggestions are suggested searches related to entities identified in the at least some of the content.

14. The method of claim 1 , wherein the context suggestion is a description of the main content.

15. Detecting a scroll input; modifying the contextual suggestions based on the main content within a viewport of the display area in response to detecting the scroll input; The method of claim 1 further comprising:

16. Modifying the context proposal may include: re-extracting content from the main content currently in the viewport; receiving context suggestions associated with the re-extracted content; 16. The method of claim 15, comprising:

17. The context proposal is a first context proposal of a plurality of context proposals, and modifying the context proposal includes: reordering the plurality of context suggestions based on the scrolling input; and replacing the first context suggestion with a second context suggestion of the plurality of context suggestions based on the reordering; 16. The method of claim 15, comprising:

18. The method of claim 1 , wherein the at least some content comprises an indication of content that is visible within a viewport.

19. The method of claim 18 , wherein the content that is visible in the viewport has a higher weight in determining the context suggestions than content that is not visible in the viewport.

20. 1. A computer-implemented method comprising: receiving a contextual search request for main content, the main content being displayed within a display area of ​​a browser, the method further comprising: receiving a prompt input; extracting content from the main content; receiving content generated based on the prompt input and the extracted content; displaying the generated content in a contextual search area within the browser; A method comprising:

21. receiving a selection of the main content; The method of claim 20 , wherein the selection area is used as the prompt input.

22. The selection area is an area determined to correspond to an object, and receiving the selection area includes: Detecting a hover input over the region; In response to detecting the hover input, modifying an appearance of the region during the hover input; receiving a selection of the region; 22. The method of claim 21, comprising:

23. providing a dialog entry area in the condensed context search area; The method of claim 20 , wherein the prompt input is received in the dialog input area.

24. The generated content includes search suggestions, and the method further comprises: receiving a selection of the search suggestions; submitting a query associated with the search suggestion to a search engine; receiving a search result page for the query; displaying the search results page within the contextual search area; 21. The method of claim 20, comprising:

25. receiving a selection of search results from the search results page; fetching second main content of a resource associated with the search result; and Extracting second content from the second main content; receiving context suggestions associated with the second content in response to providing the content to the search engine; displaying the context suggestions within the context search area; 25. The method of claim 24, further comprising:

26. The method of claim 20 , wherein the generated content is determined based on the extracted content, the prompt input, and preference information.

27. The method of claim 20 , wherein the extracted content is extracted by a model that uses a document object model of the main content as input.

28. The method of claim 20 , wherein the extracted content is extracted by a model that uses the accessibility tree of the main content as input.

29. 21. The method of claim 20, wherein the extracted content is identified by a model trained to exclude user information from the extracted content.

30. 21. The method of claim 20, wherein the extracted content is identified by a model trained to filter out from the extracted content information provided by entities not associated with a provider of the main content.

31. The method of claim 20 , wherein the prompt input identifies an entity and the generated content includes information about the entity.

32. receiving contextual search suggestions based on the extracted content; and displaying the contextual search suggestions along with the generated content in the contextual search area; 21. The method of claim 20, further comprising:

33. The method of claim 20 , wherein the generated content relates to an entity identified in the extracted content.

34. 1. A computing device, comprising: a processor formed in the substrate; A computing device comprising: a memory storing instructions that, when executed by the processor, cause the computing device to perform a method according to any of claims 1 to 33.

Images