Suggesting related content while browsing and searching for content

Abstract

A system and method for presenting an interface for suggesting additional content can include obtaining data indicative of displayed content and determining additional content associated with the displayed content. An interface can then be provided that displays the data associated with the displayed content and the additional content. The interface can include a first display window for displaying a portion of the displayed content and a second display window for displaying a snippet associated with the additional content.

Description

[Technical Field]

[0001] Related Applications This application is based on and claims the benefit of U.S. Nonprovisional Patent Application No. 18 / 081,832, filed December 15, 2022, and claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 410,433, filed September 27, 2022. Applicant claims priority to and the benefit of each of such applications and incorporates by reference herein all such applications in their entireties.

[0002] The present disclosure relates generally to presenting additional content based on currently displayed content, and more particularly to obtaining data indicative of the displayed content being provided, determining additional content associated with the displayed content, and providing an interface to data associated with the displayed content and the additional content. [Background technology]

[0003] When viewing a content item, such as a web page, a user may only read or view a small portion of the information provided on a topic. Furthermore, the information may be outdated and / or may not be the most reliable information. Alternatively and / or additionally, a user may desire to better understand and / or interact with the information. However, the user may be limited to manually performing additional searches and / or bookmarking the web page.

[0004] Articles and other content items may be lengthy and / or may only briefly touch on peripheral topics. Their length and / or lack of full context may pose an additional obstacle to the reader and require further searching, which may be time-consuming. Summary of the Invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0006] One exemplary aspect of the present disclosure is directed to a computing system for content prediction. The computing system may include one or more processors and one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining content data. The content data may include an indication of display content to be provided for display to a user. The operations may include determining additional content associated with the display content. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during presentation of the display content. The operations may include, in response to determining the additional content associated with the display content, providing an interface for viewing the display content and data associated with the additional content. The interface may include a suggestion state. The suggestion state may include a display window displaying at least a portion of the display content. The suggestion state may include a suggestion interface element indicating the determination of the additional content.

[0007] In some implementations, the displayed content can be associated with a web page. The content data can include a uniform resource locator. The interface can include a web page viewer and a preview bubble. In some implementations, the web page viewer can provide a portion of the displayed content for display. The preview bubble can provide a snippet associated with the additional content. The interface can include a scroll indicator and a bubble interface element. In some implementations, the scroll indicator can indicate the location of a currently displayed portion of the displayed content relative to other portions of the displayed content. The bubble interface element can be provided in an interface adjacent to the scroll indicator. The additional content can include a purchase link. The purchase link can be associated with a product associated with the displayed content. In some implementations, the additional content can include an augmented reality experience. The interface can include a selectable user interface element for providing the augmented reality experience.

[0008] In some implementations, the action may include providing a suggested interface element for display in a first state. The suggested interface element may indicate whether additional content has been determined. The action may include providing a suggested interface element for display in a second state in response to determining additional content associated with the displayed content. The second state may indicate the additional content that has been determined. In some implementations, the action may include obtaining input data. The input data may indicate a selection of a suggested interface element in the interface. The action may include providing a portion of the additional content for display.

[0009] In some implementations, determining additional content associated with the displayed content may include determining a uniform resource locator associated with the displayed content and determining an additional web page associated with the uniform resource locator. Determining additional content associated with the displayed content may further include generating the additional content based on the additional web page. In some implementations, determining additional content associated with the displayed content may include determining a plurality of additional resources associated with the displayed content, determining a plurality of predicted actions associated with one or more resources of the plurality of additional resources, and generating a plurality of action interface elements. The plurality of action interface elements may be associated with the plurality of predicted actions. The plurality of action interface elements may be provided for display within the interface.

[0010] In some implementations, determining the additional content associated with the displayed content can include processing at least a portion of the displayed content with a machine-learned model to determine a machine-learned output, and determining the additional content based on the machine-learned output. The interface can include a swipe-up interface element configured to display the portion of the additional content based on user input.

[0011] In some implementations, providing an interface for viewing data associated with the displayed content and the additional content can include providing suggested interface elements for display on at least a portion of the displayed content, obtaining a selection of the suggested interface elements, and providing at least a portion of the additional content for display. The operations can include processing the portion of the displayed content to generate semantic data. The semantic data can indicate a semantic understanding of the portion of the displayed content. The operations can include querying a database based at least in part on the semantic data. The additional content can be determined based on the query of the database.

[0012] In some implementations, the interface may include a type indicator associated with a content type of the additional content. The type indicator may indicate an action type. The additional content may be associated with performing a particular action. In some implementations, the type indicator may indicate a comprehension type. The additional content may provide supplemental information for understanding a particular topic associated with the displayed content.

[0013] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for providing additional content. The method may include, by a computing system including one or more processors, obtaining content data. The content data may include an indication of display content to be provided for display to a user. The method may include, by the computing system, processing the content data with a machine-learned model to generate a machine-learned model output. The machine-learned output may indicate a semantic understanding of the display content. The method may include, by the computing system, determining, based on the machine-learned model output, additional content associated with the display content. In some implementations, the additional content may be obtained based on the content data. The additional content may be determined by processing the content data during presentation of the display content. The method may include, in response to the computing system determining, by the computing system, the additional content associated with the display content, providing an interface for viewing the display content and data associated with the additional content. The interface may include a display window displaying at least a portion of the display content. In some implementations, the interface may include a suggestion notification indicating the additional content.

[0014] Another example aspect of the present disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations can include obtaining content data. The content data can include an indication of display content to be provided for display to a user. The operations can include processing the content data to determine an entity associated with the display content. The operations can include determining additional content associated with the display content based on the entity. The additional content can be obtained based on the content data. In some implementations, the additional content can be determined by processing the content data during presentation of the display content. The operations can include providing an interface for viewing the display content and data associated with the additional content. The interface can include a display window that displays at least a portion of the display content. In some implementations, the interface can include a suggestion notification indicating the additional content.

[0015] Other aspects of the present disclosure relate to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0016] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain associated principles.

[0017] Detailed descriptions of embodiments directed to those skilled in the art are provided herein with reference to the accompanying drawings. [Brief explanation of the drawings]

[0018] [Figure 1] 1 illustrates a block diagram of an exemplary additional content suggestion system according to an exemplary embodiment of the present disclosure. [Figure 2A] 1 illustrates a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 2B] 1 illustrates a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 3] 1 illustrates a diagram of an exemplary suggestions interface element, according to an exemplary embodiment of the present disclosure. [Figure 4] 1 illustrates a diagram of an exemplary scrolling interface, according to an exemplary embodiment of the present disclosure. [Figure 5A] 1 illustrates a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 5B] 1 illustrates a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 5C] 1 illustrates a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 6] 1 illustrates an exemplary tray diagram of an action interface according to an exemplary embodiment of the present disclosure. [Figure 7] 1 illustrates a diagram of an exemplary entry point element according to an exemplary embodiment of the present disclosure. [Figure 8] 10 shows an exemplary preview bubble diagram according to an exemplary embodiment of the present disclosure. [Figure 9] 1A-1C show diagrams of exemplary type indicators according to exemplary embodiments of the present disclosure; [Figure 10] 10 shows a diagram of an exemplary additional content window according to an exemplary embodiment of the present disclosure. [Figure 11] 1 illustrates a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 12] 10A-10C illustrate diagrams of exemplary suggested interface element transitions, according to exemplary embodiments of the present disclosure. [Figure 13] 1 illustrates a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 14] 1 illustrates a diagram of an exemplary suggestions interface element, according to an exemplary embodiment of the present disclosure. [Figure 15] 1 illustrates a flowchart diagram of an exemplary method for performing additional content interface presentation, according to an exemplary embodiment of the present disclosure. [Figure 16] 1 illustrates a flowchart diagram of an exemplary method for making additional content decisions, according to an exemplary embodiment of the present disclosure. [Figure 17] 1 illustrates a flowchart diagram of an exemplary method for making entity-based additional content determinations, according to an exemplary embodiment of the present disclosure. [Figure 18A] 1 illustrates a block diagram of an exemplary computing system for implementing additional content interface presentation, according to an exemplary embodiment of the present disclosure. [Figure 18B] 1 illustrates a block diagram of an exemplary computing device that performs additional content interface presentation, according to an exemplary embodiment of the present disclosure. [Figure 18C] 1 illustrates a block diagram of an exemplary computing device that performs additional content interface presentation, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0019] Reference numbers repeated among the figures are intended to identify like features in the various embodiments.

[0020] In general, the present disclosure is directed to systems and methods for providing an interface for accessing additional content associated with a displayed content item. In particular, the systems and methods disclosed herein can leverage additional content predictions to provide information associated with the displayed content, which may provide supplemental information for a more comprehensive understanding of a topic and / or provide user interface elements for performing actions associated with the displayed content. The systems and methods may utilize one or more search engines, one or more databases, one or more machine-learned models, and / or one or more user interface elements. The systems and methods disclosed herein provide suggestions that can proactively determine other information and / or other actions that may be useful to a user. For example, the systems and methods may include obtaining content data. The content data may include an indication of the displayed content to be provided for display to the user. The systems and methods may include determining additional content associated with the displayed content. The additional content may be obtained based on the content data. The systems and methods may include providing an interface for viewing data associated with the displayed content and the additional content.

[0021] The systems and methods may include obtaining content data. The content data may include an indication of display content to be provided for display to a user. In some implementations, the display content may be associated with a web page. The content data may include a uniform resource locator. The display content may include a web page, a video, a book, and / or a mobile application. The content data may include a uniform resource locator, text data, image data, latent encoded data, and / or other metadata associated with the display content. The display content may include a web page, a document, and / or other information provided for display on a computing device. Obtaining the content data may include obtaining text data, image data, structural data, and / or latent encoded data currently provided to a viewer, and generating content data indicative of the obtained data. Alternatively and / or additionally, obtaining the content data may include processing source code, obtaining database data associated with a uniform resource locator, and / or processing the complete web page to generate one or more embeddings.

[0022] The systems and methods may include determining additional content associated with the displayed content. The additional content may be obtained based on the content data. In some implementations, the additional content may include a purchase link. The purchase link may be associated with a product associated with the displayed content. The additional content may include an augmented reality experience. The additional content may be obtained from one or more databases and / or may be generated based on the displayed content and / or one or more other resources. The additional content determination may be performed automatically in the background without prompting by the user. Alternatively and / or additionally, the user may select one or more user interface elements to request the additional content determination. In some implementations, the additional content determination may be performed during display of the displayed content.

[0023] In some implementations, determining additional content associated with the displayed content can include determining a uniform resource locator associated with the displayed content and determining additional web pages associated with the uniform resource locator. Additionally and / or alternatively, the additional content can be generated based on the additional web pages. The additional web pages can include web pages that cite the displayed content and / or web pages associated with the uniform resource locator by search engines and / or knowledge graphs. The additional web pages can provide similar and / or contradictory information.

[0024] In some implementations, determining additional content associated with the displayed content can include determining a plurality of additional resources associated with the displayed content, determining a plurality of predicted actions associated with one or more of the plurality of additional resources, and generating a plurality of action interface elements. The plurality of action interface elements can be associated with the plurality of predicted actions. The plurality of action interface elements can be provided for display within the interface.

[0025] Alternatively and / or additionally, determining additional content associated with the displayed content may include processing at least a portion of the displayed content with a machine-learned model to determine a machine-learned output, and determining the additional content based on the machine-learned output.

[0026] The systems and methods may include providing an interface for viewing data associated with the displayed content and the additional content. The interface may include a web page viewer and a preview bubble. In some implementations, the web page viewer may provide a portion of the displayed content for display. The preview bubble may provide a snippet associated with the additional content. In some implementations, the interface may include a swipe-up interface element configured to display a portion of the additional content based on user input. The interface may include a type indicator associated with a content type of the additional content. For example, the type indicator may indicate an action type, and the additional content may be associated with performing a particular action. Alternatively and / or additionally, the type indicator may indicate a comprehension type. The additional content may provide supplemental information for understanding a particular topic associated with the displayed content. The interface may include user interface elements selectable to provide an augmented reality experience associated with the topic of the displayed content.

[0027] In some implementations, the interface may include a scroll indicator and a bubble interface element. The scroll indicator may indicate the location of a currently displayed portion of the displayed content relative to other portions of the displayed content. Additionally and / or alternatively, the bubble interface element may be provided in the interface adjacent to the scroll indicator. The bubble interface element may move within the display as the scroll indicator moves. The bubble interface element may provide data for display associated with the determined additional content. In some implementations, the data provided for display in the bubble interface element may change as different additional content is determined. For example, an initial portion of a web page may discuss a first topic, and an additional web page that discusses the first topic in detail may be determined and provided as suggested additional content. A user may scroll to a middle portion of a web page that discusses a second topic, and a second additional web page that discusses the second topic in detail may be determined and provided as suggested additional content. A user may then scroll to the bottom of a web page offering objects for sale at a set price. The bubble interface element may then provide an option to track the price and / or suggest other web resources that have the object for sale at a lower cost.

[0028] In some implementations, providing an interface for viewing data associated with the displayed content and the additional content may include providing suggested interface elements on at least a portion of the displayed content for display, obtaining a selection of the suggested interface elements, and providing at least a portion of the additional content for display.

[0029] Additionally and / or alternatively, the systems and methods can include providing a suggestion interface element for display in a first state. The suggestion interface element can indicate whether additional content has been determined. In response to determining additional content associated with the displayed content, the systems and methods can provide a suggestion interface element for display in a second state. The second state can indicate the additional content that has been determined.

[0030] In some implementations, the systems and methods can include obtaining input data, the input data can indicate a selection of a suggested interface element of the interface, and providing a portion of the additional content for display based on the input data.

[0031] Alternatively and / or additionally, the systems and methods can include processing the portion of the displayed content (e.g., using one or more machine-learned models) to generate semantic data. The semantic data can indicate a semantic understanding of the portion of the displayed content. The systems and methods can include querying a database based at least in part on the semantic data. In some implementations, the additional content can be determined based on the query of the database.

[0032] The Internet can provide a wealth of resources on a variety of topics. A user may be browsing and / or reading information provided about a topic. Additional information about the topic may be relevant to the user. The related information may be unknown to the user and / or desired by the user. However, the information may not be readily available to the user due to additional search effort. The systems and methods disclosed herein can automatically process displayed content to determine additional relevant content that may be suggested to the user.

[0033] Additionally and / or alternatively, the information may be outdated and / or may not be the most reliable information. The systems and methods disclosed herein can determine entities associated with a displayed content item (e.g., topics, authors, publishers, and / or areas of knowledge associated with the topic of the displayed content) and can determine more current and / or more reliable information about a particular entity to suggest to the user.

[0034] Alternatively and / or additionally, a user may desire to better understand and / or interact with the information. However, traditionally, a user may be limited to manually performing additional searches and / or bookmarking web pages. The systems and methods disclosed herein may leverage one or more machine-learned models to suggest summaries of displayed content. In some implementations, the systems and methods may determine an action associated with a content type of the displayed content, which may be suggested to the user. For example, the displayed content may include an advertisement for a product or service. The systems and methods may determine an advertisement content type and suggest a price tracking feature that can keep the user informed of future price changes. In some implementations, the displayed content may include an event (e.g., a football game), and an event content type may be determined. The systems and methods may suggest tracking event updates (e.g., score updates). The actions may include a summary action, a track action, a save action, and / or a related resource search action (e.g., in response to determining a movie review content type, the systems and methods may suggest a theater webpage for reserving tickets and / or suggest a web resource containing information about the movie's cast and director).

[0035] Articles and other content items may be long and / or may only briefly touch on peripheral topics. Their length and / or lack of complete context may pose an additional obstacle to the reader, requiring further searching, which may be time-consuming. The systems and methods disclosed herein can proactively determine and suggest summaries of the content. Additionally and / or alternatively, the systems and methods can proactively determine unrelated topics related to the displayed content. The systems and methods can determine additional content associated with the unrelated topics and suggest the additional content to the user.

[0036] In response to the information provided in the displayed content item, the user may desire additional information and / or may seek to perform one or more additional actions based on the information provided in the displayed content item. Obtaining additional information and / or performing additional actions may include searching for supplemental information, searching a purchasing portal to purchase a product discussed in the displayed content item, and / or one or more other additional actions. Additional actions may be time-consuming, and the user may not know how to perform such additional actions, which may further cause confusion. The systems and methods disclosed herein can automatically determine the additional information and / or additional actions associated with the displayed content and can suggest the additional information and / or additional actions to the user.

[0037] The systems and methods of the present disclosure provide several technical effects and advantages. As an example, the systems and methods may provide an interface for providing additional content predictions. The additional content predictions may enable a user to perform one or more actions and / or obtain additional information about a topic. The additional content predictions may be provided in an interface that allows a user to view portions of the additional content while still viewing portions of the initial content item.

[0038] Another technical advantage of the disclosed systems and methods is that they can utilize one or more machine-learned models to determine that a particular portion of displayed content illustrates a particular topic, thereby determining and providing multiple different additional content items, each of which may be associated with a corresponding portion of the displayed content item.

[0039] Other examples of technical effects and benefits relate to improved computational efficiency and improved functionality of computing systems. For example, the systems and methods disclosed herein can leverage additional content prediction to proactively provide resources that a user may desire, thereby saving time and computational power over navigating to one or more additional web pages to find resources associated with the additional content.

[0040] Referring now to the drawings, exemplary embodiments of the present disclosure will be discussed in further detail.

[0041] 1 illustrates a block diagram of an exemplary additional content suggestion system 10 according to an exemplary embodiment of the present disclosure. The additional content suggestion system 10 may include obtaining content data associated with displayed content 12, determining additional content 14 associated with the displayed content 12, and providing a suggestion interface 16 for display.

[0042] In particular, display content 12 may include at least a portion of a web page and / or a portion of a document displayed in a user interface. Content data may include data indicative of display content 12. Content data may include uniform resource locators, text embeddings, image embeddings, portions of source code, text data, latent encoded data, and / or image data.

[0043] The content data can be processed to determine entities 20 associated with the displayed content 12. The determined entities 20 can then be used to determine additional content 14. For example, the determined entities 20 can be used to generate a search query, which can be used to query a search engine and / or database to determine additional content associated with the determined entities 20.

[0044] Alternatively and / or additionally, the content data can be processed by one or more machine-learned models 22 to generate machine-learned model output. The machine-learned model output can be and / or can be used to determine additional content 14. For example, the machine-learned model 22 can be trained to summarize content, and the additional content 14 can be a summary of the displayed content 12. Alternatively and / or additionally, the machine-learned model 22 can be a semantic understanding model (e.g., a natural language processing model trained for semantic understanding) that can process the displayed content 12 to generate semantic understanding output. The semantic understanding output can then be used to determine other web resources and / or other documents associated with the semantic understanding.

[0045] In some implementations, the display content 24 can be processed to determine one or more actions 24 associated with the display content 12. User interface elements for performing the one or more actions 24 can be provided as additional content 14. For example, the display content 12 can be determined to include content that can potentially change over time, and tracking actions can be optionally provided to the user. Alternatively and / or additionally, the display content 12 can be determined to include objects associated with an augmented reality experience (e.g., a live, wearable experience), and the augmented reality experience can be optionally provided.

[0046] The display content 12 and the suggested additional content 14 may be provided for display within a suggestion interface 16. The suggestion interface 16 may be provided for display via a mobile device 30, a desktop device, a smart wearable, and / or other display device. The suggestion interface 16 may include a display window 32 for the display content 12 and a pop-up interface element 34 for the additional content 14. Alternatively and / or additionally, the additional content 14 may be provided for display within a dynamically moving bubble interface element that moves in concert with scroll indicators.

[0047] 2A and 2B illustrate diagrams of an exemplary search interface according to an exemplary embodiment of the present disclosure. In particular, FIG. 2A illustrates a suggestion interface element in three different states. A first state 202 can include a suggestion interface element provided without color and / or badge, which can indicate additional content that has not yet been determined. A second state 204 can include a suggestion interface element in a different color than the first state 202, which can indicate additional content that has been determined. A third state 206 can include the suggestion interface element of the second state 206 with the addition of a badge, which can indicate that the determined additional content has been determined to be highly relevant to the displayed content.

[0048] 2B may show additional content data being provided within the interface. At 208, a preview bubble is provided within the interface. The preview bubble may include a snippet associated with the determined additional content. The snippet may indicate information provided by the additional content. The preview bubble may be provided in response to a selection of a suggestion interface element and / or may be provided automatically.

[0049] At 210, an expanded panel can be provided for display, which can include additional content and / or more information about auxiliary content associated with the displayed content. The interface shown at 210 may be provided in response to selection of the suggestion interface element and / or preview bubble. The auxiliary content can include additional resources associated with entities discussed in the displayed content.

[0050] FIG. 3 shows a diagram of exemplary suggestion interface elements according to an exemplary embodiment of the present disclosure. In some implementations, the suggestion interface elements can vary based on determined information provided by the displayed content. For example, the suggestion interface elements can include selectable action elements for performing one or more actions. At 302, in response to determining that the displayed content is associated with a product for sale, a track price action element and an express checkout element are provided for display. The track price action can be utilized to configure an application programming interface that can provide a notification to a user when the price of the product changes. The express checkout action element can be utilized to interface with a web platform to purchase the product for sale using stored user data. At 304, in response to determining that the displayed content discusses a music artist and / or album, a music action element can be provided. The music action element can be utilized to play a song and / or playlist associated with information provided by the displayed content.

[0051] FIG. 4 shows a diagram of an exemplary scrolling interface according to an exemplary embodiment of the present disclosure. In some implementations, an interface for providing additional content may include a scrolling interface. The scrolling interface may include a scroll indicator 420 that may indicate the position of a currently displayed portion of the displayed content relative to the entire displayed content. The scrolling interface may further include a bubble interface element 430 that may be provided adjacent to the scroll indicator 420. The bubble interface element 430 may move in conjunction with the scroll indicator 420 as the user navigates through the displayed content. Additionally and / or alternatively, snippets provided in the bubble interface element 430 may indicate additional content that may be viewed. The snippets may change as the user navigates through the displayed content. Additionally and / or alternatively, the additional content may change based on the particular portion of the displayed content that is currently displayed. In some implementations, the scrolling interface may be presented based on a tutorial interface element (e.g., as shown at 402). Thereafter, the additional content may be determined based on the data provided in the view window and provided for display in the bubble interface element 430 (e.g., as shown at 404). As the user scrolls further down the page (e.g., displayed content), new additional content items can be determined and the snippet in the bubble interface element 430 can change (e.g., as shown at 406).

[0052] 5A-5C illustrate diagrams of exemplary interfaces according to exemplary embodiments of the present disclosure. In particular, the interfaces of FIGS. 5A-5C include a scrolling interface that dynamically changes the snippets in the bubble interface element as suggested additional content items change. The dynamic changes can be based on changes in the information provided as the user navigates through the displayed content. For example, in FIG. 5A, entity-specific information can be obtained to generate a first additional content item. The entity can be determined based on information obtained in a first viewing portion 502 of the displayed content item. The bubble interface element can then be selected, which opens a first additional window 504 displaying the first additional content item, which can include a link to a mobile application, contact information for the entity, and a link to more information about the entity.

[0053] 5B, a second viewing portion 506 of the displayed content item can be displayed in an updated bubble interface element that can be interacted with to open a second additional window 508 that can include a second additional content item generated based on the second viewing portion that describes a particular product. The second additional content item can include a link to open a live augmented reality try-on experience, allowing the user to view the product in their environment.

[0054] 5C, a third browsing portion 510 of the displayed content item may be displayed with an updated bubble interface element that may be interacted with to open a third additional window 512 that may contain a third additional content item generated based on the third browsing portion describing the routine or process. The third additional content item may include one or more resources that show how to perform the routine or process, which may include a video and / or a step-by-step list.

[0055] FIG. 6 illustrates an exemplary tray diagram of an interface according to an exemplary embodiment of the present disclosure. In some implementations, an interface for presenting additional content may include a tray of action interfaces. The tray of action interfaces may include one or more predictive actions determined based on the displayed content and / or one or more predefined actions that may be provided regardless of information provided by the displayed content. For example, the first tray of action interface 602, the second tray of action interface 604, the third tray of action interface 606, and the fourth tray of action interface 608 may all include a bookmark action element that allows a user to bookmark and / or save the displayed content. However, the trays of other action interfaces may change based on the particular displayed content. In particular, the first tray of action interface 602 includes a price track action element, a similar search action element, and a compare action element in response to determining that the displayed content is associated with a product for purchase. Additionally and / or alternatively, the second tray of the action interface includes a mention action element (e.g., to see other resources that mention the particular displayed content), a compare action element, and a clip action element (e.g., to save a portion of the particular displayed content) in response to determining that the displayed content is associated with a media content item (e.g., a video). The third tray 606 of the action interface includes an ingredients action element (e.g., add the recipe to a cookbook and / or obtain and save an ingredients list), a compare action element, and a clip action element in response to determining that the displayed content is associated with a recipe. The fourth tray 608 of the action interface includes a mention action element, a find similar action element, and a clip action element in response to determining that the displayed content is associated with a product advertisement.

[0056] FIG. 7 illustrates a diagram of exemplary entry point elements according to an exemplary embodiment of the present disclosure. Different entry point elements may be uniformly utilized, may vary across platforms, may vary based on displayed content, and / or may vary based on user preference. For example, the entry point element at 702 includes a multicolored circular element with a sparkle icon, while the entry point element at 704 may dynamically change, expand, include text, and include multiple icons. The entry point element at 706 includes a modified entry point element that includes an icon associated with a determined action associated with determined additional content. Additionally and / or alternatively, the entry point element may differ in color and / or shape when the element is in a dormant state (e.g., when no additional content is currently being determined).

[0057] FIG. 8 shows a diagram of an exemplary preview bubble, according to an exemplary embodiment of the present disclosure. The additional content determined and / or generated based on the displayed data may include price insights 802 (e.g., one or more shopping lists for products determined to be associated with the displayed content), summaries 804 (e.g., the displayed content may be processed with a machine-learned model to generate a summary of the displayed content), augmented reality previews 806 (e.g., an augmented reality experience may be obtained and provided to the user based on the displayed content), ingredient extraction 808 (e.g., ingredients in a recipe may be extracted and stored in a user-specific database), and / or related reading material 810 (e.g., supplemental resources associated with a topic in the displayed content may be determined and provided to the user). Each of the different additional content types may be determined and provided based on the displayed content, the context, and / or one or more user preferences. A preview bubble containing a snippet may then be provided to the user to provide a preview of the obtained and / or generated additional content. The preview bubble and / or suggested interface elements may be provided via a number of different interface element shapes and sizes.

[0058] 9 shows a diagram of exemplary type indicators, according to an exemplary embodiment of the present disclosure. In particular, in some implementations, the preview bubble may include a type indicator that can indicate a type of action associated with the additional content and / or a level of importance associated with the additional content. For example, an active action with low to medium security concerns may be associated with a first color indicator 902, and a predetermined issue with high security concerns may be associated with a second color indicator 904.

[0059] FIG. 10 shows a diagram of an exemplary additional content window, according to an exemplary embodiment of the present disclosure. In response to interaction with the suggestion interface element and / or the preview bubble, the additional content window can be provided for display. The additional content window may differ based on the additional content type. For example, at 1002, based on displayed content including a product for sale, a list of multiple prices from different vendors may be provided, including links to each vendor's web page and a track price action slider. At 1004, based on displayed content including an article, a text summary may be provided in a text bubble. At 1006, based on displayed content including a search results page, multiple different tabs and multiple different search results may be provided.

[0060] 11 illustrates an exemplary interface diagram according to an exemplary embodiment of the present disclosure. In particular, FIG. 11 illustrates an interface transition from a suggested interface element display 1102 to a preview bubble display 1104 to an additional content window display 1106. The suggested interface element display 1102 may include a display content window for displaying a portion of the displayed content and suggested interface elements that can be interacted with to provide additional content for display. The preview bubble display 1104 may include a display content window, suggested interface elements, and a preview bubble that may include a snippet that provides a preview of the additional content. The additional content window display 1106 may be provided for display in response to one or more obtained inputs and may include an expanded additional content window for viewing one or more additional content items.

[0061] 12 illustrates a transition diagram of an exemplary suggestions interface element, according to an exemplary embodiment of the present disclosure. In some implementations, the suggestions interface element can be expanded or collapsed. In an initial state 1202, the suggestions interface element can include a round icon. In a secondary state 1204, the suggestions interface element can include an expanded pill-shaped shape with an icon and a text label.

[0062] 13 shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. The interface can include an entry point state 1302, a nudge state 1304, and a panel state 1306. The entry point state 1302 can include a display content display window and suggested interface elements for selection. The nudge state 1304 can include a display content display window, suggested interface elements for selection, and a preview bubble that provides a snippet showing a possible action to perform. The panel state can include an expanded panel for displaying additional content. The interface can transition from one state to another based on one or more inputs and / or one or more decisions.

[0063] 14 shows a diagram of an exemplary suggestions interface element, according to an exemplary embodiment of the present disclosure. The suggestions interface element may include an icon that may be displayed in different colors and / or with different badges based on one or more determinations. For example, a first state 1402 may include a gray icon to indicate that additional content has not yet been determined. A second state 1404 may include an icon of one or more other colors to indicate that additional content items have been determined and may be provided. In some implementations, a badge 1406 may be provided in the second state based on a determined high correlation between the displayed content and the additional content.

[0064] 15 shows a flowchart diagram of an exemplary method performed in accordance with an exemplary embodiment of the present disclosure. While FIG. 15 shows steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the particularly shown order or arrangement. Various steps of method 1500 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0065] At 1502, the computing system may obtain content data. The content data may include an indication of display content to be provided for display to a user. In some implementations, the display content may be associated with a web page. The content data may include a uniform resource locator. The display content may include text data, image data, white space, structural data, and / or latent encoded data. The display content may be provided for display via a browser application, a messaging application, a social media application, and / or a widget. The content data may be obtained via an overlay application, a browser extension, a built-in function of the application, and / or an operating system function. The display content may be associated with a first web page. The first web page may be associated with a first web resource.

[0066] At 1504, the computing system may determine additional content associated with the displayed content. The additional content may be obtained based on the content data. The additional content may be determined by processing the content data during presentation of the displayed content. In some implementations, the additional content may include a purchase link. The purchase link may be associated with a product associated with the displayed content. The additional content may include an augmented reality experience. The additional content may be associated with a second web page. The second web page may be different from the first web page. Additionally and / or alternatively, the additional content may be associated with a second web resource different from the first web resource.

[0067] In some implementations, determining the additional content associated with the displayed content may include determining a uniform resource locator associated with the displayed content and determining an additional web page associated with the uniform resource locator. Additionally and / or alternatively, the additional content may be generated based on the additional web page.

[0068] In some implementations, determining additional content associated with the displayed content can include determining a plurality of additional resources associated with the displayed content, determining a plurality of predicted actions associated with one or more of the plurality of additional resources, and generating a plurality of action interface elements. The plurality of action interface elements can be associated with the plurality of predicted actions. The plurality of action interface elements can be provided for display within the interface.

[0069] Alternatively and / or additionally, determining additional content associated with the displayed content may include processing at least a portion of the displayed content with a machine-learned model to determine a machine-learned output, and determining the additional content based on the machine-learned output.

[0070] At 1506, the computing system may provide an interface for viewing data associated with the displayed content and the additional content. The interface may be provided in response to determining additional content associated with the displayed content. The interface may include a webpage viewer and a preview bubble. In some implementations, the webpage viewer may provide a portion of the displayed content for display. The preview bubble may provide a snippet associated with the additional content. In some implementations, the interface may include a swipe-up interface element configured to display a portion of the additional content based on user input. The interface may include a type indicator associated with a content type of the additional content. For example, the type indicator may indicate an action type, and the additional content may be associated with performing a particular action. Alternatively and / or additionally, the type indicator may indicate an understanding type. The additional content may provide supplemental information for understanding a particular topic associated with the displayed content. The interface may include a selectable user interface element for providing an augmented reality experience. In some implementations, the interface may include a suggestion state. The suggestion state may include a display window displaying at least a portion of the displayed content. Additionally and / or alternatively, the suggestion state may include a suggestion interface element indicating a determination of the additional content. The suggested interface element can be selected and an additional content preview window showing at least a portion of the additional content can be provided. The additional content preview window can include one or more other additional content items in addition to the initially suggested additional content.

[0071] In some implementations, the interface can include scroll indicators and bubble interface elements. The scroll indicators can indicate the location of a currently displayed portion of the displayed content relative to other portions of the displayed content. Additionally and / or alternatively, the bubble interface elements can be provided in the interface adjacent to the scroll indicators.

[0072] In some implementations, providing an interface for viewing data associated with the displayed content and the additional content may include providing suggested interface elements on at least a portion of the displayed content for display, obtaining a selection of the suggested interface elements, and providing at least a portion of the additional content for display.

[0073] Additionally and / or alternatively, the systems and methods can include providing a suggestion interface element for display in a first state. The suggestion interface element can indicate whether additional content has been determined. In response to determining additional content associated with the displayed content, the systems and methods can provide a suggestion interface element for display in a second state. The second state can indicate the additional content that has been determined.

[0074] In some implementations, the systems and methods can include obtaining input data, which can indicate a selection of a suggested interface element of the interface, and providing a portion of the additional content for display.

[0075] Alternatively and / or additionally, the systems and methods can include processing the portion of the displayed content to generate semantic data. The semantic data can indicate a semantic understanding of the portion of the displayed content. The systems and methods can include querying a database based at least in part on the semantic data. In some implementations, the additional content can be determined based on the query of the database.

[0076] 16 shows a flowchart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. While FIG. 16 shows steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the particularly shown order or arrangement. Various steps of method 1600 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0077] At 1602, a computing system can obtain content data. The content data can include an indication of display content to be provided for display to a user. The content data can include data indicative of the display content. The display content can include a web page and / or a document. The display content can be displayed within a browser application, a search application, and / or a dedicated application for a particular content type.

[0078] At 1604, the computing system can process the content data with the machine-learned model to generate a machine-learned model output. The machine-learned output can indicate a semantic understanding of the displayed content. The machine-learned model can include a natural language processing model, a segmentation model, a classification model, a detection model, and / or an augmentation model. The machine-learned model can include a convolutional neural network, a feedforward neural network, a transformer model, and / or a recurrent neural network. The machine-learned model output can include embeddings, text data, image data, latent coding data, audio data, and / or code.

[0079] At 1606, the computing system may determine additional content associated with the displayed content based on the machine-learned model output. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during presentation of the displayed content. The additional content may include a summary. In some implementations, the additional content may include additional information and / or additional actions determined based on the machine-learned model output. The machine-learned model output may indicate a semantic understanding of the displayed content and may be utilized to determine additional content associated with the semantic understanding. In some implementations, the machine-learned model output may include a topic determination, which may be utilized to determine additional content associated with the topic.

[0080] At 1608, the computing system can provide an interface for viewing data associated with the displayed content and the additional content. The interface can be provided in response to determining the additional content associated with the displayed content. In some implementations, the interface can include a display window that displays at least a portion of the displayed content. The interface can include a suggestion notification indicating the additional content.

[0081] 17 shows a flowchart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. While FIG. 17 shows steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the particularly shown order or arrangement. Various steps of method 1700 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0082] At 1702, the computing system can obtain content data. The content data can include an indication of display content to be provided for display to a user. The content data can include data indicative of the display content. The display content can include a portion of a web page, a portion of a document, and / or other information provided for display.

[0083] At 1704, the computing system can process the content data to determine entities associated with the displayed content. The entities can be determined based on content within the displayed content (e.g., based on the title, images within the displayed content, and / or information provided in the body of the displayed content), based on data associated with a uniform resource locator, and / or based on an index lookup.

[0084] At 1706, the computing system may determine additional content associated with the displayed content based on the entity. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during presentation of the displayed content. The additional content may be determined by generating a search query based on the entity, providing the search query to a search engine, and receiving one or more search results from the search engine.

[0085] At 1708, the computing system can provide an interface for viewing data associated with the displayed content and the additional content. The interface can include a display window that displays at least a portion of the displayed content. In some implementations, the interface can include a suggestion notification indicating the additional content.

[0086] 18A illustrates a block diagram of an exemplary computing system 100 that performs additional content interface presentation, according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0087] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0088] The computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0089] In some implementations, the user computing device 102 may store or include one or more predictive models 120. For example, the content predictive models 120 may be or otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks), or other types of machine-learned models, including nonlinear and / or linear models. The neural networks may include feed-forward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other types of neural networks. Exemplary content predictive models 120 are discussed with reference to FIGS. 2-11.

[0090] In some implementations, one or more content prediction models 120 may be received from server computing system 130 over network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 can implement multiple parallel instances of a single content prediction model 120 (e.g., to perform parallel additional content prediction across multiple instances of a displayed content item).

[0091] More specifically, the content prediction model 120 can be configured to process content data (e.g., uniform resource locators, text data, image data, latent coding data, and / or other metadata) to determine additional content associated with the displayed content. The additional content can be determined by generating semantic data associated with the displayed content and querying a database based on the semantic data. Alternatively and / or additionally, the additional content can be determined by generating a search query based on the content data. In some implementations, a predicted action type can be determined, and the additional content can be determined based on the predicted action type.

[0092] Additionally or alternatively, one or more content prediction models 140 may be included in or otherwise stored and implemented on a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the content prediction model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a content prediction service). Thus, one or more models 120 may be stored and implemented on the user computing device 102 and / or one or more models 140 may be stored and implemented on the server computing system 130.

[0093] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may function to implement a virtual keyboard. Other exemplary user input components include a microphone, a conventional keyboard, or other means by which a user can provide user input.

[0094] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0095] In some implementations, server computing system 130 includes or is implemented by one or more server computing devices. When server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0096] As described above, the server computing system 130 may store or otherwise include one or more machine-learned content prediction models 140. For example, the models 140 may be or otherwise include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are discussed with reference to FIGS. 2-11.

[0097] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 can be separate from the server computing system 130 or can be part of the server computing system 130.

[0098] Training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operably connected processors. Memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 154 may store data 156 and instructions 158 that are executed by processor 152 to cause training computing system 150 to perform operations. In some implementations, training computing system 150 includes or is implemented by one or more server computing devices.

[0099] The training computing system 150 may include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored on the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent may be used to iteratively update the parameters over several training iterations.

[0100] In some implementations, performing backpropagation may include performing truncated backpropagation over time. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0101] In particular, model trainer 160 can train content prediction model 120 and / or 140 based on a set of training data 162. Training data 162 can include an example training dataset, which can include, for example, training examples and ground truth data. The training examples can include example content data (e.g., uniform resource locators, example text, example images, example latent coding data, and / or example embeddings). The ground truth data can include ground truth labels, ground truth predictions, ground truth action types, ground truth queries, and / or ground truth semantic data outputs.

[0102] In some implementations, if the user consents, training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 with user-specific data received from the user computing device 102. In some instances, this process may be referred to as personalizing the model.

[0103] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some embodiments, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0104] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. Generally, communications over network 180 may occur over any type of wired and / or wireless connection, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or security schemes (e.g., VPN, Secure HTTP, SSL).

[0105] The machine-learned models described herein may be used in a variety of tasks, applications, and / or use cases.

[0106] In some implementations, the input to the machine-learned model(s) of the present disclosure may be image data. The machine-learned model(s) may process the image data to generate an output. As an example, the machine-learned model(s) may process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) may process the image data to generate an image segmentation output. As another example, the machine-learned model(s) may process the image data to generate an image classification output. As another example, the machine-learned model(s) may process the image data to generate an image data modification output (e.g., a modification of the image data, etc.). As another example, the machine-learned model(s) may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine-learned model(s) may process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) may process the image data to generate a predicted output.

[0107] In some implementations, input to the machine-learned model(s) of the present disclosure may be text or natural language data. The machine-learned model(s) may process the text or natural language data to generate an output. As an example, the machine-learned model(s) may process the natural language data to generate a language-encoding output. As another example, the machine-learned model(s) may process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) may process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) may process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) may process the text or natural language data to generate a text segmentation output. As another example, the machine-learned model(s) may process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) may process the text or natural language data to generate upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine-learned model(s) may process the text or natural language data to generate a predicted output.

[0108] In some implementations, the input to the machine-learned model(s) of the present disclosure can be latent-encoded data (e.g., a latent space representation of the input, etc.). The machine-learned model(s) can process the latent-encoded data to generate an output. As an example, the machine-learned model(s) can process the latent-encoded data to generate a recognition output. As another example, the machine-learned model(s) can process the latent-encoded data to generate a reconstruction output. As another example, the machine-learned model(s) can process the latent-encoded data to generate a retrieval output. As another example, the machine-learned model(s) can process the latent-encoded data to generate a reclustering output. As another example, the machine-learned model(s) can process the latent-encoded data to generate a prediction output.

[0109] In some implementations, input to the machine-learned model(s) of the present disclosure can be statistical data. The machine-learned model(s) can process the statistical data to generate an output. As an example, the machine-learned model(s) can process the statistical data to generate a recognition output. As another example, the machine-learned model(s) can process the statistical data to generate a prediction output. As another example, the machine-learned model(s) can process the statistical data to generate a classification output. As another example, the machine-learned model(s) can process the statistical data to generate a segmentation output. As another example, the machine-learned model(s) can process the statistical data to generate a segmentation output. As another example, the machine-learned model(s) can process the statistical data to generate a visualization output. As another example, the machine-learned model(s) can process the statistical data to generate a diagnostic output.

[0110] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data of one or more images and the task is an image processing task. For example, the image processing task can be image classification, and the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to that object class. The image processing task can be object detection, and the image processing output identifies one or more regions in one or more images and, for each region, the likelihood that the region depicts an object of interest. As another example, the image processing task can be image segmentation, and the image processing output specifies, for each pixel in one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the category set can be foreground and background. As another example, the category set can be object classes. As another example, the image processing task can be depth estimation, and the image processing output specifies, for each pixel in one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images and the image processing output specifies, for each pixel of one of the input images, the motion of the scene depicted in pixels between the images in the network input.

[0111] 18A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, a user computing device 102 can include a model trainer 160 and a training dataset 162. In such implementations, a model 120 can be trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 can implement a model trainer 160 that personalizes the model 120 based on user-specific data.

[0112] 18B illustrates a block diagram of an exemplary computing device 40 for performing in accordance with an exemplary embodiment of the present disclosure. The computing device 40 may be a user computing device or a server computing device.

[0113] Computing device 40 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine-learned model(s). For example, each application may include a machine-learned model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0114] 18B, each application may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0115] 18C depicts a block diagram of an exemplary computing device 50 for performing according to an exemplary embodiment of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0116] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a common API across all applications).

[0117] The central intelligence layer includes several machine-learned models. For example, as shown in FIG. 18C, each machine-learned model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., single model) for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by the operating system of computing device 50.

[0118] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As shown in FIG. 18C , the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0119] The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions performed on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes described herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0120] While the present subject matter has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided by way of explanation and not as a limitation of the present disclosure. Those skilled in the art, upon understanding the foregoing, may readily create modifications, variations, and equivalents to such embodiments. Accordingly, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the present subject matter that would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used with other embodiments to create yet another embodiment. Accordingly, the present disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. 1. A computing system for content prediction, comprising: one or more processors; one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations including: obtaining content data, the content data including an indication of display content to be provided for display to a user; determining additional content associated with the displayed content, the additional content being obtained based on the content data, and the additional content being determined by processing the content data during presentation of the displayed content; and a non-transitory computer-readable medium, comprising: in response to determining additional content associated with the displayed content, providing an interface for viewing data associated with the displayed content and the additional content, the interface including a suggested state, the suggested state including a display window displaying at least a portion of the displayed content, and the suggested state including a suggested interface element indicating the determination of the additional content.

2. The computing system of claim 1 , wherein the displayed content is associated with a web page and the content data comprises a uniform resource locator.

3. 10. The computing system of claim 1, wherein the interface includes a web page viewer and a preview bubble, the web page viewer providing a portion of the displayed content for viewing and the preview bubble providing a snippet associated with the additional content.

4. 2. The computing system of claim 1, wherein the interface includes a scroll indicator and a bubble interface element, the scroll indicator indicating a position of a currently displayed portion of the displayed content relative to other portions of the displayed content, and the bubble interface element provided in the interface adjacent to the scroll indicator.

5. The computing system of claim 1 , wherein the additional content includes a purchase link, the purchase link being associated with a product associated with the displayed content.

6. The computing system of claim 1 , wherein the additional content comprises an augmented reality experience, and the interface comprises selectable user interface elements for providing the augmented reality experience.

7. The operation is providing a suggestions interface element for display in a first state, the suggestions interface element indicating whether additional content has been determined; 10. The computing system of claim 1, further comprising: in response to determining the additional content associated with the displayed content, providing the suggested interface element for display in a second state, the second state indicating the determined additional content.

8. The operation is acquiring input data, the input data indicating a selection of a proposed interface element of the interface; The computing system of claim 1 , further comprising: providing a portion of the additional content for display.

9. Determining the additional content associated with the displayed content includes: determining a uniform resource locator associated with the displayed content; and determining additional web pages associated with the uniform resource locator.

10. Determining the additional content associated with the displayed content includes: The computing system of claim 1 , further comprising generating additional content based on the additional web page.

11. Determining the additional content associated with the displayed content includes: determining a plurality of additional resources associated with the displayed content; determining a plurality of predicted actions associated with one or more resources of the plurality of additional resources; generating a plurality of action interface elements, the plurality of action interface elements being associated with the plurality of predicted actions; The computing system of claim 1 , wherein the plurality of action interface elements are provided for display within the interface.

12. Determining the additional content associated with the displayed content includes: processing at least a portion of the displayed content with a machine-learned model to determine a machine-learned output; and determining the additional content based on the machine-learned output.

13. The computing system of claim 1 , wherein the interface includes a swipe-up interface element configured to reveal a portion of the additional content based on user input.

14. Providing the interface for viewing data associated with the displayed content and the additional content includes: providing at least a portion of the display content for display with the suggested interface element; obtaining a selection of the suggested interface element; and providing at least a portion of the additional content for display.

15. The operation is processing the portion of the display content to generate semantic data, the semantic data indicating a semantic understanding of the portion of the display content; querying a database based at least in part on the semantic data; The computing system of claim 1 , wherein the additional content is determined based on the query to the database.

16. The computing system of claim 1 , wherein the interface includes a type indicator associated with a content type of the additional content.

17. The computing system of claim 16 , wherein the type indicator indicates an action type, and the additional content is associated with performing a particular action.

18. The computing system of claim 16 , wherein the type indicator indicates a comprehension type, and the additional content provides supplemental information for understanding a particular topic associated with the displayed content.

19. 1. A computer-implemented method for providing additional content, comprising: obtaining, by a computing system having one or more processors, content data, the content data including an indication of display content to be provided for display to a user; processing, by the computing system, the content data with a machine-learned model to generate a machine-learned model output, the machine-learned model output indicating a semantic understanding of the displayed content; and determining, by the computing system, additional content associated with the displayed content based on the machine-learned model, the additional content being obtained based on the content data, and the additional content being determined by processing the content data during presentation of the displayed content; and in response to determining additional content associated with the displayed content, providing, by the computing system, an interface for viewing data associated with the displayed content and the additional content, the interface including a display window that displays at least a portion of the displayed content, and the interface including a suggestion notification indicating the additional content.

20. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations including: obtaining content data, the content data including an indication of display content to be provided for display to a user; processing the content data to determine entities associated with the displayed content; determining additional content associated with the displayed content based on the entity, the additional content being obtained based on the content data, and the additional content being determined by processing the content data during presentation of the displayed content; providing an interface for viewing data associated with the displayed content and the additional content, the interface including a display window displaying at least a portion of the displayed content, and the interface including a suggestion notification indicating the additional content.

Images