Surface relevant content while browsing and searching

Abstract

Systems and methods for presenting an interface for additional content suggestions can include obtaining data describing displayed content and determining additional content associated with the displayed content. An interface displaying data associated with the displayed content and the additional content can then be provided. The interface can include a first viewing window for displaying a portion of the displayed content and a second viewing window for displaying a snippet associated with the additional content.

Description

[0001] Related applications

[0002] This application is based upon and claims priority to U.S. Non-Provisional Patent Application No. 18 / 081,832, filed December 15, 2022, which claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 410,433, filed September 27, 2022. Applicants claim priority to and the benefit of each of such applications and all of such applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to presenting additional content based on currently displayed content. More particularly, the present disclosure relates to obtaining data indicating displayed content being provided, determining additional content associated with the displayed content, and providing an interface having data associated with the displayed content and the additional content. Background Art

[0004] When viewing a content item (such as a web page), a user may read the information provided about the topic from beginning to end and / or view only a small portion of the information provided about the topic. Additionally, the information may be outdated and / or may not be the most reliable information. Alternatively and / or additionally, the user may want to better understand and / or interact with the information; however, the user may be limited to manually performing additional searches and / or bookmarking the web page.

[0005] Articles and other content items may be long and / or may discuss off-topic topics only in passing. The length and / or lack of full context may create additional barriers for readers that may lead to further searching and may be time-consuming. Summary of the Invention

[0006] Various aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] An example aspect of the present disclosure relates to a computing system for content prediction. The computing system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining content data. The content data may include an indication of displayed content provided for display to a user. The operations may include determining additional content associated with the displayed content. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during the presentation of the displayed content. The operations may include providing an interface for viewing data associated with the displayed content and the additional content in response to determining the additional content associated with the displayed content. The interface may include a suggestion state. The suggestion state may include a viewing window that displays at least a portion of the displayed content. The suggestion state may include a suggestion interface element indicating the determination of the additional content.

[0008] In some implementations, the displayed content may be associated with a web page. The content data may include a uniform resource locator. The interface may include a web page viewer and a preview bubble. In some implementations, the web page viewer may provide a portion of the displayed content for display. The preview bubble may provide a snippet associated with the additional content. The interface may include a scroll indicator and a bubble interface element. In some implementations, the scroll indicator may indicate the position of the currently viewed portion of the displayed content relative to other portions of the displayed content. The bubble interface element may be provided adjacent to the scroll indicator in the interface. The additional content may include a purchase link. The purchase link may be associated with a product associated with the displayed content. In some implementations, the additional content may include an augmented reality experience. The interface may include selectable user interface elements for providing the augmented reality experience.

[0009] In some implementations, the operation may include providing a suggested interface element for display in a first state. The suggested interface element may describe whether additional content has been determined. The operation may include providing a suggested interface element for display in a second state in response to determining additional content associated with the displayed content. The second state may describe that the additional content is being determined. In some implementations, the operation may include obtaining input data. The input data may describe a selection of a suggested interface element of the interface. The operation may include providing a portion of the additional content for display.

[0010] In some implementations, determining additional content associated with the displayed content may include: determining a uniform resource locator associated with the displayed content; and determining an additional webpage associated with the uniform resource locator. Determining additional content associated with the displayed content may also include generating the additional content based on the additional webpage. In some implementations, determining additional content associated with the displayed content may include: determining multiple additional resources associated with the displayed content; determining multiple predicted actions associated with one or more of the multiple additional resources; and generating multiple action interface elements. Multiple action interface elements may be associated with multiple predicted actions. Multiple action interface elements may be provided for display in the interface.

[0011] In some implementations, determining additional content associated with the displayed content can include: processing at least a portion of the displayed content with a machine learning model to determine a machine learning output; and determining the additional content based on the machine learning output. The interface can include a swipe-up interface element configured to display a portion of the additional content based on user input.

[0012] In some implementations, providing an interface for viewing data associated with displayed content and additional content may include: providing at least a portion of the displayed content for display using a suggested interface element; obtaining a selection of the suggested interface element; and providing at least a portion of the additional content for display. The operations may include processing a portion of the displayed content to generate semantic data. The semantic data may describe a semantic understanding of the portion of the displayed content. The operations may query a database based at least in part on the semantic data. The additional content may be determined based on the query of the database.

[0013] In some implementations, the interface can include a type indicator associated with the content type of the additional content. The type indicator can describe the action type. The additional content can be associated with performing a specific action. In some implementations, the type indicator can describe the understanding type. The additional content can provide supplemental information for understanding a specific topic associated with the displayed content.

[0014] Another example aspect of the present disclosure relates to a computer-implemented method for providing additional content. The method may include obtaining content data by a computing system including one or more processors. The content data may include an indication of displayed content provided for display to a user. The method may include processing the content data with a machine learning model by the computing system to generate a machine learning model output. The machine learning output may describe a semantic understanding of the displayed content. The method may include determining, by the computing system, additional content associated with the displayed content based on the machine learning model output. In some implementations, the additional content may be obtained based on the content data. The additional content may be determined by processing the content data during presentation of the displayed content. The method may include providing, by the computing system, an interface for viewing data associated with the displayed content and the additional content in response to determining the additional content associated with the displayed content. The interface may include a viewing window that displays at least a portion of the displayed content. In some implementations, the interface may include a suggestion notification describing the additional content.

[0015] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations may include obtaining content data. The content data may include an indication of displayed content provided for display to a user. The operations may include processing the content data to determine an entity associated with the displayed content. The operations may include determining additional content associated with the displayed content based on the entity. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during presentation of the displayed content. The operations may include providing an interface for viewing data associated with the displayed content and the additional content. The interface may include a viewing window that displays at least a portion of the displayed content. In some implementations, the interface may include a suggestion notification describing the additional content.

[0016] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0017] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] With reference to the accompanying drawings, a detailed discussion of the embodiments for persons of ordinary skill in the art is set forth in this specification, in which:

[0019] Figure 1 Depicted is a block diagram of an example additional content suggestion system according to an example embodiment of the present disclosure.

[0020] Figures 2A-2B Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0021] Figure 3 Depicted are diagrams of example suggestion interface elements according to example embodiments of the present disclosure.

[0022] Figure 4 Depicted is a diagram of an example scrolling interface according to an example embodiment of the present disclosure.

[0023] Figures 5A-5C Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0024] Figure 6 Depicted is a diagram of an example tray of actions interface according to an example embodiment of the present disclosure.

[0025] Figure 7 Depicted is a diagram of an example entry point element according to an example embodiment of the present disclosure.

[0026] Figure 8 Depicted is a diagram of an example preview bubble according to an example embodiment of the present disclosure.

[0027] Figure 9 Depicted are diagrams of example type indicators according to example embodiments of the present disclosure.

[0028] Figure 10 Depicted is a diagram of an example additional content window according to an example embodiment of the present disclosure.

[0029] Figure 11 Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0030] Figure 12 A diagram depicting example suggested interface element transitions according to an example embodiment of the present disclosure is depicted.

[0031] Figure 13 Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0032] Figure 14 Depicted are diagrams of example suggestion interface elements according to example embodiments of the present disclosure.

[0033] Figure 15 Depicted is a flow diagram of an example method for performing additional content interface presentation according to an example embodiment of the present disclosure.

[0034] Figure 16 Depicted is a flow diagram of an example method for performing additional content determination according to an example embodiment of the present disclosure.

[0035] Figure 17 Depicted is a flow diagram of an example method for performing entity-based additional content determination according to an example embodiment of the present disclosure.

[0036] Figure 18A Depicted is a block diagram of an example computing system that performs additional content interface presentation according to an example embodiment of the present disclosure.

[0037] Figure 18B Depicted is a block diagram of an example computing device performing additional content interface presentation according to an example embodiment of the present disclosure.

[0038] Figure 18C Depicted is a block diagram of an example computing device performing additional content interface presentation according to an example embodiment of the present disclosure.

[0039] Reference numerals that are repeated across multiple drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0040] Generally, the present disclosure relates to systems and methods for providing an interface for accessing additional content associated with a displayed content item. In particular, the systems and methods disclosed herein can utilize additional content predictions to provide information associated with the displayed content, which can provide supplemental information to more fully understand the topic and / or provide user interface elements to perform actions associated with the displayed content. The systems and methods can utilize one or more search engines, one or more databases, one or more machine learning models, and / or one or more user interface elements. The systems and methods disclosed herein provide suggestions that can proactively determine other information and / or other actions that may be useful to the user. For example, the systems and methods may include obtaining content data. The content data may include an indication of the displayed content provided for display to the user. The systems and methods may include determining additional content associated with the displayed content. The additional content may be obtained based on the content data. The systems and methods may include providing an interface for viewing data associated with the displayed content and the additional content.

[0041] The systems and methods can include obtaining content data. The content data can include an indication of displayed content provided for display to a user. In some implementations, the displayed content can be associated with a webpage. The content data can include a uniform resource locator. The displayed content can include a webpage, a video, a book, and / or a mobile application. The content data can include a uniform resource locator, textual data, image data, latent encoded data, and / or other metadata associated with the displayed content. The displayed content can include a webpage, a document, and / or other information provided for display on a computing device. Obtaining content data can include obtaining textual data, image data, structural data, and / or latent encoded data currently being provided in a viewer and generating content data describing the obtained data. Alternatively and / or additionally, obtaining content data can include processing source code, obtaining database data associated with a uniform resource locator, and / or processing a complete webpage to generate one or more inclusions.

[0042] The systems and methods can include determining additional content associated with the displayed content. The additional content can be obtained based on the content data. In some implementations, the additional content can include a purchase link. The purchase link can be associated with a product associated with the displayed content. The additional content can include an augmented reality experience. The additional content can be obtained from one or more databases and / or can be generated based on the displayed content and / or one or more other resources. The additional content determination can be performed automatically in the background without prompting by the user. Alternatively and / or additionally, the user can select one or more user interface elements to request the additional content determination. In some implementations, the additional content determination can occur during display of the displayed content.

[0043] In some implementations, determining additional content associated with the displayed content can include determining a uniform resource locator associated with the displayed content and determining an additional webpage associated with the uniform resource locator. Additionally and / or alternatively, the additional content can be generated based on the additional webpage. The additional webpage can include a webpage that references the displayed content and / or a webpage associated with the uniform resource locator by a search engine and / or a knowledge graph. The additional webpage can provide similar and / or contradictory information.

[0044] In some implementations, determining additional content associated with the displayed content can include determining a plurality of additional resources associated with the displayed content, determining a plurality of predicted actions associated with one or more resources of the plurality of additional resources, and generating a plurality of action interface elements. The plurality of action interface elements can be associated with the plurality of predicted actions. The plurality of action interface elements can be provided for display in an interface.

[0045] Alternatively and / or additionally, determining additional content associated with the displayed content may include: processing at least a portion of the displayed content with a machine learning model to determine a machine learning output; and determining the additional content based on the machine learning output.

[0046] The systems and methods may include providing an interface for viewing data associated with displayed content and additional content. The interface may include a web viewer and preview bubbles. In some implementations, the web viewer may provide a portion of the displayed content for display. The preview bubble may provide a snippet associated with the additional content. In some implementations, the interface may include an upward swipe interface element that is configured to display a portion of the additional content based on user input. The interface may include a type indicator associated with the content type of the additional content. For example, the type indicator may describe an action type, and the additional content may be associated with performing a specific action. Alternatively and / or additionally, the type indicator may describe an understanding type. The additional content may provide supplemental information for understanding a specific topic associated with the displayed content. The interface may include a selectable user interface element for providing an augmented reality experience associated with the topic of the displayed content.

[0047] In some implementations, the interface may include a scroll indicator and a bubble interface element. The scroll indicator may indicate the position of the currently viewed portion of the displayed content relative to other portions of the displayed content. Additionally and / or alternatively, a bubble interface element may be provided in the interface adjacent to the scroll indicator. The bubble interface element may move within the display as the scroll indicator moves. The bubble interface element may provide data associated with the determined additional content for display. In some implementations, the data provided for display in the bubble interface element may change as different additional content is determined. For example, the beginning of a webpage may discuss a first topic, and an additional webpage that discusses the first topic in detail may be identified and provided as suggested additional content. The user may scroll to the middle portion of the webpage that discusses a second topic, and a second additional webpage that discusses the second topic in detail may be identified and provided as suggested additional content. The user may then scroll to the bottom portion of the webpage that offers an object for sale at a set price. The bubble interface element may then provide options to track prices and / or suggest different web resources that sell the object at a lower price.

[0048] In some implementations, providing an interface for viewing data associated with displayed content and additional content may include: providing at least a portion of the displayed content for display with a suggested interface element; obtaining a selection of the suggested interface element; and providing at least a portion of the additional content for display.

[0049] Additionally and / or alternatively, the systems and methods may include providing a suggestion interface element for display in a first state. The suggestion interface element may describe whether additional content has been determined. In response to determining additional content associated with the displayed content, the systems and methods may provide a suggestion interface element for display in a second state. The second state may describe that additional content is being determined.

[0050] In some implementations, the systems and methods can include obtaining input data. The input data can describe a selection of a suggested interface element for an interface. The systems and methods can include providing a portion of the additional content for display based on the input data.

[0051] Alternatively and / or additionally, the systems and methods can include processing a portion of the displayed content (e.g., using one or more machine learning models) to generate semantic data. The semantic data can describe a semantic understanding of the portion of the displayed content. The systems and methods can include querying a database based at least in part on the semantic data. In some implementations, the additional content can be determined based on the query of the database.

[0052] The Internet can provide a vast array of resources on a variety of topics. A user may be viewing and / or reading information provided about a topic. Additional information about the topic may be relevant to the user. The relevant information may be unknown to the user and / or may be desired by the user; however, due to additional search barriers, the user may not obtain the information until later. The systems and methods disclosed herein can automatically process displayed content to determine relevant additional content that can be suggested to the user.

[0053] Additionally and / or alternatively, the information may be outdated and / or may not be the most reliable information. The systems and methods disclosed herein can determine entities associated with displayed content items (e.g., subject matter, author, publisher, and / or knowledge domains associated with the subject matter of the displayed content) and can determine more recent and / or more reliable information about a particular entity to suggest to a user.

[0054] Alternatively and / or additionally, users may want to better understand and / or interact with information; however, users may traditionally be limited to manually performing additional searches and / or bookmarking web pages. The systems and methods disclosed herein can utilize one or more machine learning models to suggest summaries of displayed content. In some implementations, the systems and methods can determine actions associated with the content type of the displayed content and suggest these actions to the user. For example, the displayed content may include an advertisement for a product or service. The systems and methods can determine the advertising content type and suggest a price tracking feature that recursively updates the user about future price changes. In some implementations, the displayed content may include an event (e.g., a football game) and the event content type can be determined. The systems and methods can suggest tracking event updates (e.g., score updates). Actions can include summarizing actions, tracking actions, saving actions, and / or related resource lookup actions (e.g., in response to determining a movie review content type, the systems and methods can suggest a movie theater webpage for booking tickets and / or a web resource containing information about the movie's cast and director).

[0055] Articles and other content items may be very long and / or may discuss off-topic topics only in passing. The length and / or lack of complete context may create additional barriers for readers, which traditionally may result in further searching and may be time-consuming. The systems and methods disclosed herein can proactively determine and suggest summaries of content. Additionally and / or alternatively, the systems and methods can proactively determine relevant off-topic topics in the displayed content. The systems and methods can determine additional content associated with the off-topic topics and can suggest the additional content to the user.

[0056] In response to the information provided in the displayed content item, the user may desire additional information and / or attempt to perform one or more additional actions based on the information provided in the displayed content item. Obtaining additional information and / or performing additional actions may include searching for supplemental information, searching for a purchase portal for purchasing the product discussed in the displayed content item, and / or one or more other additional actions. Additional actions may be time-consuming, and the user may be unsure how to perform such additional actions, which may cause further confusion. The systems and methods disclosed herein can automatically determine additional information and / or additional actions associated with displayed content, and can suggest additional information and / or additional actions to the user.

[0057] The systems and methods of the present disclosure provide a number of technical effects and benefits. As one example, the systems and methods can provide an interface for providing additional content predictions. The additional content predictions can enable a user to perform one or more actions and / or obtain additional information about a topic. The additional content predictions can be provided in an interface that allows a user to view a portion of the additional content while still displaying a portion of the initial content item.

[0058] Another technical benefit of the systems and methods of the present disclosure is the ability to utilize one or more machine learning models to determine that a particular portion of displayed content describes a particular topic, to determine a plurality of different additional content items to provide, where each respective additional content item can be associated with a respective portion of the displayed content item.

[0059] Another example of technical effects and benefits relates to improved computational efficiency and improvements in functionality of computing systems. For example, the systems and methods disclosed herein can utilize additional content predictions to proactively provide resources that a user can desire, which can save time and computational power to navigate to one or more additional webpages to find resources associated with the additional content.

[0060] Example embodiments of the present disclosure will now be discussed in greater detail.

[0061] Figure 1 A block diagram of an example additional content suggestion system 10 in accordance with example embodiments of the present disclosure is depicted. The additional content suggestion system 10 can include obtaining content data associated with displayed content 12, determining additional content 14 associated with the displayed content 12, and providing a suggestion interface 16 for display.

[0062] In particular, the displayed content 12 can include at least a portion of a webpage and / or a portion of a document displayed in a user interface. The content data can include data describing the displayed content 12. The content data can include a uniform resource locator, a text embedding, an image embedding, a portion of source code, textual data, latent encoded data, and / or image data.

[0063] The content data can be processed to determine an entity 20 associated with the displayed content 12. The determined entity 20 can then be utilized to determine the additional content 14. For example, the determined entity 20 can be utilized to generate a search query that can be used to query a search engine and / or a database to determine additional content associated with the determined entity 20.

[0064] Alternatively and / or additionally, the content data may be processed with one or more machine learning models 22 to generate a machine learning model output. The machine learning model output may be additional content 14 and / or may be used to determine the additional content 14. For example, the machine learning model 22 may be trained to summarize content, and the additional content 14 may be a summary of the displayed content 12. Alternatively and / or additionally, the machine learning model 22 may be a semantic understanding model (e.g., a natural language processing model trained for semantic understanding) that may process the displayed content 12 to generate a semantic understanding output. The semantic understanding output may then be used to determine other web resources and / or other documents associated with the semantic understanding.

[0065] In some implementations, the displayed content 24 may be processed to determine one or more actions 24 associated with the displayed content 12. User interface elements for performing the one or more actions 24 may be provided as additional content 14. For example, the displayed content 12 may be determined to include content that may potentially change over time, and tracking actions may be provided as an option to the user. Alternatively and / or additionally, the displayed content 12 may be determined to include an object associated with an augmented reality experience (e.g., a live trial experience), and the augmented reality experience may be provided as an option.

[0066] Displayed content 12 and suggested additional content 14 may be provided for display in a suggestion interface 16. Suggestion interface 16 may be provided for display on a mobile device 30, a desktop device, a smart wearable device, and / or via other display devices. Suggestion interface 16 may include a viewing window 32 for displayed content 12 and a pop-up interface element 34 for additional content 14. Alternatively and / or additionally, additional content 14 may be provided for display in a dynamically moving bubble interface element that moves in unison with a scroll indicator.

[0067] Figures 2A-2B Depicted are diagrams of example interfaces according to example embodiments of the present disclosure. In particular, Figure 2A Depicted are suggestion interface elements in three different states. A first state 202 may include a suggestion interface element provided without a color and / or a badge, which may indicate that additional content has not yet been determined. A second state 204 may include a suggestion interface element having a different color than first state 202, which may indicate that additional content has been determined. A third state 206 may include the suggestion interface element of second state 206 with a badge attached, which may indicate that the determined additional content is provided with a high level of confidence associated with the displayed content.

[0068] Figure 2BAdditional content data being provided in the interface can be depicted. At 208, a preview bubble is provided in the interface. The preview bubble can include a snippet associated with the determined additional content. The snippet can describe information provided by the additional content. The preview bubble can be provided in response to selection of the suggested interface element and / or can be provided automatically.

[0069] At 210, an expanded panel can be provided for display that can include more information about additional content and / or supplemental content associated with the displayed content. The interface depicted in 210 can be provided in response to selection of the suggested interface element and / or the preview bubble. The supplemental content can include additional resources associated with an entity discussed in the displayed content.

[0070] Figure 3 A diagram depicting an example suggested interface element in accordance with example embodiments of the present disclosure is depicted. In some implementations, the suggested interface element can differ based on determined information provided by the displayed content. For example, the suggested interface element can include selectable action elements for performing one or more actions. At 302, a track price action element and a quick checkout element are provided for display in response to determining that the displayed content is associated with a product for sale. The track price action can be used to set an application programming interface that can provide notifications to a user when a price of the product changes. The quick checkout action element can be used to interact with a web platform to purchase the product for sale using stored user data. At 304, a music action element can be provided in response to determining that the displayed content discusses a music artist and / or an album. The music action element can be used to play a song and / or a playlist associated with information provided by the displayed content.

[0071] Figure 4A diagram depicting an example scrolling interface according to example embodiments of the disclosure is illustrated. In some implementations, an interface for providing additional content can include a scrolling interface. The scrolling interface can include a scroll indicator 420, which can indicate the location of the currently viewed portion of the displayed content relative to the entire displayed content. The scrolling interface can additionally include a bubble interface element 430, which can be provided adjacent to the scroll indicator 420. As the user navigates through the displayed content, the bubble interface element 430 can move in unison with the scroll indicator 420. Additionally and / or alternatively, a snippet provided in the bubble interface element 430 can describe additional content that can be viewed. The snippet can change as the user navigates through the displayed content. Additionally and / or alternatively, the additional content can vary based on the particular portion of the displayed content that is currently being displayed. In some implementations, the scrolling interface can be indicated based on a tutorial interface element (e.g., as shown in 402). The additional content can then be determined based on the data provided in the viewing window, and the bubble interface element 430 can be provided for display (e.g., as shown in 404). As the user scrolls further down the page (e.g., the displayed content), a new additional content item can be determined, and the snippet in the bubble interface element 430 can change (e.g., as shown in 406).

[0072] Figures 5A-5C A diagram depicting an example interface according to example embodiments of the disclosure is illustrated. In particular, Figures 5A-5C The interface of FIG. 4 includes a scrolling interface that dynamically changes the snippet of the bubble interface element as the suggested additional content item changes. The dynamic change can be based on changes in information provided as the user navigates through the displayed content. For example, in FIG. 4A, entity-specific information can be obtained to generate a first additional content item. The entity can be determined based on information obtained in a first viewed portion 502 of the displayed content item. A bubble interface element can then be selected to open a first additional window 504 that displays the first additional content item, which can include a link to a mobile application, entity contact information, and a link to learn more about the entity. Figure 5A

[0073] In FIG. 4B, a second viewed portion 506 of the displayed content item can be displayed with an updated bubble interface element. The bubble interface element can be interacted with to open a second additional window 508 that can include a second additional content item generated based on the second viewed portion discussing a particular product. The second additional content item can include a link to open an augmented reality in-store experience to co-view the product in the user’s environment. Figure 5B

[0074] In FIG. 4C, a third viewed portion 510 of the displayed content item can be displayed with an updated bubble interface element. The bubble interface element can be interacted with to open a third additional window 512 that can include a third additional content item generated based on the third viewed portion discussing a particular product. The third additional content item can include a link to open an augmented reality in-store experience to co-view the product in the user’s environment. Figure 5C ​​In the example embodiment, a third viewed portion 510 of the displayed content item can be displayed along with an updated bubble interface element. The bubble interface element can be interacted with to open a third additional window 512, which can include a third additional content item generated based on the third viewed portion discussing the routine or process. The third additional content item can include one or more resources that explain how to perform the routine or process, which can include a video and / or a step-by-step list.

[0075] Figure 6 A diagram of an example action tray interface according to an example embodiment of the present disclosure is depicted. In some implementations, an interface for presenting additional content may include an action tray interface. The action tray interface may include one or more predicted actions determined based on displayed content and / or one or more predetermined actions that may be provided regardless of the information provided by the displayed content. For example, the first action tray interface 602, the second action tray interface 604, the third action tray interface 606, and the fourth action tray interface 608 may all include a bookmark action element to enable a user to bookmark and / or save the displayed content; however, other action elements in the action tray interface may vary based on the specific displayed content. In particular, in response to determining that the displayed content is associated with a product for purchase, the first action tray interface 602 includes a track price action element, a find similar action element, and a compare action element. Additionally and / or alternatively, in response to determining that the displayed content is associated with a media content item (e.g., a video), the second action tray interface includes a mention action element (e.g., viewing other resources that mention the specific displayed content), a compare action element, and a clip action element (e.g., saving a portion of the specific displayed content). In response to determining that the displayed content is associated with a recipe, the third action tray interface 606 includes an ingredient action element (e.g., adding a recipe to a cookbook and / or obtaining and saving an ingredient list), a compare action element, and a clip action element. In response to determining that the displayed content is associated with a product advertisement, the fourth action tray interface 608 includes a mention action element, a find similar action element, and a clip action element.

[0076] Figure 7Illustrations of example entry point elements according to example embodiments of the present disclosure are depicted. Different entry point elements can be utilized uniformly, can differ across platforms, can differ based on displayed content, and / or can differ based on user preferences. For example, the entry point element in 702 includes a multi-colored circular element with a flashing icon, while the entry point element in 704 dynamically changes, can be expanded, can include text, and can include multiple icons. The entry point element in 706 includes a modified entry point element that includes an icon associated with a determined action associated with the determined additional content. Additionally and / or alternatively, the entry point element can differ in color and / or shape when the element is dormant (e.g., when no additional content is currently determined).

[0077] Figure 8 A diagram of an example preview bubble according to an example embodiment of the present disclosure is depicted. Additional content determined and / or generated based on displayed data may include price insights 802 (e.g., one or more shopping lists for products determined to be associated with the displayed content), summaries 804 (e.g., the displayed content may be processed by a machine learning model to generate a summary of the displayed content), augmented reality previews 806 (e.g., an augmented reality experience may be obtained based on the displayed content and provided to the user), ingredient extraction 808 (e.g., ingredients in a recipe may be extracted and saved in a user-specific database), and / or related readings 810 (e.g., supplementary resources associated with the topic in the displayed content may be determined and provided to the user). Each of the different additional content types may be determined and provided based on the displayed content, context, and / or one or more user preferences. A preview bubble with a snippet may then be provided to the user to provide a preview of the obtained and / or generated additional content. The preview bubble and / or suggestion interface elements may be provided via a variety of different interface element shapes and sizes.

[0078] Figure 9 902 , a diagram of an example type indicator is depicted according to an example embodiment of the present disclosure. In particular, in some implementations, a preview bubble can include a type indicator that can indicate a type of action associated with the additional content and / or a level of importance associated with the additional content. For example, a proactive action with low to medium security concern can be associated with a first color indicator 902, while a predetermined question associated with high security concern can be associated with a second color indicator 904.

[0079] Figure 10An illustration of an example additional content window according to an example embodiment of the present disclosure is depicted. An additional content window can be provided for display in response to an interaction with a suggestion interface element and / or a preview bubble. The additional content window can vary based on the additional content type. For example, at 1002, multiple price lists from different suppliers are provided based on the displayed content including products for sale, the multiple price lists having links to access the web pages of the different suppliers and a slider to track price actions. At 1004, a text summary can be provided in a text bubble based on the displayed content including an article. At 1006, multiple different tabs and multiple different search results can be provided based on the displayed content including a search results page.

[0080] Figure 11 Depicted are diagrams of example interfaces according to example embodiments of the present disclosure. In particular, Figure 11 Depicts a transition of the interface from suggestion interface element display 1102 to preview bubble display 1104 to additional content window display 1106. Suggestion interface element display 1102 may include a displayed content window for displaying a portion of displayed content and a suggestion interface element that can be interacted with to provide additional content for display. Preview bubble display 1104 may include the displayed content window, the suggestion interface element, and a preview bubble that may include a snippet providing a preview of the additional content. Additional content window display 1106 may be provided for display in response to one or more obtained inputs and may include an expanded additional content window for viewing one or more additional content items.

[0081] Figure 12 A diagram depicts an example suggestion interface element transition according to an example embodiment of the present disclosure. In some implementations, the suggestion interface element can expand and collapse. In an initial state 1202, the suggestion interface element can include a circular icon. In a secondary state 1204, the suggestion interface element can include an expanded pill with an icon and a text label.

[0082] Figure 13 A diagram of an example interface according to an example embodiment of the present disclosure is depicted. The interface may include an entry point state 1302, a nudge state 1304, and a panel state 1306. The entry point state 1302 may include a displayed content viewing window and suggested interface elements for selection. The nudge state 1304 may include a displayed content viewing window, suggested interface elements for selection, and a preview bubble that provides a snippet indicating possible actions to be performed. The panel state may include an expanded panel for displaying additional content. The interface may transition from one state to another based on one or more inputs and / or one or more determinations.

[0083] Figure 14 An illustration of an example suggestion interface element according to an example embodiment of the present disclosure is depicted. The suggestion interface element can include an icon that can be displayed in different colors and / or with different badges based on one or more determinations. For example, a first state 1402 can include an icon in gray to indicate that additional content has not yet been determined. A second state 1404 can include an icon in one or more other colors to indicate that an additional content item has been determined and can be provided. In some implementations, a badge 1406 can be provided in the second state based on a high confidence level of correlation between the displayed content and the additional content.

[0084] Figure 15 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 15 The steps are depicted as being performed in a particular order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of method 1500 may be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure.

[0085] At 1502, a computing system may obtain content data. The content data may include an indication of displayed content provided for display to a user. In some implementations, the displayed content may be associated with a webpage. The content data may include a uniform resource locator. The displayed content may include text data, image data, white space, structural data, and / or potentially encoded data. The displayed content may be provided for display via a browser application, a messaging application, a social media application, and / or via a widget. The content data may be obtained via an overlay application, a browser extension, a built-in feature of an application, and / or an operating system feature. The displayed content may be associated with a first webpage. The first webpage may be associated with a first web resource.

[0086] At 1504, the computing system may determine additional content associated with the displayed content. The additional content may be obtained based on the content data. The additional content may be determined by processing the content data during presentation of the displayed content. In some implementations, the additional content may include a purchase link. The purchase link may be associated with a product associated with the displayed content. The additional content may include an augmented reality experience. The additional content may be associated with a second web page. The second web page may be different from the first web page. Additionally and / or alternatively, the additional content may be associated with a second web resource that is different from the first web resource.

[0087] In some implementations, determining additional content associated with the displayed content can include: determining a uniform resource locator associated with the displayed content; and determining an additional webpage associated with the uniform resource locator. Additionally and / or alternatively, the additional content can be generated based on the additional webpage.

[0088] In some implementations, determining additional content associated with the displayed content can include: determining multiple additional resources associated with the displayed content; determining multiple predicted actions associated with one or more of the multiple additional resources; and generating multiple action interface elements. The multiple action interface elements can be associated with the multiple predicted actions. The multiple action interface elements can be provided for display in the interface.

[0089] Alternatively and / or additionally, determining additional content associated with the displayed content may include: processing at least a portion of the displayed content with a machine learning model to determine a machine learning output; and determining the additional content based on the machine learning output.

[0090] At 1506, the computing system may provide an interface for viewing data associated with the displayed content and the additional content. The interface may be provided in response to determining additional content associated with the displayed content. The interface may include a web page viewer and a preview bubble. In some implementations, the web page viewer may provide a portion of the displayed content for display. The preview bubble may provide a snippet associated with the additional content. In some implementations, the interface may include a swipe-up interface element configured to display a portion of the additional content based on user input. The interface may include a type indicator associated with the content type of the additional content. For example, the type indicator may describe an action type, and the additional content may be associated with performing a specific action. Alternatively and / or additionally, the type indicator may describe a comprehension type. The additional content may provide supplemental information for understanding a specific topic associated with the displayed content. The interface may include a selectable user interface element for providing an augmented reality experience. In some implementations, the interface may include a suggestion state. The suggestion state may include a viewing window displaying at least a portion of the displayed content. Additionally and / or alternatively, the suggestion state may include a suggestion interface element indicating the determination of the additional content. The suggestion interface element can be selected and an additional content preview window describing at least a portion of the additional content can be provided.In addition to the initially suggested additional content, the additional content preview window can also include one or more other additional content items.

[0091] In some implementations, the interface can include a scroll indicator and a bubble interface element. The scroll indicator can indicate the position of the currently viewed portion of the displayed content relative to other portions of the displayed content. Additionally and / or alternatively, the bubble interface element can be provided adjacent to the scroll indicator in the interface.

[0092] In some implementations, providing an interface for viewing data associated with displayed content and additional content may include: providing at least a portion of the displayed content for display with a suggested interface element; obtaining a selection of the suggested interface element; and providing at least a portion of the additional content for display.

[0093] Additionally and / or alternatively, the systems and methods may include providing a suggestion interface element for display in a first state. The suggestion interface element may describe whether additional content has been determined. In response to determining additional content associated with the displayed content, the systems and methods may provide a suggestion interface element for display in a second state. The second state may describe that additional content is being determined.

[0094] In some implementations, the systems and methods can include obtaining input data. The input data can describe a selection of a suggested interface element for an interface. The systems and methods can include providing a portion of the additional content for display.

[0095] Alternatively and / or additionally, the systems and methods can include processing a portion of the displayed content to generate semantic data. The semantic data can describe a semantic understanding of the portion of the displayed content. The systems and methods can include querying a database based at least in part on the semantic data. In some implementations, the additional content can be determined based on the query of the database.

[0096] Figure 16 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 16 The steps are depicted as being performed in a particular order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement shown. The steps of method 1600 may be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure.

[0097] At 1602, a computing system may obtain content data. The content data may include an indication of displayed content provided for display to a user. The content data may include data describing the displayed content. The displayed content may include a web page and / or a document. The displayed content may be displayed in a browser application, a search application, and / or a dedicated application for a specific content type.

[0098] At 1604, the computing system may process the content data with a machine learning model to generate a machine learning model output. The machine learning output may describe a semantic understanding of the displayed content. The machine learning model may include a natural language processing model, a segmentation model, a classification model, a detection model, and / or an enhancement model. The machine learning model may include a convolutional neural network, a feedforward neural network, a transformer model, and / or a recurrent neural network. The machine learning model output may include embeddings, text data, image data, latent coding data, audio data, and / or code.

[0099] At 1606, the computing system may determine additional content associated with the displayed content based on the machine learning model output. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during presentation of the displayed content. The additional content may include a summary. In some implementations, the additional content may include additional information and / or additional actions determined based on the machine learning model output. The machine learning model output may describe a semantic understanding of the displayed content, which may be used to determine additional content associated with the semantic understanding. In some implementations, the machine learning model output may include a topic determination, which may be used to determine additional content associated with the topic.

[0100] At 1608, the computing system may provide an interface for viewing data associated with the displayed content and the additional content. The interface may be provided in response to determining the additional content associated with the displayed content. In some implementations, the interface may include a viewing window that displays at least a portion of the displayed content. The interface may include a suggestion notification describing the additional content.

[0101] Figure 17 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 17 The steps are depicted as being performed in a particular order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of method 1700 may be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure.

[0102] At 1702, a computing system may obtain content data. The content data may include an indication of displayed content provided for display to a user. The content data may include data describing the displayed content. The displayed content may include a portion of a web page, a portion of a document, and / or other information provided for display.

[0103] At 1704, the computing system may process the content data to determine entities associated with the displayed content. The entities may be determined based on content in the displayed content (e.g., based on a title, an image in the displayed content, and / or information described in a body paragraph), based on data associated with a uniform resource locator, and / or based on an index lookup.

[0104] At 1706, the computing system may determine additional content associated with the displayed content based on the entity. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during presentation of the displayed content. The additional content may be determined by generating a search query based on the entity, providing the search query to a search engine, and receiving one or more search results from the search engine.

[0105] At 1708, the computing system may provide an interface for viewing data associated with the displayed content and the additional content. The interface may include a viewing window that displays at least a portion of the displayed content. In some implementations, the interface may include a suggestion notification that describes the additional content.

[0106] Figure 18A A block diagram of an example computing system 100 that performs additional content interface presentation according to an example embodiment of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0107] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0108] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0109] In some implementations, the user computing device 102 may store or include one or more content prediction models 120. For example, the content prediction model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Figure 11 An example content prediction model 120 is discussed.

[0110] In some implementations, one or more content prediction models 120 may be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single content prediction model 120 (e.g., to perform parallel additional content predictions across multiple instances of a displayed content item).

[0111] More particularly, the content prediction model 120 can be configured to process content data (e.g., uniform resource locators, text data, image data, potential coding data, and / or other metadata) to determine additional content associated with the displayed content. The additional content can be determined by generating semantic data associated with the displayed content and querying a database based on the semantic data. Alternatively and / or additionally, the additional content can be determined by generating a search query based on the content data. In some implementations, the predicted action type can be determined, and the additional content can be determined based on the predicted action type.

[0112] Additionally or alternatively, one or more content prediction models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 in a client-server relationship. For example, the content prediction models 140 may be implemented by the server computing system 140 as part of a web service (e.g., a content prediction service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0113] The user computing device 102 may also include one or more user input components 122 for receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0114] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0115] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0116] As described above, the server computing system 130 may store or otherwise include one or more machine learning content prediction models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Figure 11 An example model 140 is discussed.

[0117] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0118] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc. and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0119] The training computing system 150 can include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, error backpropagation. For example, a loss function can be backpropagated through a model to update one or more parameters of the model (e.g., based on gradients of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over multiple training iterations.

[0120] In some implementations, performing error backpropagation can include performing truncated backpropagation through time. The model trainer 160 can perform various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0121] In particular, the model trainer 160 can train the content prediction model 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, a set of example training data that can include training examples and ground truth data. The training examples can include example content data (e.g., uniform resource locators, example text, example images, example latent encoding data, and / or example embeddings). The ground truth data can include ground truth labels, ground truth predictions, ground truth action types, ground truth queries, and / or ground truth semantic data outputs.

[0122] In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process can be referred to as personalizing the model.

[0123] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium (such as RAM, a hard disk, or optical or magnetic media).

[0124] The network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over the network 180 can be conducted via any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0125] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0126] In some implementations, the input to the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded representation and / or a compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0127] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a latent text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0128] In some implementations, the input to the machine learning model of the present disclosure may be latently coded data (e.g., a latent space representation of the input, etc.). The machine learning model may process the latently coded data to generate an output. As an example, the machine learning model may process the latently coded data to generate a recognition output. As another example, the machine learning model may process the latently coded data to generate a reconstruction output. As another example, the machine learning model may process the latently coded data to generate a search output. As another example, the machine learning model may process the latently coded data to generate a reclustering output. As another example, the machine learning model may process the latently coded data to generate a prediction output.

[0129] In some implementations, the input to a machine learning model of the present disclosure may be statistical data. The machine learning model may process the statistical data to generate an output. As an example, the machine learning model may process the statistical data to generate an identification output. As another example, the machine learning model may process the statistical data to generate a prediction output. As another example, the machine learning model may process the statistical data to generate a classification output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a visualization output. As another example, the machine learning model may process the statistical data to generate a diagnostic output.

[0130] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data from one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to that object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, identifies the likelihood that that region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, the corresponding likelihood of each of a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel in one of the input images, the motion of the scene depicted at that pixel between the images in the network input.

[0131] Figure 18A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 can be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0132] Figure 18B Depicted is a block diagram of an example computing device 40 performing in accordance with an example embodiment of the present disclosure. Computing device 40 may be a user computing device or a server computing device.

[0133] Computing device 40 includes multiple applications (e.g., Application 1 to Application N). Each application includes its own machine learning library and one or more machine learning models. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0134] like Figure 18BAs shown, each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0135] Figure 18C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0136] The computing device 50 includes a plurality of applications (e.g., Application 1 through Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0137] The central intelligence layer includes multiple machine learning models. For example, Figure 18C As shown, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0138] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for the computing device 50. Figure 18C As shown, the central device data layer can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, the central device data layer can use an API (e.g., a private API) to communicate with each device component.

[0139] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0140] Although the present invention has been described in detail with respect to various specific example embodiments of the present invention, each example is provided by way of explanation, not limitation of the present invention. Those skilled in the art, after understanding the foregoing, can easily produce changes, modifications and equivalents of such embodiments. Therefore, the present invention does not exclude such modifications, changes and / or additions to the present invention that will be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present invention is intended to cover such changes, modifications and equivalents.

Claims

1. A computing system for content prediction, the computing system comprising: one or more processors; as well as one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining content data, wherein the content data includes an indication of displayed content provided for display to a user; determining additional content associated with the displayed content, wherein the additional content is obtained based on the content data, wherein the additional content is determined by processing the content data during presentation of the displayed content, wherein the additional content is associated with one or more resources; determining, based on the additional content, that a predicted action is associated with the one or more resources; and In response to determining additional content associated with the displayed content, an interface for viewing data associated with the displayed content and the additional content is provided, wherein the interface includes a suggestion state, wherein the suggestion state includes a viewing window that displays at least a portion of the displayed content, wherein the suggestion state includes a suggestion interface element that indicates the determination of the additional content, and wherein the interface includes an action interface element associated with the predicted action, wherein the action interface element is selectable to perform the predicted action associated with the one or more resources.

2. The computing system of claim 1, wherein the displayed content is associated with a web page, and wherein the content data comprises a uniform resource locator.

3. The computing system of claim 1, wherein the interface comprises a web viewer and a preview bubble, wherein the web viewer provides a portion of the displayed content for display, and wherein the preview bubble provides a snippet associated with the additional content.

4. The computing system of claim 1 , wherein the interface includes a scroll indicator and a bubble interface element, wherein the scroll indicator indicates a position of a currently viewed portion of the displayed content relative to other portions of the displayed content, and wherein the bubble interface element is provided in the interface adjacent to the scroll indicator.

5. The computing system of claim 1, wherein the additional content includes a purchase link, wherein the purchase link is associated with a product associated with the displayed content.

6. The computing system of claim 1, wherein the additional content comprises an augmented reality experience, and wherein the interface comprises selectable user interface elements for providing the augmented reality experience.

7. The computing system of claim 1 , wherein the operations further comprise: providing a suggestion interface element for display in the first state, wherein the suggestion interface element describes whether additional content has been determined; as well as In response to determining the additional content associated with the displayed content, the suggestion interface element is provided for display in a second state, wherein the second state describes that the additional content is being determined.

8. The computing system of claim 1 , wherein the operations further comprise: obtaining input data, wherein the input data describes a selection of a suggested interface element of the interface; as well as A portion of the additional content is provided for display.

9. The computing system of claim 1 , wherein determining the additional content associated with the displayed content comprises: determining a uniform resource locator associated with the displayed content; as well as An additional web page associated with the uniform resource locator is determined.

10. The computing system of claim 9, wherein determining the additional content associated with the displayed content further comprises: Additional content is generated based on the additional web page.

11. The computing system of claim 1 , wherein determining the additional content associated with the displayed content comprises: determining a plurality of additional resources associated with the displayed content; determining a plurality of predicted actions associated with one or more resources of the plurality of additional resources; as well as generating a plurality of action interface elements, wherein the plurality of action interface elements are associated with the plurality of predicted actions; The plurality of action interface elements are provided for display in the interface.

12. The computing system of claim 1 , wherein determining the additional content associated with the displayed content comprises: processing at least a portion of the displayed content with a machine learning model to determine a machine learning output; as well as The additional content is determined based on the machine learning output.

13. The computing system of claim 1, wherein the interface comprises a swipe-up interface element configured to display a portion of the additional content based on user input.

14. The computing system of claim 1 , wherein providing the interface for viewing data associated with the displayed content and the additional content comprises: providing at least a portion of the displayed content for display with a suggested interface element; Obtaining a selection of the suggested interface element; as well as At least a portion of the additional content is provided for display.

15. The computing system of claim 1 , wherein the operations further comprise: processing a portion of the displayed content to generate semantic data, wherein the semantic data describes a semantic understanding of the portion of the displayed content; querying a database based at least in part on the semantic data; and The additional content is determined based on the querying of the database.

16. The computing system of claim 1, wherein the interface includes a type indicator associated with a content type of the additional content.

17. The computing system of claim 16, wherein the type indicator describes an action type, wherein the additional content is associated with performing a particular action.

18. The computing system of claim 16, wherein the type indicator describes a type of comprehension, wherein the additional content provides supplemental information for understanding a particular topic associated with the displayed content.

19. A computer-implemented method for providing additional content, the method comprising: obtaining, by a computing system including one or more processors, content data, wherein the content data includes an indication of displayed content provided for display to a user; processing, by the computing system, the content data with a machine learning model to generate a machine learning model output, wherein the machine learning model output describes a semantic understanding of the displayed content; determining, by the computing system, additional content associated with the displayed content based on the machine learning model output, wherein the additional content is obtained based on the content data, wherein the additional content is determined by processing the content data during presentation of the displayed content; and In response to determining additional content associated with the displayed content, providing, by the computing system, a suggestion interface element for display, wherein the suggestion interface element includes a suggestion notification describing that additional content was determined; obtaining, by the computing system, a selection of the suggested interface element; In response to obtaining the selection of the suggested interface element, the computing system provides a swipe-up interface element for display with the displayed content, an interface for viewing data associated with the displayed content and the additional content, wherein the swipe-up interface element includes a viewing window that displays at least a portion of the displayed content.

20. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: obtaining content data, wherein the content data includes an indication of displayed content provided for display to a user; processing the content data to determine an entity associated with the displayed content; determining, based on the entity, additional content associated with the displayed content, wherein the additional content is obtained based on the content data, wherein the additional content is determined by processing the content data during presentation of the displayed content, wherein the additional content is associated with one or more resources; determining, based on the additional content, that a predicted action is associated with the one or more resources; and An interface is provided for viewing data associated with the displayed content and the additional content, wherein the interface includes a viewing window that displays at least a portion of the displayed content, wherein the interface includes a suggestion notification describing the additional content, and wherein the interface includes an action interface element associated with the predicted action, wherein the action interface element is selectable to perform the predicted action associated with the one or more resources.

Images