Presenting relevant content while browsing and searching content

Abstract

Systems and methods for presenting an interface for additional content suggestions may include obtaining data describing displayed content and determining additional content associated with the displayed content. An interface may then be provided that displays data associated with the displayed content and the additional content. The interface may include a first viewing window for displaying a portion of the displayed content and a second viewing window for displaying segments associated with additional content.

Description

[0001] Related Applications

[0002] This application is based upon and claims priority to U.S. Non-Provisional Patent Application No. 18 / 081,832, filed on December 15, 2022, which claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 410,433, filed on September 27, 2022. Applicants claim priority to and the benefit of each of such applications and all of such applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to presenting additional content based on currently displayed content. More particularly, the present disclosure relates to obtaining data indicating displayed content being provided, determining additional content associated with the displayed content, and providing an interface having data associated with the displayed content and the additional content. Background Art

[0004] When viewing a content item (such as a web page), a user may read the information provided about the subject from beginning to end and / or view only a small portion of the information provided about the subject. Additionally, the information may be outdated and / or may not be the most reliable information. Alternatively and / or additionally, the user may want to better understand and / or interact with the information; however, the user may be limited to manually performing additional searches and / or bookmarking the web page.

[0005] Articles and other content items may be long and / or may only discuss off-topic topics in passing. The length and / or lack of full context may create additional barriers for readers that may result in further searching and may be time consuming. Summary of the invention

[0006] Various aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] An example aspect of the present disclosure relates to a computing system for content prediction. The computing system may include one or more processors and one or more non-transitory computer-readable media, which together store instructions that, when executed by one or more processors, cause the computing system to perform operations. The operations may include obtaining content data. The content data may include an indication of displayed content provided for display to a user. The operations may include determining additional content associated with the displayed content. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during the presentation of the displayed content. The operations may include providing an interface for viewing data associated with the displayed content and the additional content in response to determining the additional content associated with the displayed content. The interface may include a suggested state. The suggested state may include a viewing window that displays at least a portion of the displayed content. The suggested state may include a suggested interface element indicating the determination of the additional content.

[0008] In some implementations, the displayed content may be associated with a web page. The content data may include a uniform resource locator. The interface may include a web page viewer and a preview bubble. In some implementations, the web page viewer may provide a portion of the displayed content for display. The preview bubble may provide a snippet associated with the additional content. The interface may include a scroll indicator and a bubble interface element. In some implementations, the scroll indicator may indicate the position of the currently viewed portion of the displayed content relative to other portions of the displayed content. The bubble interface element may be provided adjacent to the scroll indicator in the interface. The additional content may include a purchase link. The purchase link may be associated with a product associated with the displayed content. In some implementations, the additional content may include an augmented reality experience. The interface may include selectable user interface elements for providing an augmented reality experience.

[0009] In some implementations, the operation may include providing a suggested interface element for display in a first state. The suggested interface element may describe whether the additional content has been determined. The operation may include providing a suggested interface element for display in a second state in response to determining the additional content associated with the displayed content. The second state may describe that the additional content is being determined. In some implementations, the operation may include obtaining input data. The input data may describe a selection of a suggested interface element of the interface. The operation may include providing a portion of the additional content for display.

[0010] In some implementations, determining additional content associated with the displayed content may include: determining a uniform resource locator associated with the displayed content; and determining an additional web page associated with the uniform resource locator. Determining additional content associated with the displayed content may also include generating additional content based on the additional web page. In some implementations, determining additional content associated with the displayed content may include: determining multiple additional resources associated with the displayed content; determining multiple predicted actions associated with one or more of the multiple additional resources; and generating multiple action interface elements. Multiple action interface elements may be associated with multiple predicted actions. Multiple action interface elements may be provided for display in an interface.

[0011] In some implementations, determining additional content associated with the displayed content may include: processing at least a portion of the displayed content with a machine learning model to determine a machine learning output; and determining the additional content based on the machine learning output. The interface may include a swipe-up interface element configured to display a portion of the additional content based on a user input.

[0012] In some implementations, providing an interface for viewing data associated with displayed content and additional content may include: providing at least a portion of the displayed content for display with a suggested interface element; obtaining a selection of the suggested interface element; and providing at least a portion of the additional content for display. The operations may include processing a portion of the displayed content to generate semantic data. The semantic data may describe a semantic understanding of the portion of the displayed content. The operations may query a database based at least in part on the semantic data. The additional content may be determined based on the query of the database.

[0013] In some implementations, the interface may include a type indicator associated with the content type of the additional content. The type indicator may describe the action type. The additional content may be associated with performing a specific action. In some implementations, the type indicator may describe the understanding type. The additional content may provide supplemental information for understanding a specific topic associated with the displayed content.

[0014] Another example aspect of the present disclosure relates to a computer-implemented method for providing additional content. The method may include obtaining content data by a computing system including one or more processors. The content data may include an indication of displayed content provided for display to a user. The method may include processing the content data with a machine learning model by the computing system to generate a machine learning model output. The machine learning output may describe a semantic understanding of the displayed content. The method may include determining, by the computing system, additional content associated with the displayed content based on the machine learning model output. In some implementations, the additional content may be obtained based on the content data. The additional content may be determined by processing the content data during the presentation of the displayed content. The method may include providing an interface for viewing data associated with the displayed content and the additional content by the computing system in response to determining the additional content associated with the displayed content. The interface may include a viewing window that displays at least a portion of the displayed content. In some implementations, the interface may include a suggestion notification describing the additional content.

[0015] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause one or more computing devices to perform operations. The operations may include obtaining content data. The content data may include an indication of displayed content provided for display to a user. The operations may include processing the content data to determine an entity associated with the displayed content. The operations may include determining additional content associated with the displayed content based on the entity. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during presentation of the displayed content. The operations may include providing an interface for viewing data associated with the displayed content and the additional content. The interface may include a viewing window that displays at least a portion of the displayed content. In some implementations, the interface may include a suggestion notification describing the additional content.

[0016] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0017] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0019] Figure 1 Depicted is a block diagram of an example additional content suggestion system according to an example embodiment of the present disclosure.

[0020] FIG. 2A to FIG. 2B Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0021] Figure 3 Depicted are diagrams of example suggestion interface elements according to example embodiments of the present disclosure.

[0022] Figure 4 Depicted is a diagram of an example scrolling interface according to an example embodiment of the present disclosure.

[0023] FIG. 5A to FIG. 5C Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0024] Figure 6 Depicted is a diagram of an example tray of actions interface according to an example embodiment of the present disclosure.

[0025] Figure 7 Depicted is a diagram of an example entry point element according to an example embodiment of the present disclosure.

[0026] Figure 8 Depicted is a diagram of an example preview bubble in accordance with an example embodiment of the present disclosure.

[0027] Fig. 9 Depicted is a diagram of an example type indicator according to an example embodiment of the present disclosure.

[0028] Fig.10 Depicted is a diagram of an example additional content window according to an example embodiment of the present disclosure.

[0029] Fig.11 Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0030] Fig.12 Depicted are diagrams of example suggested interface element transitions according to example embodiments of the present disclosure.

[0031] Fig.13 Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0032] Fig.14 Depicted are diagrams of example suggestion interface elements according to example embodiments of the present disclosure.

[0033] Fig.15 Depicted is a flow chart of an example method for performing additional content interface presentation according to an example embodiment of the present disclosure.

[0034] Fig.16 Depicted is a flow diagram of an example method for performing additional content determination according to an example embodiment of the present disclosure.

[0035] Fig.17 Depicted is a flow chart of an example method for performing entity-based additional content determination according to an example embodiment of the present disclosure.

[0036] Fig.18A Depicted is a block diagram of an example computing system that performs additional content interface presentation according to an example embodiment of the present disclosure.

[0037] Fig.18B A block diagram of an example computing device performing additional content interface presentation is depicted according to an example embodiment of the present disclosure.

[0038] Fig.18C A block diagram of an example computing device performing additional content interface presentation is depicted according to an example embodiment of the present disclosure.

[0039] Reference numerals that are repeated across multiple drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0040] In general, the present disclosure relates to systems and methods for providing an interface for accessing additional content associated with a displayed content item. In particular, the systems and methods disclosed herein can utilize additional content predictions to provide information associated with displayed content, which can provide supplementary information to more fully understand the subject and / or provide user interface elements to perform actions associated with displayed content. The systems and methods can utilize one or more search engines, one or more databases, one or more machine learning models, and / or one or more user interface elements. The systems and methods disclosed herein provide suggestions for actively determining other information and / or other actions that may be useful to users. For example, the systems and methods may include obtaining content data. The content data may include an indication of displayed content provided for display to a user. The systems and methods may include determining additional content associated with displayed content. Additional content may be obtained based on content data. The systems and methods may include providing an interface for viewing data associated with displayed content and additional content.

[0041] The system and method may include obtaining content data. The content data may include an indication of displayed content provided for display to a user. In some implementations, the displayed content may be associated with a web page. The content data may include a uniform resource locator. The displayed content may include a web page, a video, a book, and / or a mobile application. The content data may include a uniform resource locator, text data, image data, potential coding data, and / or other metadata associated with the displayed content. The displayed content may include a web page, a document, and / or other information provided for display on a computing device. Obtaining content data may include obtaining text data, image data, structural data, and / or potential coding data currently provided in a viewer and generating content data describing the obtained data. Alternatively and / or additionally, obtaining content data may include processing source code, obtaining database data associated with a uniform resource locator, and / or processing a complete web page to generate one or more embeddings.

[0042] The system and method may include determining additional content associated with displayed content. The additional content may be obtained based on content data. In some implementations, the additional content may include a purchase link. The purchase link may be associated with a product associated with the displayed content. The additional content may include an augmented reality experience. The additional content may be obtained from one or more databases and / or may be generated based on the displayed content and / or one or more other resources. The additional content determination may be performed automatically in the background without being prompted by the user. Alternatively and / or additionally, the user may select one or more user interface elements to request the additional content determination. In some implementations, the additional content determination may occur during the display of the displayed content.

[0043] In some implementations, determining additional content associated with the displayed content may include: determining a uniform resource locator associated with the displayed content; and determining an additional web page associated with the uniform resource locator. Additionally and / or alternatively, the additional content may be generated based on the additional web page. The additional web page may include a web page that references the displayed content and / or a web page associated with the uniform resource locator through a search engine and / or a knowledge graph. The additional web page may provide similar and / or contradictory information.

[0044] In some implementations, determining additional content associated with the displayed content may include: determining multiple additional resources associated with the displayed content; determining multiple predicted actions associated with one or more of the multiple additional resources; and generating multiple action interface elements. The multiple action interface elements may be associated with the multiple predicted actions. The multiple action interface elements may be provided for display in the interface.

[0045] Alternatively and / or additionally, determining additional content associated with the displayed content may include: processing at least a portion of the displayed content with a machine learning model to determine a machine learning output; and determining the additional content based on the machine learning output.

[0046] The system and method may include an interface for viewing data associated with displayed content and additional content. The interface may include a web viewer and a preview bubble. In some implementations, the web viewer may provide a portion of the displayed content for display. The preview bubble may provide a fragment associated with the additional content. In some implementations, the interface may include an upward swipe interface element, which is configured to display a portion of the additional content based on user input. The interface may include a type indicator associated with the content type of the additional content. For example, the type indicator may describe the action type, and the additional content may be associated with performing a specific action. Alternatively and / or additionally, the type indicator may describe the understanding type. The additional content may provide supplementary information for understanding a specific topic associated with the displayed content. The interface may include a selectable user interface element for providing an augmented reality experience associated with the topic of the displayed content.

[0047] In some implementations, the interface may include a scroll indicator and a bubble interface element. The scroll indicator may indicate the position of the currently viewed portion of the displayed content relative to other portions of the displayed content. Additionally and / or alternatively, the bubble interface element may be provided adjacent to the scroll indicator in the interface. The bubble interface element may move in the display as the scroll indicator moves. The bubble interface element may provide data associated with the determined additional content for display. In some implementations, the data provided for display in the bubble interface element may change as different additional content is determined. For example, the beginning portion of a web page may discuss a first topic, and an additional web page discussing the first topic in detail may be determined and provided as the suggested additional content. The user may scroll to the middle portion of the web page discussing the second topic, and may determine a second additional web page discussing the second topic in detail and provide it as the suggested additional content. Then, the user may scroll to the bottom portion of the web page providing an object sold at a set price. Then, the bubble interface element may provide options for tracking prices and / or suggest selling different web resources of the object at a lower price.

[0048] In some implementations, providing an interface for viewing data associated with displayed content and additional content may include: providing at least a portion of the displayed content for display with a suggested interface element; obtaining a selection of the suggested interface element; and providing at least a portion of the additional content for display.

[0049] Additionally and / or alternatively, the systems and methods may include providing a suggestion interface element for display in a first state. The suggestion interface element may describe whether additional content has been determined. In response to determining additional content associated with the displayed content, the systems and methods may provide a suggestion interface element for display in a second state. The second state may describe that additional content is being determined.

[0050] In some implementations, the systems and methods may include obtaining input data. The input data may describe a selection of a suggested interface element for an interface. The systems and methods may include providing a portion of the additional content for display based on the input data.

[0051] Alternatively and / or additionally, the systems and methods may include processing a portion of the displayed content to generate semantic data (e.g., using one or more machine learning models). The semantic data may describe a semantic understanding of the portion of the displayed content. The systems and methods may include querying a database based at least in part on the semantic data. In some implementations, additional content may be determined based on a query to a database.

[0052] The Internet can provide a large number of resources on various topics. A user may be viewing and / or reading information provided on a topic. Additional information on the topic may be relevant to the user. The relevant information may be unknown to the user and / or may be desired by the user; however, due to additional search barriers, the user may not obtain the information until later. The systems and methods disclosed herein can automatically process displayed content to determine relevant additional content that can be suggested to the user.

[0053] Additionally and / or alternatively, the information may be outdated and / or may not be the most reliable information. The systems and methods disclosed herein may determine entities associated with displayed content items (e.g., topics, authors, publishers, and / or knowledge fields associated with the topics of the displayed content), and may determine more recent and / or more reliable information about specific entities to suggest to the user.

[0054] Alternatively and / or additionally, the user may want to better understand and / or interact with the information; however, the user may traditionally be limited to manually performing additional searches and / or bookmarking web pages. The systems and methods disclosed herein may utilize one or more machine learning models to suggest a summary of the displayed content. In some implementations, the system and method may determine an action associated with the content type of the displayed content and may suggest the action to the user. For example, the displayed content may include an advertisement for a product or service. The system and method may determine the advertising content type and may suggest a price tracking feature that may recursively update the user with future price changes. In some implementations, the displayed content may include an event (e.g., a football game) and the event content type may be determined. The system and method may suggest tracking event updates (e.g., score updates). The action may include a summary action, a tracking action, a save action, and / or a related resource lookup action (e.g., in response to determining a movie review content type, the system and method may suggest a movie theater webpage for ticket booking and / or may suggest a web resource including actor and director information for the movie).

[0055] Articles and other content items may be long and / or may only discuss off-topic topics in passing. Length and / or lack of complete context may create additional barriers for readers, which traditionally may result in further searching and may be time consuming. The systems and methods disclosed herein can actively determine and suggest a summary of content. Additionally and / or alternatively, the systems and methods can actively determine relevant off-topic topics in the displayed content. The systems and methods can determine additional content associated with the off-topic topics, and can suggest additional content to the user.

[0056] In response to the information provided in the displayed content item, the user may desire additional information and / or attempt to perform one or more additional actions based on the information provided in the displayed content item. Obtaining additional information and / or performing additional actions may include searching for supplementary information, searching for a purchase portal for purchasing the product discussed in the displayed content item, and / or one or more other additional actions. Additional actions may be time consuming, and the user may be unsure how to perform such additional actions, which may cause further confusion. The systems and methods disclosed herein can automatically determine additional information and / or additional actions associated with the displayed content, and can suggest additional information and / or additional actions to the user.

[0057] The systems and methods of the present disclosure provide many technical effects and benefits. As an example, the systems and methods may provide an interface for providing additional content predictions. The additional content predictions may enable a user to perform one or more actions and / or obtain additional information about a topic. The additional content predictions may be provided in an interface that allows a user to view a portion of the additional content while still displaying a portion of the initial content item.

[0058] Another technical benefit of the systems and methods of the present disclosure is the ability to utilize one or more machine learning models to determine that a particular portion of displayed content describes a specific topic to determine multiple different additional content items to provide, where each respective additional content item can be associated with a respective portion of the displayed content item.

[0059] Another example of technical effects and benefits relates to improved computing efficiency and improved functionality of computing systems. For example, the systems and methods disclosed herein can utilize additional content prediction to proactively provide resources that a user may desire, which can save time and computing power to navigate to one or more additional web pages to find resources associated with the additional content.

[0060] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0061] Figure 1 Depicted is a block diagram of an example additional content suggestion system 10 according to an example embodiment of the present disclosure. Additional content suggestion system 10 may include obtaining content data associated with displayed content 12, determining additional content 14 associated with displayed content 12, and providing a suggestion interface 16 for display.

[0062] In particular, the displayed content 12 may include at least a portion of a web page and / or a portion of a document displayed in a user interface. The content data may include data describing the displayed content 12. The content data may include a uniform resource locator, a text embedding, an image embedding, a portion of source code, text data, potentially encoded data, and / or image data.

[0063] The content data may be processed to determine entities 20 associated with the displayed content 12. The determined entities 20 may then be utilized to determine additional content 14. For example, the determined entities 20 may be utilized to generate a search query that may be used to query a search engine and / or database to determine additional content associated with the determined entities 20.

[0064] Alternatively and / or additionally, the content data may be processed with one or more machine learning models 22 to generate a machine learning model output. The machine learning model output may be additional content 14 and / or may be used to determine the additional content 14. For example, the machine learning model 22 may be trained to summarize the content, and the additional content 14 may be a summary of the displayed content 12. Alternatively and / or additionally, the machine learning model 22 may be a semantic understanding model (e.g., a natural language processing model trained for semantic understanding) that may process the displayed content 12 to generate a semantic understanding output. The semantic understanding output may then be used to determine other web resources and / or other documents associated with the semantic understanding.

[0065] In some implementations, the displayed content 24 may be processed to determine one or more actions 24 associated with the displayed content 12. User interface elements for performing the one or more actions 24 may be provided as additional content 14. For example, the displayed content 12 may be determined to include content that may potentially change over time, and tracking actions may be provided to the user as an option. Alternatively and / or additionally, the displayed content 12 may be determined to include an object associated with an augmented reality experience (e.g., a live trial experience), and the augmented reality experience may be provided as an option.

[0066] The displayed content 12 and the suggested additional content 14 may be provided for display in a suggestion interface 16. The suggestion interface 16 may be provided for display on a mobile device 30, a desktop device, a smart wearable device, and / or via other display devices. The suggestion interface 16 may include a viewing window 32 for the displayed content 12 and a pop-up interface element 34 for the additional content 14. Alternatively and / or additionally, the additional content 14 may be provided for display in a dynamically moving bubble interface element that moves in unison with a scroll indicator.

[0067] FIG. 2A to FIG. 2B Depicted are diagrams of example interfaces according to example embodiments of the present disclosure. In particular, Figure 2A Depicted are suggestion interface elements in three different states. First state 202 may include a suggestion interface element provided without a color and / or without a badge, which may describe that additional content has not yet been determined. Second state 204 may include a suggestion interface element having a different color than first state 202, which may describe that additional content has been determined. Third state 206 may include a suggestion interface element of second state 206 with a badge attached, which may indicate that the determined additional content is provided with a high confidence level associated with the displayed content.

[0068] Figure 2BThe additional content data being provided in the interface may be depicted. At 208, a preview bubble is provided in the interface. The preview bubble may include a snippet associated with the determined additional content. The snippet may describe information provided by the additional content. The preview bubble may be provided in response to selection of a suggested interface element and / or may be provided automatically.

[0069] At 210, an expanded panel may be provided for display, which may include more information about additional content and / or auxiliary content associated with the displayed content. The interface depicted in 210 may be provided in response to selection of a suggestion interface element and / or preview bubble. The auxiliary content may include additional resources associated with the entity discussed in the displayed content.

[0070] Figure 3 An illustration of an example suggestion interface element according to an example embodiment of the present disclosure is depicted. In some implementations, the suggestion interface element may be different based on the determined information provided by the displayed content. For example, the suggestion interface element may include a selectable action element for performing one or more actions. At 302, a tracking price action element and a quick checkout element are provided for display in response to determining that the displayed content is associated with the product to be sold. The tracking price action can be used to set an application programming interface that can provide notifications to users when the price of the product changes. The quick checkout action element can be used to interact with a web platform to purchase products to be sold using stored user data. At 304, a music action element can be provided in response to determining that the displayed content discusses a music artist and / or album. The music action element can be used to play songs and / or playlists associated with the information provided by the displayed content.

[0071] Figure 4A diagram of an example scrolling interface according to an example embodiment of the present disclosure is depicted. In some implementations, an interface for providing additional content may include a scrolling interface. The scrolling interface may include a scrolling indicator 420, which may indicate the position of the currently viewed portion of the displayed content relative to the entire displayed content. The scrolling interface may additionally include a bubble interface element 430 that may be provided adjacent to the scrolling indicator 420. As the user navigates through the displayed content, the bubble interface element 430 may move in unison with the scrolling indicator 420. Additionally and / or alternatively, the fragment provided in the bubble interface element 430 may describe the additional content that may be viewed. The fragment may change as the user navigates through the displayed content. Additionally and / or alternatively, the additional content may change based on a specific portion of the currently displayed content. In some implementations, the scrolling interface may be indicated based on a tutorial interface element (e.g., as shown in 402). The additional content may then be determined based on the data provided in the viewing window, and the bubble interface element 430 may be provided for display (e.g., as shown in 404). As the user scrolls further down the page (eg, through displayed content), new additional content items may be determined, and the segment in bubble interface element 430 may change (eg, as shown in 406 ).

[0072] FIG. 5A to FIG. 5C Depicted are diagrams of example interfaces according to example embodiments of the present disclosure. In particular, FIG. 5A to FIG. 5C The interface includes a scrolling interface that dynamically changes a segment of the bubble interface element when the suggested additional content items change. The dynamic change can be based on changes in information provided as the user navigates through the displayed content. For example, in Figure 5A 504, the entity-specific information can be obtained to generate a first additional content item. The entity can be determined based on the information obtained in the first viewed portion 502 of the displayed content items. The bubble interface element can then be selected to open a first additional window 504, which displays the first additional content item, which can include a link to a mobile application, entity contact information, and a link to learn more about the entity.

[0073] exist Figure 5B , a second viewed portion 506 of the displayed content item can be displayed with the updated bubble interface element. The bubble interface element can be interacted with to open a second additional window 508, which can include a second additional content item generated based on the second viewed portion discussing the particular product. The second additional content item can include a link to open an augmented reality live try-on experience to view the product in the user's environment.

[0074] exist Figure 5CIn the embodiment of the present invention, the third viewed portion 510 of the displayed content item can be displayed together with the updated bubble interface element. The bubble interface element can be interacted with to open a third additional window 512, which can include a third additional content item generated based on the third viewed portion discussing the routine or process. The third additional content item can include one or more resources that explain how to perform the routine or process, and the one or more resources can include a video and / or a step-by-step list.

[0075] Figure 6 A diagram of an example action tray interface according to an example embodiment of the present disclosure is depicted. In some implementations, an interface for presenting additional content may include an action tray interface. The action tray interface may include one or more predicted actions determined based on displayed content and / or one or more predetermined actions that can be provided regardless of the information provided by the displayed content. For example, the first action tray interface 602, the second action tray interface 604, the third action tray interface 606, and the fourth action tray interface 608 may all include a bookmark action element to enable a user to bookmark and / or save displayed content; however, other action elements in the action tray interface may vary based on specific displayed content. In particular, in response to determining that the displayed content is associated with a product for purchase, the first action tray interface 602 includes a track price action element, a find similar action element, and a compare action element. Additionally and / or alternatively, in response to determining that the displayed content is associated with a media content item (e.g., a video), the second action tray interface includes a mention action element (e.g., viewing other resources that mention the specific displayed content), a comparison action element, and a clip action element (e.g., saving a portion of the specific displayed content). In response to determining that the displayed content is associated with a recipe, the third action tray interface 606 includes an ingredient action element (e.g., adding a recipe to a cookbook and / or obtaining and saving an ingredient list), a comparison action element, and a clip action element. In response to determining that the displayed content is associated with a product advertisement, the fourth action tray interface 608 includes a mention action element, a find similar action element, and a clip action element.

[0076] Figure 7A diagram of an example entry point element according to an example embodiment of the present disclosure is depicted. Different entry point elements can be utilized uniformly, can differ across platforms, can differ based on displayed content, and / or can differ based on user preferences. For example, the entry point element in 702 includes a multi-colored circular element with a flashing icon, while the entry point element in 704 dynamically changes, can be expanded, can include text, and can include multiple icons. The entry point element in 706 includes a modified entry point element, which includes an icon associated with a determined action associated with the determined additional content. Additionally and / or alternatively, the entry point element can differ in color and / or shape when the element is dormant (e.g., when no additional content is currently determined).

[0077] Figure 8 An illustration of an example preview bubble according to an example embodiment of the present disclosure is depicted. Additional content determined and / or generated based on displayed data may include price insights 802 (e.g., one or more shopping lists determined to be associated with the displayed content), summaries 804 (e.g., the displayed content may be processed by a machine learning model to generate a summary of the displayed content), augmented reality previews 806 (e.g., an augmented reality experience may be obtained based on the displayed content and provided to the user), ingredient extraction 808 (e.g., ingredients in a recipe may be extracted and saved in a user-specific database), and / or related readings 810 (e.g., supplementary resources associated with a topic in the displayed content may be determined and provided to the user). Each of the different additional content types may be determined and provided based on the displayed content, context, and / or one or more user preferences. A preview bubble with a fragment may then be provided to the user to provide a preview of the additional content obtained and / or generated. Preview bubbles and / or suggestion interface elements may be provided via a variety of different interface element shapes and sizes.

[0078] Fig. 9 An illustration of an example type indicator according to an example embodiment of the present disclosure is depicted. In particular, in some implementations, the preview bubble can include a type indicator that can indicate a type of action associated with the additional content and / or a level of importance associated with the additional content. For example, an active action with low to medium security concern can be associated with a first color indicator 902, while a predetermined question associated with high security concern can be associated with a second color indicator 904.

[0079] Fig.10An illustration of an example additional content window according to an example embodiment of the present disclosure is depicted. An additional content window may be provided for display in response to interaction with a suggestion interface element and / or a preview bubble. The additional content window may vary based on the additional content type. For example, in 1002, multiple price lists from different suppliers are provided based on the displayed content including products for sale, the multiple price lists having links to access web pages of different suppliers and a tracking price action slider. In 1004, a text summary may be provided in a text bubble based on the displayed content including an article. In 1006, multiple different tabs and multiple different search results may be provided based on the displayed content including a search results page.

[0080] Fig.11 Depicted are diagrams of example interfaces according to example embodiments of the present disclosure. In particular, Fig.11 The transition of the interface from suggestion interface element display 1102 to preview bubble display 1104 to additional content window display 1106 is depicted. Suggestion interface element display 1102 may include a displayed content window for displaying a portion of displayed content and a suggestion interface element that can be interacted with to provide additional content for display. Preview bubble display 1104 may include a displayed content window, suggestion interface elements, and preview bubbles, which may include a fragment providing a preview of the additional content. Additional content window display 1106 may be provided for display in response to one or more obtained inputs, and may include an expanded additional content window for viewing one or more additional content items.

[0081] Fig.12 An illustration of an example suggestion interface element transition according to an example embodiment of the present disclosure is depicted. In some implementations, the suggestion interface element can be expanded and collapsed. In an initial state 1202, the suggestion interface element can include a circular icon. In a secondary state 1204, the suggestion interface element can include an expanded pill with an icon and a text label.

[0082] Fig.13 A diagram of an example interface according to an example embodiment of the present disclosure is depicted. The interface may include an entry point state 1302, a nudge state 1304, and a panel state 1306. The entry point state 1302 may include a displayed content viewing window and a suggestion interface element for selection. The nudge state 1304 may include a displayed content viewing window, a suggestion interface element for selection, and a preview bubble, which provides a fragment indicating a possible action to be performed. The panel state may include an expanded panel for displaying additional content. The interface may transition from one state to another state based on one or more inputs and / or one or more determinations.

[0083] Fig.14 An illustration of an example suggestion interface element according to an example embodiment of the present disclosure is depicted. The suggestion interface element may include an icon that may be displayed in different colors and / or with different badges based on one or more determinations. For example, a first state 1402 may include an icon in gray to indicate that additional content has not yet been determined. A second state 1404 may include an icon in one or more other colors to indicate that an additional content item has been determined and can be provided. In some implementations, a badge 1406 may be provided in the second state based on a high confidence correlation level between the displayed content and the additional content.

[0084] Fig.15 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Fig.15 The steps are depicted as being performed in a particular order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of method 1500 may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0085] At 1502, a computing system may obtain content data. The content data may include an indication of displayed content provided for display to a user. In some implementations, the displayed content may be associated with a web page. The content data may include a uniform resource locator. The displayed content may include text data, image data, white space, structural data, and / or potentially encoded data. The displayed content may be provided for display via a browser application, a messaging application, a social media application, and / or via a widget. The content data may be obtained via an overlay application, a browser extension, a built-in feature of an application, and / or an operating system feature. The displayed content may be associated with a first web page. The first web page may be associated with a first web resource.

[0086] At 1504, the computing system may determine additional content associated with the displayed content. The additional content may be obtained based on the content data. The additional content may be determined by processing the content data during presentation of the displayed content. In some implementations, the additional content may include a purchase link. The purchase link may be associated with a product associated with the displayed content. The additional content may include an augmented reality experience. The additional content may be associated with a second web page. The second web page may be different from the first web page. Additionally and / or alternatively, the additional content may be associated with a second web resource that is different from the first web resource.

[0087] In some implementations, determining additional content associated with the displayed content may include: determining a uniform resource locator associated with the displayed content; and determining an additional web page associated with the uniform resource locator. Additionally and / or alternatively, the additional content may be generated based on the additional web page.

[0088] In some implementations, determining additional content associated with the displayed content may include: determining multiple additional resources associated with the displayed content; determining multiple predicted actions associated with one or more of the multiple additional resources; and generating multiple action interface elements. The multiple action interface elements may be associated with the multiple predicted actions. The multiple action interface elements may be provided for display in the interface.

[0089] Alternatively and / or additionally, determining additional content associated with the displayed content may include: processing at least a portion of the displayed content with a machine learning model to determine a machine learning output; and determining the additional content based on the machine learning output.

[0090] At 1506, the computing system may provide an interface for viewing data associated with the displayed content and the additional content. The interface may be provided in response to determining the additional content associated with the displayed content. The interface may include a web viewer and a preview bubble. In some implementations, the web viewer may provide a portion of the displayed content for display. The preview bubble may provide a fragment associated with the additional content. In some implementations, the interface may include an upward swipe interface element, which is configured to display a portion of the additional content based on user input. The interface may include a type indicator associated with the content type of the additional content. For example, the type indicator may describe the action type, and the additional content may be associated with performing a specific action. Alternatively and / or additionally, the type indicator may describe the understanding type. The additional content may provide supplementary information for understanding a specific topic associated with the displayed content. The interface may include a selectable user interface element for providing an augmented reality experience. In some implementations, the interface may include a suggestion state. The suggestion state may include a viewing window that displays at least a portion of the displayed content. Additionally and / or alternatively, the suggestion state may include a suggestion interface element indicating the determination of the additional content. The suggestion interface element may be selected, and an additional content preview window describing at least a portion of the additional content may be provided. In addition to the initially suggested additional content, the additional content preview window may also include one or more other additional content items.

[0091] In some implementations, the interface can include a scroll indicator and a bubble interface element. The scroll indicator can indicate the position of the currently viewed portion of the displayed content relative to other portions of the displayed content. Additionally and / or alternatively, the bubble interface element can be provided adjacent to the scroll indicator in the interface.

[0092] In some implementations, providing an interface for viewing data associated with displayed content and additional content may include: providing at least a portion of the displayed content for display with a suggested interface element; obtaining a selection of the suggested interface element; and providing at least a portion of the additional content for display.

[0093] Additionally and / or alternatively, the systems and methods may include providing a suggestion interface element for display in a first state. The suggestion interface element may describe whether additional content has been determined. In response to determining additional content associated with the displayed content, the systems and methods may provide a suggestion interface element for display in a second state. The second state may describe that additional content is being determined.

[0094] In some implementations, the systems and methods may include obtaining input data. The input data may describe a selection of a suggested interface element of an interface. The systems and methods may include providing a portion of the additional content for display.

[0095] Alternatively and / or additionally, the system and method may include processing a portion of the displayed content to generate semantic data. The semantic data may describe a semantic understanding of the portion of the displayed content. The system and method may include querying a database based at least in part on the semantic data. In some implementations, the additional content may be determined based on a query to a database.

[0096] Fig.16 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Fig.16 The steps are depicted in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement shown in particular. The various steps of method 1600 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0097] At 1602, the computing system may obtain content data. The content data may include an indication of displayed content provided for display to a user. The content data may include data describing the displayed content. The displayed content may include a web page and / or a document. The displayed content may be displayed in a browser application, a search application, and / or a dedicated application for a specific content type.

[0098] At 1604, the computing system may process the content data with a machine learning model to generate a machine learning model output. The machine learning output may describe a semantic understanding of the displayed content. The machine learning model may include a natural language processing model, a segmentation model, a classification model, a detection model, and / or an enhancement model. The machine learning model may include a convolutional neural network, a feedforward neural network, a transformer model, and / or a recurrent neural network. The machine learning model output may include embeddings, text data, image data, latent coding data, audio data, and / or code.

[0099] At 1606, the computing system may determine additional content associated with the displayed content based on the machine learning model output. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during the presentation of the displayed content. The additional content may include a summary. In some implementations, the additional content may include additional information and / or additional actions determined based on the machine learning model output. The machine learning model output may describe a semantic understanding of the displayed content, which may be used to determine additional content associated with the semantic understanding. In some implementations, the machine learning model output may include a topic determination, which may be used to determine additional content associated with the topic.

[0100] At 1608, the computing system may provide an interface for viewing data associated with the displayed content and the additional content. The interface may be provided in response to determining the additional content associated with the displayed content. In some implementations, the interface may include a viewing window that displays at least a portion of the displayed content. The interface may include a suggestion notification that describes the additional content.

[0101] Fig.17 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Fig.17 The steps are depicted in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement shown in particular. The various steps of method 1700 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0102] At 1702, the computing system may obtain content data. The content data may include an indication of displayed content provided for display to a user. The content data may include data describing the displayed content. The displayed content may include a portion of a web page, a portion of a document, and / or other information provided for display.

[0103] At 1704, the computing system may process the content data to determine entities associated with the displayed content. The entities may be determined based on content in the displayed content (e.g., based on a title, an image in the displayed content, and / or information described in a body paragraph), based on data associated with a uniform resource locator, and / or based on an index lookup.

[0104] At 1706, the computing system may determine additional content associated with the displayed content based on the entity. The additional content may be obtained based on the content data. In some implementations, the additional content may be determined by processing the content data during presentation of the displayed content. The additional content may be determined by generating a search query based on the entity, providing the search query to a search engine, and receiving one or more search results from the search engine.

[0105] At 1708, the computing system may provide an interface for viewing data associated with the displayed content and the additional content. The interface may include a viewing window that displays at least a portion of the displayed content. In some implementations, the interface may include a suggestion notification that describes the additional content.

[0106] Fig.18A A block diagram of an example computing system 100 that performs additional content interface presentation according to an example embodiment of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0107] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0108] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0109] In some implementations, the user computing device 102 may store or include one or more content prediction models 120. For example, the content prediction model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Fig.11 An example content prediction model 120 is discussed.

[0110] In some implementations, one or more content prediction models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single content prediction model 120 (e.g., to perform parallel additional content predictions across multiple instances of a displayed content item).

[0111] More particularly, the content prediction model 120 can be configured to process content data (e.g., uniform resource locators, text data, image data, potential coding data and / or other metadata) to determine additional content associated with the displayed content. The additional content can be determined by generating semantic data associated with the displayed content and querying a database based on the semantic data. Alternatively and / or additionally, the additional content can be determined by generating a search query based on the content data. In some implementations, the predicted action type can be determined, and the additional content can be determined based on the predicted action type.

[0112] Additionally or alternatively, one or more content prediction models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the content prediction model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a content prediction service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0113] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0114] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0115] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0116] As described above, the server computing system 130 may store or otherwise include one or more machine learning content prediction models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Fig.11 An example model 140 is discussed.

[0117] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0118] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158, which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0119] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, error back propagation. For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations.

[0120] In some implementations, performing error backpropagation may include performing truncated backpropagation through time. The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0121] In particular, model trainer 160 can train content prediction models 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, an example training data set, which can include training examples and reference real data. Training examples can include example content data (e.g., uniform resource locators, example text, example images, example latent coding data, and / or example embeddings). Reference real data can include reference real labels, reference real predictions, reference real action types, reference real queries, and / or reference real semantic data outputs.

[0122] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0123] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0124] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted via any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0125] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0126] In some implementations, the input of the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of image data, potential embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change in image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded representation and / or a compressed representation of image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0127] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an upgraded (upscale) text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0128] In some implementations, the input of the machine learning model of the present disclosure may be latent coded data (e.g., a latent space representation of an input, etc.). The machine learning model may process the latent coded data to generate an output. As an example, the machine learning model may process the latent coded data to generate a recognition output. As another example, the machine learning model may process the latent coded data to generate a reconstruction output. As another example, the machine learning model may process the latent coded data to generate a search output. As another example, the machine learning model may process the latent coded data to generate a re-clustering output. As another example, the machine learning model may process the latent coded data to generate a prediction output.

[0129] In some implementations, the input to the machine learning model of the present disclosure may be statistical data. The machine learning model may process the statistical data to generate an output. As an example, the machine learning model may process the statistical data to generate an identification output. As another example, the machine learning model may process the statistical data to generate a prediction output. As another example, the machine learning model may process the statistical data to generate a classification output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a visualization output. As another example, the machine learning model may process the statistical data to generate a diagnostic output.

[0130] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data of one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the possibility that one or more images depict an object belonging to an object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the possibility that that region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines the corresponding possibility of each category in a predetermined set of categories for each pixel in one or more images. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at the pixel between images in the network input for each pixel in one of the input images.

[0131] Fig.18A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally on the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0132] Fig.18B Depicted is a block diagram of an example computing device 40 performing in accordance with an example embodiment of the present disclosure. Computing device 40 may be a user computing device or a server computing device.

[0133] The computing device 40 includes a plurality of applications (e.g., Application 1 to Application N). Each application contains its own machine learning library and one or more machine learning models. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0134] like Fig.18BAs shown, each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0135] Fig.18C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0136] The computing device 50 includes a plurality of applications (e.g., Application 1 to Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0137] The central intelligence layer includes multiple machine learning models. For example, Fig.18C As shown, a corresponding machine learning model (e.g., model) can be provided for each application, and the machine learning model is managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0138] The central intelligence layer may communicate with the central device data layer. The central device data layer may be a centralized data repository for the computing device 50. Fig.18C As shown, the central device data layer can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, the central device data layer can use an API (e.g., a private API) to communicate with each device component.

[0139] The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0140] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, can easily produce changes, modifications, and equivalents of such embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, changes, and equivalents.

Claims

1. A computing system for content prediction, the computing system comprising: one or more processors; as well as one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining content data, wherein the content data includes an indication of displayed content provided for display to a user; determining additional content associated with the displayed content, wherein the additional content is obtained based on the content data, wherein the additional content is determined by processing the content data during presentation of the displayed content; and In response to determining additional content associated with the displayed content, an interface for viewing data associated with the displayed content and the additional content is provided, wherein the interface includes a suggestion state, wherein the suggestion state includes a viewing window that displays at least a portion of the displayed content, and wherein the suggestion state includes a suggestion interface element that indicates the determination of the additional content.

2. The computing system of claim 1, wherein the displayed content is associated with a web page, and wherein the content data comprises a uniform resource locator.

3. The computing system of any preceding claim, wherein the interface comprises a web page viewer and a preview bubble, wherein the web page viewer provides a portion of the displayed content for display, and wherein the preview bubble provides a snippet associated with the additional content.

4. A computing system as described in any preceding claim, wherein the interface includes a scroll indicator and a bubble interface element, wherein the scroll indicator indicates the position of a currently viewed portion of the displayed content relative to other portions of the displayed content, and wherein the bubble interface element is provided in the interface adjacent to the scroll indicator.

5. The computing system of any preceding claim, wherein the additional content includes a purchase link, wherein the purchase link is associated with a product associated with the displayed content.

6. The computing system of any preceding claim, wherein the additional content comprises an augmented reality experience, and wherein the interface comprises a selectable user interface element for providing the augmented reality experience.

7. The computing system of any preceding claim, wherein the operations further comprise: providing a suggestion interface element for display in the first state, wherein the suggestion interface element describes whether additional content has been determined; as well as In response to determining the additional content associated with the displayed content, the suggestion interface element is provided for display in a second state, wherein the second state describes that the additional content is being determined.

8. The computing system of any preceding claim, wherein the operations further comprise: obtaining input data, wherein the input data describes a selection of a suggested interface element of the interface; as well as A portion of the additional content is provided for display.

9. The computing system of any preceding claim, wherein determining the additional content associated with the displayed content comprises: determining a uniform resource locator associated with the displayed content; as well as An additional web page associated with the uniform resource locator is determined.

10. The computing system of any preceding claim, wherein determining the additional content associated with the displayed content further comprises: Additional content is generated based on the additional web page.

11. The computing system of any preceding claim, wherein determining the additional content associated with the displayed content comprises: determining a plurality of additional resources associated with the displayed content; determining a plurality of predicted actions associated with one or more resources of the plurality of additional resources; as well as generating a plurality of action interface elements, wherein the plurality of action interface elements are associated with the plurality of predicted actions; The plurality of action interface elements are provided for display in the interface.

12. The computing system of any preceding claim, wherein determining the additional content associated with the displayed content comprises: processing at least a portion of the displayed content with a machine learning model to determine a machine learning output; as well as The additional content is determined based on the machine learning output.

13. The computing system of any preceding claim, wherein the interface comprises a swipe up interface element configured to display a portion of the additional content based on user input.

14. The computing system of any preceding claim, wherein providing the interface for viewing data associated with the displayed content and the additional content comprises: providing at least a portion of the displayed content for display with a suggested interface element; Obtaining a selection of the suggested interface element; as well as At least a portion of the additional content is provided for display.

15. The computing system of any preceding claim, wherein the operations further comprise: processing a portion of the displayed content to generate semantic data, wherein the semantic data describes a semantic understanding of the portion of the displayed content; querying a database based at least in part on the semantic data; and Wherein the additional content is determined based on the querying of the database.

16. The computing system of any preceding claim, wherein the interface includes a type indicator associated with a content type of the additional content.

17. The computing system of claim 16, wherein the type indicator describes an action type, wherein the additional content is associated with performing a particular action.

18. The computing system of claim 16, wherein the type indicator describes a type of comprehension, wherein the additional content provides supplemental information for understanding a particular topic associated with the displayed content.

19. A computer-implemented method for providing additional content, the method comprising: obtaining, by a computing system including one or more processors, content data, wherein the content data includes an indication of displayed content provided for display to a user; processing, by the computing system, the content data with a machine learning model to generate a machine learning model output, wherein the machine learning model output describes a semantic understanding of the displayed content; determining, by the computing system based on the machine learning model output, additional content associated with the displayed content, wherein the additional content is obtained based on the content data, wherein the additional content is determined by processing the content data during presentation of the displayed content; and In response to determining additional content associated with the displayed content, the computing system provides an interface for viewing data associated with the displayed content and the additional content, wherein the interface includes a viewing window that displays at least a portion of the displayed content, and wherein the interface includes a suggested notification describing the additional content.

20. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: obtaining content data, wherein the content data includes an indication of displayed content provided for display to a user; processing the content data to determine an entity associated with the displayed content; determining, based on the entity, additional content associated with the displayed content, wherein the additional content is obtained based on the content data, wherein the additional content is determined by processing the content data during presentation of the displayed content; and An interface is provided for viewing data associated with the displayed content and the additional content, wherein the interface includes a viewing window that displays at least a portion of the displayed content, and wherein the interface includes a suggestion notification describing the additional content.

Images