Content browsing and display of related content during searches

Abstract

A system and method for presenting an interface for suggesting additional content can include obtaining data indicative of displayed content and determining additional content associated with the displayed content. An interface can then be provided that displays the data associated with the displayed content and the additional content. The interface can include a first display window for displaying a portion of the displayed content and a second display window for displaying a snippet associated with the additional content.

Description

Technical Field

[0001] Related Applications This application claims priority to U.S. Non-Provisional Patent Application No. 18 / 081,832, filed on December 15, 2022, and claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 410,433, filed on September 27, 2022. The applicant claims the priority and the benefit of each such application, and incorporates the entire content of all such applications herein by reference in their entirety.

[0002] The present disclosure generally relates to presenting additional content based on currently displayed content. More specifically, the present disclosure relates to obtaining data representing provided display content, determining additional content associated with the display content, and providing an interface with the data associated with the display content and the additional content.

Background Art

[0003] When viewing a content item such as a web page, a user may only read or view a small portion of the information provided regarding a topic. Further, the information may be old and / or not the most reliable. Alternatively and / or additionally, a user may desire to better understand the information and / or interact with the information. However, the user may be limited to manually performing additional searches and / or bookmarking web pages.

[0004] Articles and other content items may be long and / or may only briefly touch on peripheral topics. Their length and / or lack of complete context can be an additional obstacle for readers, potentially requiring additional searches that can be time-consuming.

Summary of the Invention

[0005] Aspects and advantages of the embodiments of this disclosure are partially shown in the following description, or can be learned from the following description, or can be learned through the practice of the embodiments.

[0006] One exemplary aspect of this disclosure relates to a computing system for content prediction. The computing system may include one or more processors and one or more non-temporary computer-readable media that, when executed by one or more processors, collectively store instructions causing the computing system to perform an action. The action may include retrieving content data. The content data may include instructions for display content to be provided for display to a user. The action may include determining additional content associated with the display content. The additional content may be retrieved based on the content data. In some embodiments, the additional content may be determined by processing the content data during the presentation of the display content. In response to determining additional content associated with the display content, the action may include providing an interface for viewing the display content and the data associated with the additional content. The interface may include a suggestion state. The suggestion state may include a display window that displays at least a portion of the display content. The suggestion state may include a suggestion interface element that indicates the determination of the additional content.

[0007] In some embodiments, the display content can be associated with a web page. The content data may include a uniform resource locator. The interface may include a web page viewer and a preview bubble. In some embodiments, the web page viewer may provide a portion of the display content for display. The preview bubble may provide a snippet associated with additional content. The interface may include a scroll indicator and a bubble interface element. In some embodiments, the scroll indicator may indicate the position of the currently displayed portion of the display content relative to other portions of the display content. The bubble interface element may be provided in the interface adjacent to the scroll indicator. The additional content may include a purchase link. The purchase link may be associated with a product associated with the display content. In some embodiments, the additional content may include an augmented reality experience. The interface may include selectable user interface elements for providing the augmented reality experience.

[0008] In some embodiments, the operation may include providing a suggestion interface element for display in a first state. The suggestion interface element may indicate whether additional content has been determined. In response to determining additional content associated with the display content, the operation may include providing a suggestion interface element for display in a second state. The second state may indicate the determined additional content. In some embodiments, the operation may include obtaining input data. The input data may indicate a selection of suggestion interface elements for the interface. The operation may include providing a portion of the additional content for display.

[0009] In some embodiments, determining additional content associated with the displayed content may include determining a uniform resource locator associated with the displayed content and determining additional web pages associated with the uniform resource locator. Determining additional content associated with the displayed content may further include generating additional content based on the additional web pages. In some embodiments, determining additional content associated with the displayed content may include determining a plurality of additional resources associated with the displayed content, determining a plurality of predictive actions associated with one or more of the plurality of additional resources, and generating a plurality of action interface elements. The plurality of action interface elements may be associated with a plurality of predictive actions. The plurality of action interface elements may be provided for display within the interface.

[0010] In some embodiments, determining additional content associated with the displayed content may include processing at least a portion of the displayed content with a machine learning model to determine a machine learning output, and then determining the additional content based on the machine learning output. The interface may include a swipe-up interface element configured to display a portion of the additional content based on user input.

[0011] In some embodiments, providing an interface for viewing data associated with display content and additional content may include providing proposed interface elements for display on at least a portion of the display content, obtaining a selection of the proposed interface elements, and providing at least a portion of the additional content for display. The operation may include processing a portion of the display content to generate semantic data. The semantic data may represent a semantic understanding of the portion of the display content. The operation may include querying a database based at least in part on the semantic data. The additional content may be determined based on the database query.

[0012] In some embodiments, the interface may include a type indicator associated with the content type of the additional content. The type indicator may indicate an action type. The additional content may be associated with performing a specific action. In some embodiments, the type indicator may indicate an understanding type. The additional content may provide supplementary information for understanding a specific topic associated with the displayed content.

[0013] Other exemplary embodiments of this disclosure relate to computer-implemented methods for providing additional content. The method may include a computing system comprising one or more processors acquiring content data. The content data may include instructions for display content to be presented to a user. The method may include the computing system processing the content data using a machine learning model to generate a machine learning model output. The machine learning output may demonstrate a semantic understanding of the display content. The method may include the computing system determining additional content associated with the display content based on the machine learning model output. In some embodiments, the additional content may be acquired based on content data. The additional content may be determined by processing the content data during the presentation of the display content. The method may include the computing system providing an interface for viewing the display content and the data associated with the additional content in response to determining the additional content associated with the display content. The interface may include a display window that displays at least a portion of the display content. In some embodiments, the interface may include a suggestion notice indicating the additional content.

[0014] Other exemplary embodiments of this disclosure relate to one or more non-temporary computer-readable media that, when executed by one or more computing devices, collectively store instructions causing one or more computing devices to perform an action. An action may include retrieving content data. The content data may include instructions for display content provided for display to a user. An action may include processing the content data to determine entities associated with the display content. An action may include determining additional content associated with the display content based on the entities. The additional content may be retrieved based on the content data. In some embodiments, the additional content may be determined by processing the content data during the presentation of the display content. An action may include providing an interface for viewing the data associated with the display content and the additional content. The interface may include a display window that displays at least a portion of the display content. In some embodiments, the interface may include a suggestion notice indicating the additional content.

[0015] Other aspects of this disclosure relate to various systems, apparatus, non-temporary computer-readable media, user interfaces, and electronic devices.

[0016] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and forming part of this specification illustrate exemplary embodiments of this disclosure and, together with the description, illustrate the relevant principles.

[0017] A detailed description of embodiments intended for those skilled in the art is given herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0018] [Figure 1] Shows a block diagram of an exemplary additional content proposal system according to an exemplary embodiment of the present disclosure. [Figure 2A] Shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 2B] Shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 3] Shows a diagram of an exemplary proposed interface element according to an exemplary embodiment of the present disclosure. [Figure 4] Shows a diagram of an exemplary scroll interface according to an exemplary embodiment of the present disclosure. [Figure 5A] Shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 5B] Shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 5C] Shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 6] Shows a diagram of an exemplary tray of an action interface according to an exemplary embodiment of the present disclosure. [Figure 7] Shows a diagram of an exemplary entry point element according to an exemplary embodiment of the present disclosure. [Figure 8] Shows a diagram of an exemplary previewable according to an exemplary embodiment of the present disclosure. [Figure 9] Shows a diagram of an exemplary type indicator according to an exemplary embodiment of the present disclosure. [Figure 10] Shows a diagram of an exemplary additional content window according to an exemplary embodiment of the present disclosure. [Figure 11] Shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 12] Shows a diagram of an exemplary proposed interface element transition according to an exemplary embodiment of the present disclosure. [Figure 13] Shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. [Figure 14] FIG. 1 shows an exemplary proposed interface element according to an exemplary embodiment of the present disclosure. [Figure 15] FIG. 4 shows a flowchart diagram of an exemplary method for performing additional content interface presentation according to an exemplary embodiment of the present disclosure. [Figure 16] FIG. 7 shows a flowchart diagram of an exemplary method for making additional content determination according to an exemplary embodiment of the present disclosure. [Figure 17] FIG. 10 shows a flowchart diagram of an exemplary method for making entity-based additional content determination according to an exemplary embodiment of the present disclosure. [Figure 18A] FIG. 13 shows a block diagram of an exemplary computing system for performing additional content interface presentation according to an exemplary embodiment of the present disclosure. [Figure 18B] FIG. 16 shows a block diagram of an exemplary computing device for performing additional content interface presentation according to an exemplary embodiment of the present disclosure. [Figure 18C] FIG. 19 shows a block diagram of an exemplary computing device for performing additional content interface presentation according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0019] Reference numbers repeated across multiple drawings are intended to identify the same features in various embodiments.

[0020] Generally, this disclosure covers systems and methods for providing interfaces for accessing additional content associated with display content items. In particular, the systems and methods disclosed herein may leverage additional content predictions to provide information associated with the display content, which may provide supplementary information for a more comprehensive understanding of the topic and / or provide user interface elements for performing actions associated with the display content. The systems and methods may utilize one or more search engines, one or more databases, one or more machine learning models, and / or one or more user interface elements. The systems and methods disclosed herein provide other information that may be useful to the user and / or suggestions for proactively deciding on other actions. For example, the systems and methods may include retrieving content data. The content data may include instructions for the display content to be displayed to the user. The systems and methods may include determining additional content associated with the display content. The additional content may be retrieved based on the content data. The systems and methods may include providing interfaces for viewing data associated with the display content and the additional content.

[0021] The system and method may include retrieving content data. The content data may include instructions for display content provided for display to a user. In some embodiments, the display content may be associated with a web page. The content data may include a uniform resource locator. The display content may include a web page, video, book, and / or mobile application. The content data may include a uniform resource locator, text data, image data, latent encoded data, and / or other metadata associated with the display content. The display content may include a web page, document, and / or other information provided for display on a computing device. Retrieving content data may include retrieving text data, image data, structural data, and / or latent encoded data currently provided to the viewer, and generating content data that indicates the retrieved data. Alternatively and / or additionally, retrieving content data may include processing source code, retrieving database data associated with a uniform resource locator, and / or processing a complete web page to generate one or more embeddings.

[0022] The system and method may include determining additional content associated with the displayed content. The additional content may be retrieved based on content data. In some embodiments, the additional content may include purchase links. The purchase links may be associated with products associated with the displayed content. The additional content may include augmented reality experiences. The additional content may be retrieved from one or more databases and / or generated based on the displayed content and / or one or more other resources. The additional content determination may be performed automatically in the background without prompting from the user. Alternatively and / or additionally, the user may select one or more user interface elements to request an additional content determination. In some embodiments, the additional content determination may be performed while the displayed content is being displayed.

[0023] In some embodiments, determining additional content associated with the displayed content may include determining the uniform resource locator associated with the displayed content and determining additional web pages associated with the uniform resource locator. Additionally and / or alternatively, the additional content may be generated based on the additional web pages. The additional web pages may include web pages that reference the displayed content, as well as web pages associated with the uniform resource locator by search engines and / or knowledge graphs. The additional web pages may provide similar and / or contradictory information.

[0024] In some embodiments, determining additional content associated with the display content may include determining a number of additional resources associated with the display content, determining a number of predictive actions associated with one or more of the additional resources, and generating a number of action interface elements. The number of action interface elements may be associated with a number of predictive actions. The number of action interface elements may be provided for display within the interface.

[0025] Alternatively and / or additionally, determining additional content associated with the displayed content may include processing at least a portion of the displayed content with a machine learning model to determine a machine learning output, and determining additional content based on the machine learning output.

[0026] The system and method may include providing an interface for viewing data associated with display content and additional content. The interface may include a web page viewer and a preview bubble. In some embodiments, the web page viewer may provide a portion of the display content for viewing. The preview bubble may provide a snippet associated with the additional content. In some embodiments, the interface may include a swipe-up interface element configured to display a portion of the additional content based on user input. The interface may include a type indicator associated with the content type of the additional content. For example, the type indicator may indicate an action type, and the additional content may be associated with performing a particular action. Alternatively and / or additionally, the type indicator may indicate an understanding type. The additional content may provide supplementary information for understanding a particular topic associated with the display content. The interface may include selectable user interface elements to provide an augmented reality experience associated with the topic of the display content.

[0027] In some embodiments, the interface may include a scroll indicator and a bubble interface element. The scroll indicator can indicate the position of the currently displayed portion of the display content relative to other portions of the display content. Additionally and / or alternatively, a bubble interface element may be provided in the interface adjacent to the scroll indicator. The bubble interface element may move within the display in conjunction with the movement of the scroll indicator. The bubble interface element may provide data for display associated with the determined additional content. In some embodiments, the data provided for display in the bubble interface element may change as different additional content is determined. For example, the beginning of a web page may discuss a first topic, and an additional web page discussing the first topic in detail may be determined and provided as proposed additional content. The user may scroll to the middle of a web page discussing a second topic, and a second additional web page discussing the second topic in detail may be determined and provided as proposed additional content. The user may then scroll to the bottom of a web page offering objects sold at a set price. Next, the bubble interface element can offer an option to track prices and / or suggest other web resources that have objects being sold at a lower cost.

[0028] In some embodiments, providing an interface for viewing data associated with display content and additional content may include providing suggested interface elements to at least a portion of the display content for display, obtaining a selection of suggested interface elements, and providing at least a portion of the additional content for display.

[0029] Additionally and / or alternatively, the system and method may include providing a suggestion interface element for display in a first state. The suggestion interface element may indicate whether additional content has been determined. In response to determining additional content associated with the display content, the system and method may provide a suggestion interface element for display in a second state. The second state may indicate the determined additional content.

[0030] In some embodiments, the system and method may include acquiring input data. The input data may indicate a selection of proposed interface elements of the interface. The system and method may include providing a portion of additional content for display based on the input data.

[0031] Alternatively and / or additionally, the system and method may include processing a portion of the display content (e.g., using one or more machine-learned models) to generate semantic data. The semantic data may represent a semantic understanding of the portion of the display content. The system and method may include querying a database based at least in part on the semantic data. In some embodiments, additional content may be determined based on the database query.

[0032] The internet can provide a wealth of resources on a variety of topics. Users may browse and / or read the information provided on a topic. Additional information on a topic may be relevant to the user. This relevant information may be unknown to the user and / or desired by the user. However, users cannot immediately obtain this information because it would require additional searching. The systems and methods disclosed herein can automatically process displayed content to determine relevant additional content that may be suggested to the user.

[0033] Additionally and / or alternatively, information may be outdated and / or not the most reliable information. The systems and methods disclosed herein can determine entities associated with a displayed content item (e.g., topics, authors, publishers, and / or fields of knowledge associated with the topic of the displayed content) and can determine more current and / or more reliable information about a particular entity suggested to the user.

[0034] Alternatively and / or additionally, users may desire to better understand and / or interact with information. However, traditionally, users may be limited to manually performing additional searches and / or bookmarking web pages. The systems and methods disclosed herein can leverage one or more machine learning models to suggest summaries of displayed content. In some embodiments, the systems and methods can determine actions associated with the content type of the displayed content, and these actions can be suggested to the user. For example, the displayed content may include advertisements for products or services. The systems and methods can determine the advertisement content type and suggest a price tracking function that can continuously inform the user about future price changes. In some embodiments, the displayed content may include events (e.g., football matches), and an event content type can be determined. The systems and methods may suggest tracking event updates (e.g., score updates). Actions may include summarization actions, tracking actions, save actions, and / or related resource search actions (for example, in response to determining a movie review content type, the systems and methods may suggest a cinema webpage when booking tickets and / or suggest web resources containing information about the movie's cast and director).

[0035] Articles and other content items may be lengthy and / or may only briefly touch upon related topics. Their length and / or lack of complete context may pose an additional obstacle for the reader, requiring further searching, which may be time-consuming. The systems and methods disclosed herein can proactively determine and suggest summaries of content. Additionally and / or alternatively, the systems and methods can proactively determine unrelated topics relevant to the displayed content. The systems and methods can determine and suggest additional content associated with unrelated topics.

[0036] In response to the information provided in a displayed content item, a user may desire additional information and / or attempt to take one or more additional actions based on the information provided in the displayed content item. Obtaining additional information and / or taking additional actions may include searching for supplementary information, searching for a purchase portal to buy a product discussed in the displayed content item, and / or one or more other additional actions. Additional actions may be time-consuming, and users may not know how to perform such additional actions, potentially causing further confusion. Systems and methods disclosed herein can automatically determine additional information and / or additional actions associated with the displayed content and can suggest additional information and / or additional actions to the user.

[0037] The systems and methods of this disclosure offer several technical effects and benefits. As an example, the systems and methods may provide an interface for providing additional content predictions. Additional content predictions may enable a user to perform one or more actions and / or obtain additional information about a topic. Additional content predictions may be provided in an interface that allows a user to view some of the additional content while still viewing some of the initial content items.

[0038] Another technical advantage of the system and method of this disclosure is that it can leverage one or more machine learning models to determine that a particular portion of the displayed content represents a particular topic, thereby determining and providing several different additional content items. Each additional content item may be associated with a corresponding portion of the displayed content item.

[0039] Other examples of technical effects and benefits relate to improvements in computational efficiency and the functionality of computing systems. For example, the systems and methods disclosed herein can leverage additional content prediction to proactively provide users with resources they may desire. This saves time and computational power compared to navigating to one or more additional web pages to find resources associated with additional content.

[0040] The exemplary embodiments of this disclosure will now be discussed in more detail with reference to the drawings.

[0041] Figure 1 shows a block diagram of an exemplary additional content suggestion system 10 according to an exemplary embodiment of the present disclosure. The additional content suggestion system 10 may include retrieving content data associated with display content 12, determining additional content 14 associated with display content 12, and providing a suggestion interface 16 for display.

[0042] In particular, the display content 12 may include at least a portion of a web page and / or a portion of a document displayed in the user interface. The content data may include data that represents the display content 12. The content data may include a uniform resource locator, text embedding, image embedding, a portion of source code, text data, latent encoded data, and / or image data.

[0043] Content data can be processed to determine the entity 20 associated with the displayed content 12. The determined entity 20 can then be used to determine additional content 14. For example, the determined entity 20 can be used to generate a search query, which can then be used to query a search engine and / or database to determine the additional content associated with the determined entity 20.

[0044] Alternatively and / or additionally, content data may be processed by one or more machine learning models 22 to generate machine learning model outputs. The machine learning model outputs may be additional content 14 and / or may be used to determine the additional content 14. For example, the machine learning model 22 may be trained to summarize content, and the additional content 14 may be a summary of the display content 12. Alternatively and / or additionally, the machine learning model 22 may be a semantic understanding model (e.g., a natural language processing model trained for semantic understanding) that can process the display content 12 to generate semantic understanding outputs. The semantic understanding outputs may then be used to determine other web resources and / or other documents associated with the semantic understanding.

[0045] In some embodiments, the display content 24 may be processed to determine one or more actions 24 associated with the display content 12. User interface elements for performing one or more actions 24 may be provided as additional content 14. For example, the display content 12 may be determined to include content that may potentially change over time, and tracking actions may be provided to the user as an option. Alternatively and / or additionally, the display content 12 may be determined to include objects associated with an augmented reality experience (e.g., a live wearable experience), and the augmented reality experience may be provided as an option.

[0046] The display content 12 and the proposed additional content 14 may be provided for display within the proposed interface 16. The proposed interface 16 may be provided for display via a mobile device 30, a desktop device, a smart wearable, and / or other display device. The proposed interface 16 may include a display window 32 for the display content 12 and a popup interface element 34 for the additional content 14. Alternatively and / or additionally, the additional content 14 may be provided for display within a dynamically moving bubble interface element that moves in conjunction with a scroll indicator.

[0047] Figures 2A and 2B illustrate exemplary search interfaces according to exemplary embodiments of the present disclosure. In particular, Figure 2A shows suggestion interface elements in three different states. A first state 202 may include suggestion interface elements provided without color and / or badge, which may indicate additional content that has not yet been determined. A second state 204 may include suggestion interface elements with different colors than those in the first state 202, which may indicate determined additional content. A third state 206 may include the suggestion interface elements of the second state 206 with the addition of a badge, which may indicate that the determined additional content has been determined to have a high relevance to the displayed content.

[0048] Figure 2B can illustrate additional content data provided within the interface. In 208, a preview bubble is provided within the interface. The preview bubble may contain a snippet associated with the selected additional content. The snippet may show the information provided by the additional content. The preview bubble may be provided in response to the selection of a suggested interface element and / or automatically.

[0049] In 210, an expanded panel may be provided for display, which may contain more information about additional content and / or auxiliary content associated with the displayed content. The interface shown in 210 may be provided in response to the selection of proposed interface elements and / or preview bubbles. The auxiliary content may include additional resources associated with entities discussed in the displayed content.

[0050] Figure 3 shows a diagram of exemplary suggestion interface elements according to exemplary embodiments of the present disclosure. In some embodiments, the suggestion interface elements may differ based on the information determined by the display content. For example, the suggestion interface elements may include selectable action elements for performing one or more actions. In 302, in response to the determination that the display content is associated with a product being sold, a track price action element and a fast checkout element are provided for the display. The track price action may be used to set up an application programming interface that can provide the user with a notification when the price of the product changes. The fast checkout action element may be used to interface with a web platform to purchase the product being sold using stored user data. In 304, in response to the determination that the display content is discussing a music artist and / or album, a music action element may be provided. The music action element may be used to play songs and / or playlists associated with the information provided by the display content.

[0051] Figure 4 shows a diagram of an exemplary scroll interface according to an exemplary embodiment of the present disclosure. In some embodiments, the interface for providing additional content may include a scroll interface. The scroll interface may include a scroll indicator 420 that can indicate the position of the currently displayed portion of the display content relative to the entire display content. The scroll interface may further include a bubble interface element 430 that may be provided adjacent to the scroll indicator 420. The bubble interface element 430 may move in conjunction with the scroll indicator 420 as the user navigates through the display content. Additionally and / or alternatively, snippets provided to the bubble interface element 430 may indicate additional content that can be viewed. The snippets may change as the user navigates through the display content. Additionally and / or alternatively, the additional content may change based on a particular portion of the currently displayed display content. In some embodiments, the scroll interface may be based on a tutorial interface element (e.g., as shown in 402). Additional content may then be determined based on data provided to the view window and provided for display by the bubble interface element 430 (e.g., as shown in 404). As the user scrolls further down the page (e.g., the displayed content), a new additional content item can be determined, and the snippet of the bubble interface element 430 can change (e.g., as shown in 406).

[0052] Figures 5A–5C illustrate exemplary interfaces according to exemplary embodiments of the present disclosure. In particular, the interfaces of Figures 5A–5C include a scrollable interface that dynamically changes the snippet of a bubble interface element as the proposed additional content item changes. The dynamic change may be based on changes in the information provided as the user navigates through the displayed content. For example, in Figure 5A, entity-specific information may be retrieved to generate a first additional content item. The entity may be determined based on the information retrieved in the first viewing portion 502 of the displayed content item. The bubble interface element is then selectable, thereby opening a first additional window 504 displaying a first additional content item which may include a link to a mobile application, contact information for the entity, and a link to more detailed information about the entity.

[0053] In Figure 5B, the second viewing portion 506 of the displayed content item can be displayed using an updated bubble interface element. The bubble interface element can be interacted with to open a second additional window 508, which may contain a second additional content item generated based on the second viewing portion describing a specific product. The second additional content item may contain a link to open a live try-on experience using augmented reality, allowing the user to view the product within their environment.

[0054] In Figure 5C, the third viewing portion 510 of the displayed content item can be displayed in an updated bubble interface element. The bubble interface element can be interacted with to open a third additional window 512, which may contain a third additional content item generated based on the third viewing portion describing the routine or process. The third additional content item may contain one or more resources that show how to perform the routine or process, which may include a video and / or a step-by-step list.

[0055] Figure 6 shows an exemplary tray of an interface according to an exemplary embodiment of the present disclosure. In some embodiments, the interface for presenting additional content may include an action interface tray. The action interface tray may include one or more predictive actions determined based on the displayed content, and / or one or more predetermined actions that may be provided regardless of the information provided by the displayed content. For example, the first tray of action interface 602, the second tray of action interface 604, the third tray of action interface 606, and the fourth tray of action interface 608 may all include a bookmark action element that allows the user to bookmark and / or save the displayed content. However, other action interface trays may vary based on specific displayed content. In particular, the first tray of action interface 602 may include a price tracking action element, a similar search action element, and a compare action element in response to a determination that the displayed content is associated with a product for purchase. Additionally and / or alternatively, the second tray of the action interface includes mention action elements (e.g., to view other resources that mention the particular displayed content), compare action elements, and clip action elements (e.g., to save a portion of the particular displayed content) in response to the determination that the displayed content is associated with a media content item (e.g., a video). The third tray 606 of the action interface includes ingredient action elements (e.g., to add a recipe to a cookbook and / or to retrieve and save an ingredient list), compare action elements, and clip action elements in response to the determination that the displayed content is associated with a recipe. The fourth tray 608 of the action interface includes mention action elements, similar search action elements, and clip action elements in response to the determination that the displayed content is associated with a product advertisement.

[0056] Figure 7 shows an exemplary entry point element according to an exemplary embodiment of the present disclosure. Different entry point elements may be uniformly used, may differ across platforms, may differ based on the displayed content, and / or based on user preference. For example, entry point element 702 includes a polycolored circular element with a sparkle icon, entry point element 704 may be dynamically changing and expanding, may contain text, and may contain multiple icons, entry point element 706 includes a modified entry point element that includes an icon associated with a determined action associated with determined additional content. Additionally and / or alternatively, entry point elements may differ in color and / or shape when the element is in a dormant state (e.g., when no additional content has been determined at the moment).

[0057] Figure 8 shows an exemplary preview bubble according to an exemplary embodiment of the present disclosure. Additional content determined and / or generated based on the displayed data may include price insights 802 (e.g., a purchase list of one or more products determined to be associated with the displayed content), summaries 804 (e.g., the displayed content can be processed with a machine learning model to generate a summary of the displayed content), augmented reality previews 806 (e.g., an augmented reality experience can be acquired and provided to the user based on the displayed content), ingredient extracts 808 (e.g., ingredients in a recipe can be extracted and stored in a user-specific database), and / or related readings 810 (e.g., supplemental resources associated with topics in the displayed content can be determined and provided to the user). Each of the different additional content types may be determined and provided based on the displayed content, context, and / or the preferences of one or more users. A preview bubble containing snippets may then be provided to the user to provide a preview of the acquired and / or generated additional content. The preview bubble and / or suggested interface elements may be provided through the shapes and sizes of multiple different interface elements.

[0058] Figure 9 shows an exemplary type indicator according to an exemplary embodiment of the present disclosure. In particular, in some embodiments, the preview bubble may include a type indicator that can indicate the type of action associated with additional content and / or the level of importance associated with the additional content. For example, an active action with low to moderate security concerns may be associated with a first color indicator 902, and a given issue with high security concerns may be associated with a second color indicator 904.

[0059] Figure 10 shows an exemplary additional content window according to an exemplary embodiment of the present disclosure. In response to interaction with proposed interface elements and / or preview bubbles, the additional content window may be provided for display. The additional content window may differ based on the type of additional content. For example, 1002 provides a list of multiple prices from different vendors based on display content including products for sale, with links to each vendor's webpage and a track price action slider. 1004 provides a text summary in a text bubble based on display content including articles. 1006 provides multiple different tabs and multiple different search results based on display content including search results pages.

[0060] Figure 11 shows an exemplary interface diagram according to an exemplary embodiment of the present disclosure. In particular, Figure 11 shows the interface transition from a proposed interface element display 1102 to a preview bubble display 1104 and an additional content window display 1106. The proposed interface element display 1102 may include a display content window for displaying a portion of the display content and a proposed interface element that can be interacted with to provide additional content for display. The preview bubble display 1104 may include a preview bubble that can include a display content window, a proposed interface element, and a snippet that provides a preview of the additional content. The additional content window display 1106 may be provided to be displayed in response to one or more acquired inputs and may include an expanded additional content window for viewing one or more additional content items.

[0061] Figure 12 shows a diagram illustrating the transition of an exemplary proposed interface element according to an exemplary embodiment of the present disclosure. In some embodiments, the proposed interface element can be expanded or collapsed. In the initial state 1202, the proposed interface element may include a round icon. In the secondary state 1204, the proposed interface element may include an expanded pill-shaped form with an icon and a text label.

[0062] Figure 13 shows a diagram of an exemplary interface according to an exemplary embodiment of the present disclosure. The interface may include an entry point state 1302, a nudge state 1304, and a panel state 1306. The entry point state 1302 may include a display content display window and suggestion interface elements for selection. The nudge state 1304 may include a display content display window, suggestion interface elements for selection, and a preview bubble that provides a snippet indicating possible actions to be taken. The panel state may include an expanded panel for displaying additional content. The interface may transition from one state to another based on one or more inputs and / or one or more decisions.

[0063] Figure 14 shows a diagram of exemplary proposed interface elements according to exemplary embodiments of the present disclosure. Proposed interface elements may include icons that may be displayed in different colors and / or with different badges based on one or more decisions. For example, a first state 1402 may include a gray icon to indicate that additional content has not yet been decided. A second state 1404 may include icons of one or more other colors to indicate that additional content items have been decided and can be provided. In some embodiments, a badge 1406 may be provided in the second state based on the determination that there is a high correlation between the displayed content and the additional content.

[0064] Figure 15 shows a flowchart of an exemplary method performed according to an exemplary embodiment of the present disclosure. While Figure 15 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or arrangement shown. Various steps of Method 1500 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0065] In 1502, the computing system can acquire content data. The content data may include instructions for display content to be provided for display to the user. In some embodiments, the display content may be associated with a web page. The content data may include a uniform resource locator. The display content may include text data, image data, white space, structural data, and / or latent encoded data. The display content may be provided for display via a browser application, messaging application, social media application, and / or widget. The content data may be acquired via an overlay application, browser extension, built-in application functionality, and / or operating system functionality. The display content may be associated with a first web page. The first web page may be associated with a first web resource.

[0066] In 1504, the computing system can determine additional content associated with the displayed content. The additional content can be obtained based on content data. The additional content can be determined by processing content data during the presentation of the displayed content. In some embodiments, the additional content may include a purchase link. The purchase link may be associated with a product associated with the displayed content. The additional content may include an augmented reality experience. The additional content may be associated with a second webpage. The second webpage may be different from the first webpage. Additionally and / or alternatively, the additional content may be associated with a second web resource that is different from the first web resource.

[0067] In some embodiments, determining additional content associated with the displayed content may include determining the uniform resource locator associated with the displayed content and determining additional web pages associated with the uniform resource locator. Additionally and / or alternatively, the additional content may be generated based on the additional web pages.

[0068] In some embodiments, determining additional content associated with the display content may include determining a number of additional resources associated with the display content, determining a number of predictive actions associated with one or more of the additional resources, and generating a number of action interface elements. The number of action interface elements may be associated with a number of predictive actions. The number of action interface elements may be provided for display within the interface.

[0069] Alternatively and / or additionally, determining additional content associated with the displayed content may include processing at least a portion of the displayed content with a machine learning model to determine a machine learning output, and determining additional content based on the machine learning output.

[0070] In 1506, the computing system may provide an interface for viewing data associated with display content and additional content. The interface may be provided in response to a decision on additional content associated with the display content. The interface may include a web page viewer and a preview bubble. In some embodiments, the web page viewer may provide a portion of the display content for viewing. The preview bubble may provide a snippet associated with the additional content. In some embodiments, the interface may include a swipe-up interface element configured to display a portion of the additional content based on user input. The interface may include a type indicator associated with the content type of the additional content. For example, the type indicator may indicate an action type, and the additional content may be associated with performing a particular action. Alternatively and / or additionally, the type indicator may indicate an understanding type. The additional content may provide supplementary information for understanding a particular topic associated with the display content. The interface may include selectable user interface elements for providing an augmented reality experience. In some embodiments, the interface may include a suggestion state. The suggestion state may include a display window that displays at least a portion of the display content. Additionally and / or alternatively, the suggestion state may include a suggestion interface element indicating a decision on additional content. The suggested interface elements can be selected, and an additional content preview window showing at least a portion of the additional content can be provided. The additional content preview window may include one or more other additional content items in addition to the initially suggested additional content.

[0071] In some embodiments, the interface may include a scroll indicator and a bubble interface element. The scroll indicator can indicate the position of the currently displayed portion of the display content relative to other portions of the display content. Additionally and / or alternatively, a bubble interface element may be provided in the interface adjacent to the scroll indicator.

[0072] In some embodiments, providing an interface for viewing data associated with display content and additional content may include providing suggested interface elements to at least a portion of the display content for display, obtaining a selection of suggested interface elements, and providing at least a portion of the additional content for display.

[0073] Additionally and / or alternatively, the system and method may include providing a suggestion interface element for display in a first state. The suggestion interface element may indicate whether additional content has been determined. In response to determining additional content associated with the display content, the system and method may provide a suggestion interface element for display in a second state. The second state may indicate the determined additional content.

[0074] In some embodiments, the system and method may include acquiring input data. The input data may indicate a selection of proposed interface elements for the interface. The system and method may also include providing some additional content for display.

[0075] Alternatively and / or additionally, the system and method may include processing a portion of the display content to generate semantic data. The semantic data may represent a semantic understanding of the portion of the display content. The system and method may include querying a database based at least in part on the semantic data. In some embodiments, additional content may be determined based on the database query.

[0076] Figure 16 shows a flowchart of an exemplary method for carrying out an exemplary embodiment of the present disclosure. While Figure 16 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or arrangement shown. Various steps of Method 1600 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0077] In 1602, a computing system can acquire content data. Content data may include instructions for display content provided to a user. Content data may include data indicating the display content. Display content may include web pages and / or documents. Display content may be displayed within a browser application, a search application, and / or a dedicated application for a particular content type.

[0078] In version 1604, a computing system can process content data using a machine learning model to generate a machine learning model output. The machine learning output can demonstrate a semantic understanding of the displayed content. The machine learning model can include natural language processing models, segmentation models, classification models, detection models, and / or augmentation models. The machine learning model can include convolutional neural networks, feedforward neural networks, transformer models, and / or recurrent neural networks. The machine learning model output can include embeddings, text data, image data, latent encoded data, audio data, and / or code.

[0079] In 1606, the computing system can determine additional content associated with the displayed content based on the output of a machine learning model. The additional content can be obtained based on the content data. In some embodiments, the additional content can be determined by processing the content data during the presentation of the displayed content. The additional content may include summaries. In some embodiments, the additional content may include additional information and / or additional actions determined based on the output of a machine learning model. The output of the machine learning model may represent a semantic understanding of the displayed content and can be used to determine additional content associated with that semantic understanding. In some embodiments, the output of the machine learning model may include a topic determination, which can be used to determine additional content associated with that topic.

[0080] In 1608, a computing system may provide an interface for viewing data associated with display content and additional content. The interface may be provided in response to determining additional content associated with the display content. In some embodiments, the interface may include a display window that displays at least a portion of the display content. The interface may include a suggestion notice indicating additional content.

[0081] Figure 17 shows a flowchart of an exemplary method to be carried out according to an exemplary embodiment of the present disclosure. While Figure 17 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of the present disclosure are not limited to the order or arrangement shown. Various steps of Method 1700 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0082] Under 1702, a computing system can acquire content data. Content data may include instructions for display content provided to the user. Content data may include data indicating the display content. Display content may include portions of web pages, portions of documents, and / or other information provided for display.

[0083] In 1704, the computing system may process content data to determine entities associated with the displayed content. Entities may be determined based on the content within the displayed content (e.g., the title, images within the displayed content, and / or information contained in the body), based on data associated with a uniform resource locator, and / or based on an index lookup.

[0084] In 1706, a computing system can determine additional content associated with display content based on entities. Additional content can be obtained based on content data. In some embodiments, additional content can be determined by processing content data during the presentation of display content. Additional content can be determined by generating a search query based on entities, providing the search query to a search engine, and receiving one or more search results from the search engine.

[0085] In 1708, a computing system may provide an interface for viewing data associated with display content and additional content. The interface may include a display window that displays at least a portion of the display content. In some embodiments, the interface may include a suggestion notice indicating additional content.

[0086] Figure 18A shows a block diagram of an exemplary computing system 100 that performs additional content interface presentation according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, all of which are communicably coupled via a network 180.

[0087] The user computing device 102 can be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0088] The computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.), and may be a single processor or multiple processors connected in an operable manner. The memory 114 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 114 can store data 116 and instructions 118 executed by the processors 112 to cause the user computing device 102 to perform operations.

[0089] In some embodiments, the user computing device 102 may store or include one or more predictive models 120. For example, the content predictive model 120 may be a variety of machine learning models, such as neural networks (e.g., deep neural networks), or other types of machine learning models, including nonlinear and / or linear models, or may otherwise include them. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long-short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Exemplary content predictive models 120 are discussed with reference to Figures 2 to 11.

[0090] In some embodiments, one or more content prediction models 120 may be received from a server computing system 130 via a network 180, stored in user computing device memory 114, and then used by one or more processors 112, or otherwise implemented. In some embodiments, the user computing device 102 may implement multiple parallel instances of a single content prediction model 120 (for example, to perform parallel additional content prediction across multiple instances of display content items).

[0091] More specifically, the content prediction model 120 can be configured to process content data (e.g., uniform resource locators, text data, image data, latent encoded data, and / or other metadata) to determine additional content associated with the displayed content. The additional content can be determined by generating semantic data associated with the displayed content and querying a database based on that semantic data. Alternatively and / or additionally, the additional content can be determined by generating a search query based on the content data. In some embodiments, a predicted action type can be determined, and the additional content can be determined based on the predicted action type.

[0092] Additionally or alternatively, one or more content prediction models 140 may be included in, or otherwise stored and implemented in, a server computing system 130 that communicates with a user computing device 102 according to a client-server relationship. For example, a content prediction model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a content prediction service). Thus, one or more models 120 may be stored and implemented in the user computing device 102, and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0093] Furthermore, the user computing device 102 may include one or more user input components 122 that receive user input. For example, a user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may function to implement a virtual keyboard. Other exemplary user input components include a microphone, a regular keyboard, or other means by which the user can provide user input.

[0094] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 134 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 134 can store data 136 and instructions 138 executed by the processors 132 to cause the server computing system 130 to perform operations.

[0095] In some embodiments, the server computing system 130 includes or is implemented by one or more server computing devices. If the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or any combination thereof.

[0096] As described above, the server computing system 130 may store or otherwise include one or more machine learning-based content prediction models 140. For example, model 140 may be a variety of machine learning-based models, or may otherwise include a variety of machine learning-based models. Exemplary machine learning-based models include neural networks or other multilayer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are discussed with reference to Figures 2 to 11.

[0097] The user computing device 102 and / or the server computing system 130 can train models 120 and / or 140 by interacting with a training computing system 150 which is communicatively connected via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0098] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors connected in an operable manner. The memory 154 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processors 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices or is implemented by one or more server computing devices.

[0099] The training computing system 150 may include a model trainer 160 that trains machine-learned models 120 and / or 140 stored in the user computing device 102 and / or server computing system 130 using various training or learning techniques, such as backpropagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (for example, based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent can be used to iteratively update parameters over several training iterations.

[0100] In some embodiments, backpropagation may include performing truncated backpropagation over time. The model trainer 160 may perform several generalization techniques (e.g., load damping, dropout, etc.) to improve the generalization ability of the model being trained.

[0101] In particular, the model trainer 160 can train content prediction models 120 and / or 140 based on a set of training data 162. The training data 162 may include an exemplary training dataset, which may include, for example, training examples and ground truth data. The training examples may include exemplary content data examples (e.g., uniform resource locators, exemplary text, exemplary images, exemplary latent coding data, and / or exemplary embeddings). The ground truth data may include ground truth labels, ground truth predictions, ground truth action types, ground truth queries, and / or ground truth semantic data outputs.

[0102] In some embodiments, if the user consents, training examples can be provided by the user computing device 102. Thus, in such embodiments, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 with user-specific data received from the user computing device 102. In some cases, this process is referred to as model personalization.

[0103] The model trainer 160 includes computer logic used to provide a desired function. The model trainer 160 can be implemented in hardware, firmware, and / or software that control a general-purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored in a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer executable instructions stored in a RAM hard disk or a tangible computer-readable storage medium such as an optical or magnetic medium.

[0104] Network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or any combination thereof, and may include any number of wired or wireless links. Generally, communication over Network 180 can be conducted over any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formatting (e.g., HTML, XML), and / or protection methods (e.g., VPN, Secure HTTP, SSL).

[0105] The machine learning models described herein can be used in a variety of tasks, applications, and / or use cases.

[0106] In some embodiments, the input to the machine learning model(s) of this disclosure may be image data. The machine learning model(s) may process the image data to generate outputs. For example, the machine learning model(s) may process the image data to generate image recognition outputs (e.g., recognition of image data, latent embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model(s) may process the image data to generate image segmentation outputs. As yet another example, the machine learning model(s) may process the image data to generate image classification outputs. As yet another example, the machine learning model(s) may process the image data to generate image data modification outputs (e.g., modification of image data, etc.). As yet another example, the machine learning model(s) may process the image data to generate encoded image data outputs (e.g., encoded representation and / or compressed representation of image data, etc.). As yet another example, the machine learning model(s) may process the image data to generate upscaled image data outputs. As yet another example, the machine learning model(s) may process the image data to generate prediction outputs.

[0107] In some embodiments, the input to the machine learning model(s) of this disclosure may be text or natural language data. The machine learning model(s) may process the text or natural language data to produce an output. For example, the machine learning model(s) may process natural language data to produce a language coding output. As another example, the machine learning model(s) may process the text or natural language data to produce a latent text embedding output. As yet another example, the machine learning model(s) may process the text or natural language data to produce a translation output. As yet another example, the machine learning model(s) may process the text or natural language data to produce a classification output. As yet another example, the machine learning model(s) may process the text or natural language data to produce a text segmentation output. As yet another example, the machine learning model(s) may process the text or natural language data to produce a semantic intent output. As another example, a machine learning model(s) can process text or natural language data to produce upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As yet another example, a machine learning model(s) can process text or natural language data to produce predictive output.

[0108] In some embodiments, the input to the machine learning model(s) of this disclosure may be latent coded data (e.g., a latent spatial representation of the input). The machine learning model(s) may process the latent coded data to generate an output. For example, the machine learning model(s) may process the latent coded data to generate a recognition output. For another example, the machine learning model(s) may process the latent coded data to generate a reconstruction output. For yet another example, the machine learning model(s) may process the latent coded data to generate a search output. For yet another example, the machine learning model(s) may process the latent coded data to generate a reclustering output. For yet another example, the machine learning model(s) may process the latent coded data to generate a prediction output.

[0109] In some embodiments, the input to the machine learning model(s) of this disclosure may be statistical data. The machine learning model(s) may process the statistical data to generate an output. For example, the machine learning model(s) may process the statistical data to generate a recognition output. For another example, the machine learning model(s) may process the statistical data to generate a prediction output. For yet another example, the machine learning model(s) may process the statistical data to generate a classification output. For yet another example, the machine learning model(s) may process the statistical data to generate a segmentation output. For yet another example, the machine learning model(s) may process the statistical data to generate a segmentation output. For yet another example, the machine learning model(s) may process the statistical data to generate a visualization output. For yet another example, the machine learning model(s) may process the statistical data to generate a diagnostic output.

[0110] In some cases, the input includes visual data and the task is a computer vision task. In other cases, the input includes pixel data from one or more images and the task is an image processing task. For example, an image processing task could be image classification, with the output being a set of scores, each corresponding to a different object class and representing the likelihood that one or more images depict objects belonging to that object class. An image processing task could be object detection, with the image processing output identifying one or more regions within one or more images and, for each region, the likelihood that the region depicts an object of interest. Another example is image segmentation, with the image processing output defining the likelihood for each category within a given set of categories for each pixel in one or more images. For example, the set of categories could be foreground and background. Another example is that the set of categories could be object classes. Another example is depth estimation, with the image processing output defining the depth value for each pixel in one or more images. As another example, the image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted in the pixels between the images in the network input, for each pixel in the input images.

[0111] Figure 18A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can be used in a similar manner. For example, in some embodiments, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such embodiments, the model 120 can be trained and used locally on the user computing device 102. In some such embodiments, the user computing device 102 may implement a model trainer 160 that personalizes the model 120 based on user-specific data.

[0112] Figure 18B shows a block diagram of an exemplary computing device 40 that operates according to an exemplary embodiment of the present disclosure. The computing device 40 may be a user computing device or a server computing device.

[0113] The computing device 40 contains several applications (e.g., applications 1 to N). Each application contains its own machine learning library and one or more pre-trained models. For example, each application may contain a pre-trained model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, and a browser application.

[0114] As illustrated in Figure 18B, each application can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to that application.

[0115] Figure 18C shows a block diagram of an exemplary computing device 50 that operates according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0116] The computing device 50 includes several applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Exemplary applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, and browser applications. In some embodiments, each application can communicate with the central intelligence layer (and the model(s) stored within it) using an API (e.g., a common API across all applications).

[0117] The central intelligence layer includes several machine learning models. For example, as shown in Figure 18C, each machine learning model (e.g., Model) may be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model (e.g., Single Model) for all applications. In some embodiments, the central intelligence layer is contained within the operating system of the computing device 50, or otherwise implemented by them.

[0118] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As shown in Figure 18C, the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0119] The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as actions performed and information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of feasible configurations, combinations, and divisions of tasks and functions between components. For example, the processes described herein can be implemented using a single device or component, or multiple devices or components working together. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0120] While the subject matter has been described in detail with respect to various specific and exemplary embodiments, each example is provided for illustrative purposes only and does not limit the disclosure. Those skilled in the art will readily be able to create modifications, alterations, and equivalents to such embodiments, understanding the foregoing. Therefore, the disclosure does not exclude the inclusion of such modifications, alterations, and / or additions to the subject matter that would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used together with other embodiments to create yet another embodiment. Therefore, the disclosure is intended to cover such modifications, alterations, and equivalents.

Claims

1. A computing system for content prediction, One or more processors, One or more non-temporary computer-readable media that, when executed by the one or more processors, collectively store instructions that cause the computing system to perform an operation, wherein the operation is The acquisition of content data, wherein the content data includes instructions for displaying content to be shown to the user. Determining additional content associated with the display content, wherein the additional content is obtained based on the content data, determined by processing the content data during the presentation of the display content, and associated with one or more resources. Determining that a predictive action is associated with one or more of the resources based on the additional content, A computing system comprising a non-temporary computer-readable medium, which provides an interface for viewing data associated with the display content and the additional content in response to determining additional content associated with the display content, wherein the interface includes a proposal state, the proposal state includes a display window that displays at least a portion of the display content, the proposal state includes a proposal interface element that indicates the determination of the additional content, the interface includes an action interface element associated with the prediction action, the action interface element is selectable for performing the prediction action associated with one or more resources.

2. The computing system according to claim 1, wherein the displayed content is associated with a web page and the content data includes a uniform resource locator.

3. The computing system according to claim 1, wherein the interface includes a web page viewer and a preview bubble, the web page viewer providing a portion of the display content for display, and the preview bubble providing a snippet associated with the additional content.

4. The computing system according to claim 1, wherein the interface includes a scroll indicator and a bubble interface element, the scroll indicator indicates the position of the currently displayed portion of the display content relative to other portions of the display content, and the bubble interface element is provided to the interface adjacent to the scroll indicator.

5. The computing system according to claim 1, wherein the additional content includes a purchase link, and the purchase link is associated with a product associated with the displayed content.

6. The computing system according to claim 1, wherein the additional content includes an augmented reality experience, and the interface includes selectable user interface elements for providing the augmented reality experience.

7. The aforementioned operation is, To provide a proposed interface element for displaying in a first state, wherein the proposed interface element indicates whether additional content has been determined. The computing system according to claim 1, further comprising providing the proposed interface element for display in a second state in response to determining the additional content associated with the display content, wherein the second state indicates the determined additional content.

8. The aforementioned operation is, The process involves obtaining input data, wherein the input data indicates the selection of proposed interface elements of the interface. The computing system according to claim 1, further comprising providing a portion of the additional content for display.

9. Determining the additional content associated with the display content means that To determine the uniform resource locator associated with the aforementioned display content, The computing system according to claim 1, comprising determining additional web pages associated with the uniform resource locator.

10. Determining the additional content associated with the display content means that The computing system according to claim 9, further comprising generating additional content based on the aforementioned additional web pages.

11. Determining the additional content associated with the display content means that Determining several additional resources associated with the aforementioned display content, Determining multiple predictive actions associated with one or more of the aforementioned additional resources, The process includes generating multiple action interface elements, wherein the multiple action interface elements are associated with the multiple predictive actions, The computing system according to claim 1, wherein the plurality of Action interface elements are provided for display within the interface.

12. Determining the additional content associated with the display content means that To determine the machine-learned output, at least a portion of the display content is processed with the machine-learned model, The computing system according to claim 1, comprising determining the additional content based on the machine learning output.

13. The computing system according to claim 1, wherein the interface includes a swipe-up interface element configured to display a portion of the additional content based on user input.

14. Providing the interface for viewing the data associated with the display content and the additional content means that To provide at least a portion of the display content for display together with the proposed interface elements, To obtain the selection of the proposed interface element, The computing system according to claim 1, comprising providing at least a portion of the additional content for display.

15. The aforementioned operation is, Processing a portion of the display content to generate semantic data, wherein the semantic data represents a semantic understanding of the portion of the display content. The method further includes querying a database based at least partially on the aforementioned semantic data, The computing system according to claim 1, wherein the additional content is determined based on the query to the database.

16. The computing system according to claim 1, wherein the interface includes a type indicator associated with the content type of the additional content.

17. The computing system according to claim 16, wherein the type indicator indicates an action type, and the additional content is associated with performing a specific action.

18. The computing system according to claim 16, wherein the type indicator indicates the type of understanding, and the additional content provides supplementary information for understanding a specific topic associated with the displayed content.

19. A computer-based method for providing additional content, The acquisition of content data by a computing system comprising one or more processors, wherein the content data includes instructions for display content to be provided to the user. The computing system processes the content data with a machine learning model to generate machine learning model output, wherein the machine learning model output represents a semantic understanding of the displayed content. The computing system determines additional content, wherein the additional content is determined based on the output of the machine learning model generated in response to the content of the content data, and the additional content is determined by processing the content data during the presentation of the display content. In response to determining additional content associated with the aforementioned display content, the computing system provides a suggestion interface element for display, wherein the suggestion interface element for display includes a suggestion notification indicating the determined additional content. The computing system obtains the selection of the proposed interface elements, A method comprising: providing, in response to obtaining the selection of the proposed interface element, the computing system providing a swipe-up interface element to be displayed together with the display content for viewing the display content and data associated with the additional content, wherein the swipe-up interface element includes a display window that displays at least a portion of the display content.

20. One or more computer-readable storage media that, when executed by one or more computing devices, collectively store instructions that cause the one or more computing devices to perform an operation, wherein the operation is The acquisition of content data, wherein the content data includes instructions for displaying content to be shown to the user. Processing the content data in order to determine the entities associated with the displayed content, Determining additional content associated with the display content based on the entity, wherein the additional content is obtained based on the content data, determined by processing the content data during the presentation of the display content, and associated with one or more resources, Determining that a predictive action is associated with one or more of the resources based on the additional content, A computer-readable storage medium that provides an interface for viewing data associated with the display content and the additional content, wherein the interface includes a display window for displaying at least a portion of the display content, the interface includes a suggestion notice indicating the additional content, the interface includes an action interface element associated with the predictive action, and the action interface element is selectable for performing the predictive action associated with one or more resources.

Images