Mitigating latency and / or resource usage in triggering operable suggestions related to rendered content

Abstract

Implementations relate to triggering a suggestion for a document displayed at least in part by a content access application at a user interface of a client computing device. The suggestion may be triggered when one or more triggering conditions are satisfied that provide when to trigger the suggestion. The one or more trigger conditions may include, for example, a coordinate condition, a DOM node condition, and / or a temporal condition.

Description

Background Art

[0001] To access an Internet resource or other electronic document to obtain information, a user typically launches an application and navigates to the document. For example, a user may launch a web browser and navigate to an Internet resource by performing a search or entering a uniform resource locator (URL) address of the Internet resource. An example Internet resource is a web page, typically written in a markup language such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML).

[0002] Suggestions related to the content of an electronic document being rendered at an interface of a client device and provided at the interface while the electronic document is being rendered have been proposed. For example, while a first web page is being viewed, a suggestion for a second web page determined to be related to the first web page may be provided, and if the suggestion is selected, navigation to the second web page may be caused.

[0003] However, such suggestions are usually suggestions for viewing related content, rather than suggestions for performing actions (e.g., automated assistant actions), which are based on the content of the electronic document being rendered and are different from only navigating to related content. Additionally or alternatively, such suggestions are usually rendered in a static manner. For example, they can be rendered immediately when accessing the underlying web page, or after satisfying certain fixed conditions such as scrolling through a fixed percentage of the web page. Static rendering of related content interface notifications in a static manner may cause notifications to be rendered too early or too late (or not rendered at all) during the access of the underlying web page. When rendered too early, the related content interface notification may occupy the usually limited additional screen space and / or cause the user to temporarily divert attention and not view the web page, both of which may extend the amount of time spent by the user viewing the relevant part of the web page, thereby increasing the use of client device resources when rendering the web page. When rendered too late or not rendered at all, the user may ignore the notification or even not be presented with the notification, but instead find the related content through more computer resources and / or network intensive means (e.g., performing an Internet search). Summary of the invention

[0004] Some implementations disclosed herein involve determining trigger conditions that, when satisfied, will cause a client device to render suggestions (e.g., actionable suggestions) for a document (e.g., a web page, a portable document format (PDF) document, a screen of an app) displayed by the client device. Additionally or alternatively, some implementations disclosed herein involve monitoring whether determined trigger conditions for suggestions for a document are satisfied, and in response to determining that one or more of the trigger conditions are satisfied (e.g., the occurrence of any one of a plurality of trigger conditions), causing the suggestions to be rendered along with the display of the document.

[0005] Various implementations disclosed herein dynamically determine trigger conditions, e.g., on a document-by-document basis and / or on a client device-by-client device basis, to cause rendering of suggestions when interaction with the suggestions is likely. Some of those various implementations mitigate the occurrence of late placement of suggestions and / or the occurrence of early placement of suggestions, and mitigate the associated technical disadvantages of each. Additionally or alternatively, various implementations disclosed herein determine trigger conditions, and whether the trigger conditions are satisfied can be monitored locally on the client device in a manner that is efficient with the resources of the client device (e.g., processor, memory, and / or battery resources). Some of those various implementations provide for dynamic triggering of suggestions by a client device while mitigating use of the typically constrained resources of the client device.

[0006] In some implementations, trigger conditions for generating and utilizing a given suggestion for a given document rendered at a given client device may include coordinate conditions, document object model (DOM) conditions (sometimes also referred to as "DOM node conditions"), and / or time conditions. In some of those implementations, only one of the trigger conditions may be determined and utilized. In some other implementations, multiple trigger conditions (e.g., those of this paragraph and / or other trigger conditions) may be utilized, and whichever trigger condition is determined to be satisfied first may cause the given suggestion to be rendered.

[0007] As a working example for demonstrating aspects of generating and / or utilizing coordinate, DOM, and time conditions, take a web page that relates to a recipe for cooking a main dish and that includes: a first section that provides an overview of the main dish; a second section (after the first section) that provides a list of ingredients for the main dish; a third section (after the second section) that includes a list of kitchen tools required for the main dish; a fourth section (after the third section) that includes step-by-step instructions for cooking the main dish; and a fifth section that includes optional variations for cooking the main dish.

[0008] In a working example, an actionable suggestion for a web page can be an interface element that, when selected (e.g., tapped, clicked, or voice selected), causes the automated assistant to add ingredients from the web page to a shopping list maintained by the automated assistant and / or to interface with a separate application (e.g., a note taking app) to automatically add ingredients from the web page to the shopping list of the separate application. For example, the actionable suggestion can display the text "click to add these ingredients to note taking app". When the user selects the actionable suggestion, the automated assistant can extract the ingredients from the markup language of the web page (e.g., using tags of the markup language), or can access a list of pre-extracted ingredients and interface with a separate application (e.g., via an API) to cause those ingredients to be added to the list in the separate application. Optionally, the shopping list maintained by the assistant does not need to be brought to the foreground when adding ingredients, and / or the separate application does not need to be executed, at least not in the foreground, when interfacing with the automated assistant to add ingredients. Further, such actionable suggestions can prevent the need for the user to manually copy ingredients from a web page and copy them into a corresponding list, which may involve many user inputs (compared to a single selection of an actionable suggestion) and / or may require switching back and forth between multiple interfaces (e.g., between a browser application and a separate application). Switching back and forth between interfaces may be time consuming, thereby resulting in prolonged use of client device resources (e.g., battery) required, for example, to render content from an application.

[0009] Continuing with the working example, and turning to the coordinate condition, the client device may, in response to accessing a web page (or receiving a request to access a web page), send to the server computing device: document information of the web page and user interface information of the client device. The document information may be, for example, a uniform resource identifier (URI) or other identifier of the web page and / or at least some of the markup language of the web page. The user interface information may directly or indirectly identify the type of user interface (e.g., a smartphone, a tablet, a laptop, a small smartphone, a large smartphone, etc.), size information of the user interface (e.g., 5" vertical by 3.5" wide), orientation information of the user interface (e.g., vertical or horizontal), font size and / or type of the user interface (e.g., default font setting: Times New Roman, font size 11), whether an ad blocker is on or off, and / or other user interface information.

[0010] The server computing device may utilize document information to determine actionable suggestions for a web page. For example, in the case where the document information includes a URI, the actionable suggestions may be pre-indexed using the URI. For example, the server computing device (or another device) may have previously processed a web page to generate actionable suggestions for the web page, and may have previously indexed the actionable suggestions in association with the URI, so that the actionable suggestions can be efficiently identified and the actionable suggestions may be avoided from having to be regenerated. Further, the server computing device may also utilize document information to determine a location in the web page for rendering suggestions (also referred to herein as a "trigger portion"). In other words, the trigger portion may be a location in the web page corresponding to when the suggestion should be rendered. The suggestion may, but need not, actually be rendered at the location (e.g., the suggestion may be rendered in the bottom bar or top bar of the user interface instead). For example, the location may be determined based on its correspondence to the location of the target content (e.g., a list of ingredients including milk), and the suggestion may be determined based on the location of the target content (e.g., "add milk to shopping cart A (add milk to shopping cart A)"). For example, in the aforementioned working example, the trigger portion may be, for example, in the second section. As another example, the location can be a portion of the document (a "historical portion") and can be determined based on historical interaction data (also referred to herein as "historical user data") for the web page and / or similar web pages, which indicates when the corresponding user has interacted with (e.g., selected) the actionable suggestion (or similar actionable suggestion). For example, and in view of the aforementioned working example, the fourth segment (or a portion thereof) can be determined as a triggering portion based on the highest number of users who did not select the actionable suggestion (or similar actionable suggestion) until the fourth segment was rendered, even for those users for whom the fourth segment was previously rendered (e.g., those users may first want to determine the cooking complexity before adding ingredients to a shopping list). In this case, the location corresponds to the fourth segment.

[0011] Regardless of the technology utilized when determining the actionable suggestions and the aforementioned positions, the server can utilize the position and user interface information to generate coordinate conditions for triggering the rendering of the actionable suggestions. The coordinate conditions can specify the distance, for example, in the vertical and / or horizontal directions, that the interface needs to scroll to satisfy the triggering conditions. For example, the coordinate condition can be "scroll down 800 pixels" or "scroll down 12 inches". When generating the coordinate conditions based on the position and user interface information, the server computing device uses the document information to determine the coordinate conditions that will be satisfied when the position is rendered in the user interface of the client device in view of the user interface information. Therefore, the server computing device will generate different coordinate conditions for different user interface information. For example, for a user interface that is 6" vertical by 4" wide, the position can be rendered when "scroll down 500 pixels", and for a user interface that is 4.5" vertical by 3" wide, the position can be rendered when "scroll down 750 pixels". In some implementations, the server computing device can utilize the user interface information to simulate the rendering of a web page to determine when the position will be rendered on the client device. Optionally, for publicly accessible web pages or other documents, simulated rendering may be performed in advance for each of a plurality of different user interface information sets so that corresponding coordinate conditions for each user interface information set can be pre-determined and pre-stored. The pre-stored coordinate conditions that match (exactly or most closely) the received user interface information may then be selected for the client device. In addition to simulated rendering, additional and / or alternative techniques may be utilized to generate coordinate conditions, such as techniques for estimating coordinate conditions.

[0012] The server computing device provides the coordinate conditions to the client device in response to the client device sending the document information and the user interface information. The server computing device can also provide actionable suggestions to the client device. The client device can use the received coordinate conditions to monitor whether the coordinate conditions for rendering the web page in the user interface are met. For example, the coordinate information of the web page at the user interface can be efficiently monitored to determine when the user "scrolled down 500 pixels" or other coordinate conditions are met. When the coordinate conditions are determined to be met by the client device, the rendering of the actionable suggestions can be caused. Monitoring the aforementioned coordinate information to determine whether the coordinate conditions are met can be efficiently performed by the client device (for example, via a browser application) using the coordinate information that is already available to the client device. Therefore, there is no need for the client device to perform a more computationally intensive analysis of the web page, thereby saving the resources of the client device and enabling the dynamic triggering of actionable suggestions. Further, by considering the user interface information of the client device, the server computing device can generate coordinate conditions customized for the client device, so that the server computing device can finely control the client device through the configuration of the coordinate conditions.

[0013] Continuing with the working example, and turning to the DOM condition, the client device can, in response to accessing a web page (or receiving a request to access a web page), send to the server computing device: document information of the web page. The document information can be, for example, a uniform resource identifier (URI) or other identifier of the web page and / or at least some of the markup language of the web page.

[0014] The server computing device can utilize document information to determine actionable suggestions for a web page. For example, when the document information includes a URI, the actionable suggestions can be pre-indexed using the URI. Further, the server computing device can also utilize document information to determine the DOM node of the DOM of the web page for rendering suggestions. For example, the DOM node can be determined based on its correspondence with the target content (i.e., the DOM node in the second section). As another example, the DOM node can be determined based on historical interaction data of a web page and / or similar web pages, and the historical interaction data indicates when the corresponding user has interacted with the actionable suggestions (or similar actionable suggestions) (e.g., selected the actionable suggestions). For example, based on the highest number of users who did not select the actionable suggestions (or similar actionable suggestions) until the DOM was rendered, the DOM node can be in the fourth section.

[0015] The server computing device provides a DOM node to the client device in response to the client device sending document information. The server computing device can also provide actionable suggestions to the client device. The client device can use the received DOM node to monitor the active execution of the markup language corresponding to the DOM node when rendering the document in the user interface. In other words, the client device can monitor the rendering of the content corresponding to the DOM node of the web page. When the client device determines that the DOM node is being rendered, it can cause the rendering of actionable suggestions. The monitoring of the rendering of the DOM node can be efficiently performed by the client device (for example, via a browser application) using coordinate information that is already available to the client device. Therefore, there is no need for the client device to perform a more computationally intensive analysis of the web page, thereby saving the resources of the client device and enabling dynamic triggering of actionable suggestions. Further, the server computing device finely controls the client device through the configuration of the DOM node.

[0016] Continuing the working example, and turning to the time condition, the historical distribution of the selection time can be utilized when generating the time condition for the web page and for the client device. The historical distribution of the selection time can be generated based on the historical selection time for the actionable suggestion (or similar actionable suggestion) and optionally for the web page and / or similar web page. Each historical selection time can indicate when the corresponding user interacts with the actionable suggestion (or similar actionable suggestion) (e.g., selects the actionable suggestion). As an example, the historical distribution of the selection time for the web page can be normally distributed, wherein the median is 20 seconds, 1 standard deviation above the median is 25 seconds, 1 standard deviation below the median is 15 seconds, 2 standard deviations above the median is 35 seconds, and 5 seconds below the median two standard deviations. The historical distribution of the selection time can be identified by the client device or server device (e.g., based on the document information provided by the client device), based on being indexed in association with the web page (or similar web page) and / or with the actionable suggestion (or similar actionable suggestion). Further, the client device or server device can generate the time condition based on the historical distribution (e.g., using the median of the distribution). Still further, the client device may efficiently monitor whether a temporal condition is satisfied (eg, by monitoring the amount of time since the web page was loaded), and cause rendering of an actionable suggestion when the temporal condition is satisfied.

[0017] In some implementations, when generating time conditions based on historical distributions, random sampling of the historical distribution can be optionally utilized, and the sampled time can be used as the time condition. Therefore, instead of always selecting the median (or mean), the selection can be random, but guided by the distribution. For example, for a normal distribution, the mean may have a higher probability of being randomly selected, but it will not be selected every time. This can enable the historical distribution to guide the determination of the time condition while still providing changes in the time condition. This can enable more meaningful additional data to be generated and used to improve the historical distribution that changes over time. For example, if, on the contrary, the median is always selected, the historical distribution that changes over time will shift toward a longer amount of time, causing the corresponding actionable suggestions to arrive late. These and / or other disadvantages can be alleviated by random sampling.

[0018] In some implementations, when generating a time condition based on a historical distribution, the historical distribution is sampled depending on historical metrics of selected suggestions made by a client device and / or by a user account associated with a visit to a web page at the client device. For example, assume that a first user account has a historical metric indicating that a presented actionable suggestion greater than a first threshold (e.g., 30%) is selected, and a second user account has a historical metric indicating that a presented actionable suggestion less than a second threshold (e.g., 5%) is selected. For the first user account, a time that is two standard deviations lower (e.g., 5 seconds) may be selected from the historical distribution. Conversely, for the second user account, a time that is two standard deviations higher (e.g., 35 seconds) may be selected from the historical distribution. For a third user having a historical metric indicating that the selection rate is greater than the second threshold but less than the first threshold, a median or mean time may be selected from the historical distribution. Therefore, the time condition, when guided by the historical distribution, will be selected depending on the historical metrics of the client device and / or the user account for which the time condition is being generated. This can mitigate the occurrence of early arrival and / or late arrival by customizing the arrival time depending on the selection tendency of previous actionable suggestions.

[0019] Various implementations disclosed herein can enhance the user experience by adaptively triggering suggestions for display that meet the needs of the user to view additional content or perform an action (e.g., a third-party action) based on her or his encounter with a content access application (e.g., the aforementioned web browser or browser application) that provides certain content (e.g., target content such as recipes). The triggering of suggestions can be adaptive in that different triggering conditions can be applicable to trigger suggestions for documents (public or private) accessed via the content access application. As disclosed herein, although the triggering of suggestions is generally caused by a client computing device (e.g., the aforementioned "client device"), depending on one or more factors (e.g., whether the document is public and / or the resource constraints of the client computing device), the determination of the triggering conditions (e.g., when exactly to trigger the suggestion) can be performed by the client computing device or can be delegated to one or more server computing devices (e.g., the aforementioned "server device") that communicate with the client computing device.

[0020] For example, when the document includes private information or is a private document, the determination of the trigger condition may be performed by the client computing device. In this example, to reduce computing resources, the client computing device may analyze a portion of the document that is displayed in the foreground, rather than analyzing the entire document. As another example, when the document is lengthy or complex, the determination of the trigger condition may be delegated by the client computing device to one or more server computing devices in communication with the client computing device to save local computing resources of the client computing device, thereby ensuring the efficiency of the client computing device in response to detecting a trigger portion (e.g., target content) in the document for triggering a suggestion based on the target content.

[0021] Additionally or alternatively, some implementations disclosed herein relate to detecting a triggering portion, in response to which a triggering condition is applicable to trigger one or more suggestions. In the case of a web page providing a recipe, the triggering portion of the document may include target content (e.g., a list of ingredients in a recipe) on which a suggestion (e.g., an actionable suggestion) is based and / or a specific portion of the document that is different from the target content and determined based on historical user data. Optionally, the specific portion of the document may be more remotely associated with the suggestion than the target content on which the suggestion is based. For example, a specific portion of a document (whose appearance at a user interface causes more than 40% of historical users to select an actionable suggestion displayed at the user interface along with the specific portion) may be determined as a triggering portion (or determined as a triggering portion of multiple triggering portions that includes the target content).

[0022] Similar to the determination of the trigger condition, the detection of the trigger portion can be performed by the client computing device, or delegated to one or more server computing devices in communication with the client computing device. For example, when a document provided by a content access application at a client computing device includes private content, the client computing device can detect whether a portion of the document displayed at the user interface of the client computing device includes any target content (or a specific portion) before continuing to determine one or more trigger conditions for triggering suggestions. In various implementations, the client computing device can delegate the task of determining / detecting whether the document includes the target content to one or more server computing devices to save computing resources and improve response efficiency. In various implementations, the client computing device can determine whether the portion of the document running in the foreground includes the target content, but delegate the determination of the trigger condition to one or more server computing devices.

[0023] In some implementations, whether target content is detected from a portion of a document displayed at a user interface of a client computing device can be determined by detecting whether the portion of the document displayed at the user interface of the client computing device includes one or more types of mode tags. As an example, a user of a client computing device (e.g., a cellular phone or tablet) can use a content access application (e.g., a web browser or social media) to open a document (e.g., a web page showing a recipe, a story about a recipe, an advertisement related to or unrelated to the recipe, etc.). When the document is loaded by the content access application so that the first portion of the document is displayed at the user interface of the client computing device, the client computing device can determine whether any of the aforementioned trigger conditions are met (e.g., determining whether the user interface of the client computing device in which the first portion of the document is shown includes coordinate conditions of any target content such as a recipe based at least on the location of the target content in the document and the user interface information). In response to determining that one (or more) of the target content trigger conditions are met, the client computing device can render a suggestion (or more than one suggestion) related to the target content via the user interface of the client computing device.

[0024] Optionally, whether target content is detected from the portion of the document displayed at the user interface of the client computing device may be performed by the client computing device each time a user of the client computing device loads a document and / or each time the user scrolls (e.g., scrolls up or scrolls down) to navigate the document.

[0025] In some implementations, whether a user interface of a client computing device showing a first portion of a document includes any target content can be determined by accessing, using the client computing device, schema markup for a first portion of a document (e.g., a web page) displayed at the user interface and determining whether one or more schema markup of a certain type (e.g., schema markup for a recipe) is present in the foreground of the client computing device. For example, if schema markup for a recipe is determined by the client computing device to be present in the foreground, then it can be determined that the aforementioned DOM condition is satisfied, and in response, a suggestion can be triggered for display at a user interface (e.g., a GUI) of the client computing device or at a different user interface (e.g., an audio interface) of the client computing device.

[0026] Here, using a client computing device to detect target content can be advantageous because only a portion of the document (e.g., the first portion) rather than the entire document needs to be analyzed to determine whether any target content is displayed in the foreground. The user can scroll to the left so that a second portion of the document is displayed at the user interface of the client computing device, where the second portion is different from the first portion. The client computing device can then access the second portion of the document rather than the entire document to determine whether the second portion of the document includes target content to trigger one or more suggestions based on the detection of the target content. In addition, using a client computing device to detect target content can enable the detection of target content of a document that is private or locally stored at a client computing device without sending the document over one or more networks for processing by a server computing device different from the client computing device.

[0027] The suggestion may be a selectable element having natural language content that suggests a third-party action that may be performed via a third-party application. For example, the suggestion may be a selectable element that displays “Add product X to app A shopping cart,” where “Add product X to app A shoppingcart” is a third-party action that may be performed via a third-party application “app A.” If a user of a client computing device selects the selectable element, the client computing device may cause the third-party action to be performed via the third-party application. Here, the suggestion may be generated, for example, by a remote server that is the same as or different from the aforementioned server computing device, for one or more target content of a document. For example, “product X” may be included in the natural language content of the selectable element based on a determination that “product X” provides or corresponds to ingredients listed in a recipe that is target content of a web page.

[0028] As an example, a user of a client computing device (e.g., a desktop computer) may use a content access application (e.g., a web browser) to open a document (e.g., a web page showing a recipe). When the document is loaded by the content access application for display at a user interface of the client computing device, the client computing device may send the following information to the server computing device: (1) document information of the document, including, for example, an address of the document for retrieving the document (e.g., a URL); and / or (2) user interface information of the user interface of the client computing device, including, for example, the size (width and / or height) of the user interface (e.g., a screen) of the client computing device and / or the font (type and size) of the content included in the document. The server computing device may use the document information to determine whether the document includes target content (e.g., by retrieving the web page using a URL address), and if target content is detected, the server computing device may use the document information and the user interface information to determine coordinate information of the target content in the document displayed at the user interface of the client computing device. It should be noted that the coordinate information of the target content in the document displayed at the user interface of the client computing device may vary depending on, for example, the screen size of the client computing device ("size of the user interface").

[0029] The coordinate information of the target content may include a first distance (measured in pixels or percentages) from the top (or bottom) edge of the document to the target content, and / or a second distance (measured in pixels or percentages) from the side (left or right) edge of the document to the target content. The aforementioned coordinate condition may be determined based on the coordinate information of the target content. For example, if the user scrolls down a first distance (and / or scrolls to the right a second distance), the server computing device may determine that the coordinate condition is satisfied so that the target content is displayed in the foreground of the client computing device. Optionally, in response to the server computing device determining, based on the coordinate information of the target content, that the user has scrolled the document to reach the target content so that the coordinate condition is satisfied, the server computing device may generate a suggestion trigger signal and send the suggestion trigger signal to the client computing device, so that the client computing device may trigger rendering of the suggestion at the user interface based on the receipt of the suggestion trigger signal.

[0030] In some implementations, the server computing device may generate rules for coordinate conditions (i.e., when the coordinate conditions are met to trigger suggestions based on the target content or the aforementioned triggering portion), and send the rules for coordinate conditions to the client computing device so that when the coordinate conditions are determined to be met, the client computing device may cause the suggestions to be rendered at the user interface of the client computing device. Using a server computing device to detect target content and generate rules for coordinate conditions for triggering one or more suggestions based on the target content can save computing resources of the client computing device, especially when the document is lengthy or includes complex content / features.

[0031] Optionally or alternatively, when a document (e.g., a web page) is loaded by a content access application (e.g., a browser) for display at a user interface of a client computing device, the client computing device may determine document information of the document and send the document information to the server computing device, the document information including, for example, an address (e.g., URL) of the document and / or the content of the document. Based on the address of the document (e.g., a web page), the server computing device may access the document in a tree data structure. For example, the server computing device may access a web page via a document object model (DOM), wherein the content of the web page is arranged using a plurality of nodes forming a tree data structure. In this example, the server computing device may determine whether a target node (e.g., an HTML tag with a specific name or index number) representing the target content is displayed at the user interface of the client computing device. When the server computing device determines that the target node representing the target content is displayed at the user interface, the server computing device may generate a suggestion trigger signal and send the suggestion trigger signal to the client computing device, so that the client computing device may trigger rendering suggestions at the user interface based on the receipt of the suggestion trigger signal.

[0032] The above is provided only as an overview of some implementations. Those implementations and / or other implementations are disclosed in more detail herein.

[0033] Various implementations may include a non-transitory computer-readable storage medium storing instructions executable by a processor to perform a method, such as one or more of the methods described herein. Still other various implementations may include a system including a memory and one or more hardware processors operable to execute instructions stored in the memory to perform a method, such as one or more of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings. In the accompanying drawings:

[0035] Figure 1A and Figure 1B Accordingly, a block diagram of an example environment demonstrating various aspects of the present disclosure and in which implementations disclosed herein may be implemented is depicted.

[0036] Figure 2A Describes the various implementations Figure 1A or Figure 1B A sample user interface for .

[0037] Figure 2B Describes the various implementations Figure 1A or Figure 1B Another example user interface for .

[0038] Figure 2C Describes the various implementations Figure 1A or Figure 1B Another example user interface for .

[0039] Figure 2D Describes the various implementations Figure 1A or Figure 1B Another example user interface for .

[0040] Figure 2E Describes the various implementations Figure 1A or Figure 1B Another example user interface for .

[0041] Figure 3 is a flow chart illustrating an example method of triggering on-content suggestions for a content access application according to various implementations.

[0042] Figure 4 is a flow chart illustrating another example method of triggering on-content suggestions for a content access application according to various implementations.

[0043] Figure 5 is a flow chart illustrating additional example methods of triggering on-content suggestions for a content access application according to various implementations.

[0044] Figure 6 Example architectures of computing devices according to various implementations are presented. DETAILED DESCRIPTION

[0045] The following description with reference to the accompanying drawings is provided for understanding the various implementations of the present disclosure. It should be understood that different features from different embodiments can be combined and / or exchanged with each other. In addition, those of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. For the sake of clarity and simplicity, the description of well-known or repeated functions and structures may be omitted.

[0046] The terms and words used in the following description and claims are not limited to the bibliographical meanings, and are used merely by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for the purpose of illustration only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0047] Figure 1Ais a block diagram of an example environment that demonstrates various aspects of the present disclosure and in which the implementations disclosed herein may be implemented. Figure 1A As shown, the environment may include a client computing device 11 and a server computing device 13 (or other device) communicating with the client computing device 11 via one or more networks 15. The client computing device 11 may be, for example, a cellular phone, a laptop computer, a desktop computer, a notebook computer, a tablet, a smart TV, a messaging device, or a personal digital assistant (PDA), and the present disclosure is not limited thereto. The server computing device 13 may be, for example, a web server, a proxy server, a VPN server, or any other type of server as desired. The one or more networks 15 may include, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, and / or any other appropriate network.

[0048] In some implementations, the server computing device 13 may include a suggestion generation engine 131 that generates one or more suggestions based on the document sent to the content generation engine 131 for processing. The server computing device 13 may further include a trigger condition determination engine 133 and a suggestion trigger engine 135. The trigger condition determination engine 133 may, for example, include a coordinate condition engine 1331, a DOM condition engine 1333, a time condition engine 1335, and / or other condition engines 1337 (e.g., an attention tracking condition engine).

[0049] The coordinate condition engine 1331 may, for example, determine whether a document includes a trigger portion, wherein detection of the trigger portion in the document triggers a suggestion (e.g., an actionable suggestion). The trigger portion may be the target content of the document, wherein the target content is processed to generate the suggestion. Alternatively or additionally, the trigger portion may be a specific portion of the document that is different from the target content (e.g., after the target content) and is optionally determined based on historical user data associated with the document. For example, the specific portion of the document may be determined as the trigger portion based on historical user data indicating that when the specific content of the document is displayed at a specific location of the user interface of the client computing device 11, the number (or percentage) of historical users who selected the suggestion (generated based on the document or the target content of the document) exceeds a threshold (the threshold is a quantity threshold or a percentage threshold). The coordinate condition engine 1331 may further determine the location of the trigger portion in the document, and / or the location of the trigger portion at the user interface of the client computing device 11. The location of the trigger portion at the user interface may be determined, for example, using user interface information of the document and the user interface, which may include the size of the user interface, the default font size / type of the user interface, and / or other information related to the user interface.

[0050] Optionally, based on the position of the triggering portion at the user interface of the client computing device 11 and the user interface, the coordinate condition engine 1331 can determine whether the triggering condition (e.g., the coordinate condition in this case) is met. If it is determined that the coordinate condition is met, the suggestion trigger engine 135 can trigger the rendering of the suggestion at the user interface as an overlay element of the document that is immediately adjacent to the triggering portion (e.g., adjacent to the triggering portion), at the top bar of the user interface, at the bottom bar of the user interface, or at other areas of the user interface. Optionally, the position in the document for rendering the suggestion can be determined based on historical user data, which indicates the historical position of one or more suggestions rendered for user interaction in history. For example, the historical position of the suggestion with the highest user selection frequency in (or above) the document can be determined as the position of the suggestion. The present disclosure is not intended to limit the position where the suggestion is to be rendered. In some implementations, the coordinate condition engine 1331 can determine that the coordinate condition is met by processing the user interface and based on processing the position of the scroll bar detected indicating that the user has scrolled down, for example, 800 pixels, 12 inches, or 30% of the document, which is within the range where the triggering portion is displayed at the user interface.

[0051] As a non-limiting example, the coordinate condition engine 1331 may process a document, i.e., a web page, to determine that the web page includes a list of ingredients as target content and / or the location of the target content in the document, the appearance of which at the user interface may trigger rendering of a suggestion at the user interface (“add the list of ingredients to my shopping list in app A”). In some implementations, the coordinate condition engine 1331 may further determine the location of the target content relative to the user interface (e.g., whether the user interface displays the target content, or the distance or percentage that the user needs to scroll down to see the target content) based on user interface information of the user interface of the client computing device 11 (e.g., the size of the user interface and the default font of the document when displayed at the user interface). When the coordinate condition engine 1331 determines that the user has scrolled down, for example, 500 pixels (which is between 400 and 600 pixels where all (or a desired portion) of the target content is displayed at the user interface), the coordinate condition engine 1331 may determine that the coordinate condition is satisfied. In response to the coordinate condition engine 1331 determining that the coordinate condition is satisfied, the suggestion triggering engine 135 can trigger a suggestion to be rendered at the user interface, where the suggestion can be an actionable suggestion of a first-party action that can be performed by the automated assistant (i.e., an automated assistant action) or a third-party action that can be performed by a third-party action (e.g., a calendar application, a shopping app).

[0052] The DOM condition engine 1333 may, for example, access a DOM tree structure (also referred to as "DOM") of a document having a plurality of DOM nodes, and detect whether the plurality of DOM nodes include a trigger node for triggering a suggestion. In some implementations, the trigger node may be determined based on the content of the document. For example, the trigger node may be a target node corresponding to the target content of the document, or may be an interest node determined based on historical user data and corresponding to the aforementioned specific portion of the document. Detecting whether the plurality of DOM nodes include a target node may be achieved by determining whether the markup language of the document includes a target tag (or an interest tag, such as an HTML tag with a specific name or index number). If it is determined that the markup language includes a target tag, the DOM condition engine 1333 may further determine whether the target content (or a specific portion of the document) corresponding to the target node (or the interest tag) is being rendered at the user interface of the client computing device. When the DOM condition engine 1333 detects that the trigger node (target node or interest node) is running in the foreground at the user interface of the client computing device (i.e., the trigger portion is being rendered at the user interface), the DOM condition engine 1333 may determine that the DOM condition is satisfied.

[0053] Alternatively or additionally, the DOM condition engine 1333 may determine whether the plurality of DOM nodes include a trigger node (also referred to as a "trigger DOM node") based on historical user data indicating, for example, suggestions most frequently selected by historical users when the trigger node was rendered in the foreground. The present disclosure is not limited thereto.

[0054] In response to the DOM condition engine 1331 determining that the DOM condition is satisfied, the suggestion triggering engine 135 may trigger rendering of suggestions at the user interface, where the suggestions may be actionable suggestions determined using the target content (but not necessarily using a specific portion of the document).

[0055] The time condition engine 1333 may, for example, access historical user data to obtain a historical distribution of selection times taken by historical users / readers of the document to select suggestions. The time condition engine 1333 may select, for example, the average or median time taken by historical users / readers of the document to select suggestions as a time condition for triggering suggestions. In this case, if it is detected that the user of the document has opened the document at the user interface (the currently running foreground of the client computing device 11) for a period of time exceeding the average (or median) time determined from the historical user data, the time condition engine 1333 may determine that the time condition is satisfied, and in response, cause the suggestion triggering engine 135 to trigger rendering of the suggestion in the user interface.

[0056] In some implementations, the trigger condition engine 133 may be accessed at the server computing device 13. In some other implementations, the trigger condition engine 133 may be accessed at the client computing device 11. In some other implementations, the trigger condition engine 133 may be accessed at both the client computing device 11 and the server computing device 13. In some other implementations, the trigger condition engine 133 may be partially (e.g., its time condition engine 1335) accessed at the client computing device 11, and partially (e.g., its coordinate condition engine 1331 and DOM engine 1333) accessed at the server computing device 13, and the present disclosure is not limited thereto. In some other implementations, the coordinate condition engine 1331 or a portion thereof may be accessed at the client computing device 11 (and / or the server computing device 13). The DOM condition engine 1333 or a portion thereof may be accessed at the client computing device 11 (and / or the server computing device 13). The temporal condition engine 1335 or a portion thereof may be accessed at the client computing device 11 (and / or the server computing device 13).

[0057] In various implementations, the client computing device 11 may be installed with or otherwise have access to a content access application 110. The content access application 110 may access content or data (e.g., documents) stored at a data source such as data sources 17a, 17b to 17n via one or more networks 15. Alternatively or additionally, the content access application 110 may access content or data from a data storage device 115 of the client computing device 11. As a non-limiting example, the content access application 110 may be a web browser that retrieves one or more web pages from one or more of the aforementioned data sources and displays the one or more web pages to a user of the client computing device 11. The web browser may be a stand-alone web browser, or an in-app web browser that is a web-enabled component of a larger application, and the present disclosure is not limited thereto. After being launched at the client computing device 11, the web browser may allow the user to access a web page of interest to the user by performing a search for the web page using a keyword and selecting a web page from one or more web pages listed as search results, or by entering the uniform resource locator (URL) address of the web page of interest at the search bar of the web browser.

[0058] Alternatively or additionally, the web browser may allow the user to access third-party applications, such as email, via a hyperlink or the aforementioned suggestion of a web page displayed by the web browser, for example. In some implementations, the content access application 110 may be, for example, a reader application (or other application) that can access and display documents (e.g., e-books, PDFs, word documents, etc.) stored locally in the data storage device 115 or remotely stored at a data source such as the data source 17a, rather than a web browser. Alternatively, the document may be a private document, and the client computing device 11 (or server computing device 13) may process a portion or all of the document, for example, using a license or authorization.

[0059] In various implementations, the client computing device 11 may include a target content detection engine 111 that detects whether the content of the user interface of the client computing device 11 includes target content, such as target natural language content or a target image. As a non-limiting example, the content access application 110 may run in the foreground and open a document, such as a web page, for display via the user interface of the client computing device 11, and the target content detection engine 111 of the client computing device 11 may detect whether the user interface of the client computing device 11 displaying a portion of the document (in the case of a lengthy document) includes target content, such as a recipe that the user is interested in. In some implementations, the target content detection engine 111 may parse the portion of the document displayed at the user interface of the client computing device 11 to detect the target content.

[0060] As a non-limiting example, for a portion of a web page displayed by a web browser at a user interface of the client computing device 11, the target content detection engine 111 can access the document object model (DOM) of the web page, wherein the DOM displays the HTML file of the web page as a structured data tree with multiple nodes. The multiple nodes in the structured data tree may include: a root node, one or more element nodes, one or more text nodes, and one or more attribute nodes. In this example, the target content detection engine 111 can determine whether the multiple nodes include a target node representing the target content. As another non-limiting example, the target content detection engine 111 can detect the target content by determining whether a certain mode tag exists in the portion of the document (e.g., web page) displayed in the foreground at the client computing device 11.

[0061] In some implementations, the target content detection engine 111 may be configured to detect whether the target content is present in the foreground of the client computing device 11 whenever the content displayed at the user interface of the client computing device 11 changes or whenever new content is displayed at the user interface of the client computing device 11. For example, the target content detection engine 111 may detect whether the portion of the document displayed at the user interface of the client computing device 11 includes the target content in response to the document being loaded at the content access application 110. As another non-limiting example, the target content detection engine 111 may detect whether the portion of the document displayed at the user interface of the client computing device 11 includes the target content in response to the user of the client computing device 11 scrolling down the document (e.g., a web page) displayed at the user interface of the client computing device 11. As a further non-limiting example, the target content detection engine 111 may detect whether the portion of the document displayed at the user interface of the client computing device 11 includes the target content in response to the user of the client computing device 11 scrolling up the document (e.g., a web page) displayed at the user interface of the client computing device 11.

[0062] Alternatively, the target content detection engine 111 may be included in the content access application 110 or otherwise accessible by the content access application 110. Alternatively, when the aforementioned coordinate condition engine 1331 (or DOM engine 1333) is accessed at the client computing device 11, the target content detection engine 111 may be part of the coordinate condition engine 1331 (or DOM engine 1333). Alternatively or additionally, the target content detection engine 111 may have a cloud-based counterpart accessible at the server computing device 13.

[0063] In various implementations, the client computing device 11 may include a suggestion triggering engine 113 (same or different than the aforementioned suggestion triggering engine 135) that determines when to trigger a suggestion for rendering to a user. In some implementations, the suggestion triggering engine 113 may determine whether one or more triggering conditions are met / satisfied (or otherwise monitor when any of the one or more triggering conditions are met / satisfied), and if one of the one or more triggering conditions is met, the suggestion triggering engine 113 may trigger rendering of the suggestion. The one or more triggering conditions may include, for example, an attention tracking condition that indicates whether a minimum period of time required to show user attention has elapsed since the most recent user scrolling input (e.g., the last time the user scrolled a scroll bar) and / or whether the user has attempted to copy and paste a phrase from a document since the document was loaded by the content access application 110. In this example, the minimum period of time may optionally be determined based on the aforementioned historical user data that provides a distribution of the time that historical users of the content access application 110 (or the client computing device 11) took to select suggestions after the document was loaded by the content access application 110. Alternatively or additionally, the minimum period may be determined based on other means, such as studies that publicly show the time period required for a user to show interest in reading a document. For example, when it is detected that the user has stopped scrolling the scroll bar for a minimum period of time, it may be determined (e.g., by the aforementioned attention tracking condition engine) that the attention tracking condition is satisfied. For example, when it is detected that the user has selected one or more phrases from the document for copying and pasting, highlighting, or commenting, it may also be determined that the attention tracking condition is satisfied.

[0064] As another example, one or more trigger conditions may include a target content coordinate condition. The target content coordinate condition may be part of the aforementioned coordinate condition, and the coordinate information of the target content in the document may be used to indicate whether the target content is displayed in the foreground of the client computing device 11. In this example, the coordinate information of the target content may include a first distance from the target content to the top edge of the document, measured in pixels (or inches). Alternatively or additionally, the coordinate information of the target content may include a second distance from the left (or right) edge of the document to the target content, measured in pixels. Alternatively, the coordinate information may include a percentage that the user needs to scroll to reach the target content. It should be noted that, depending on the client computing device 11 and / or the content access application 110, the coordinate information of the target content of the document displayed via the user interface of the client computing device 11 may vary. Therefore, when the suggestion trigger engine 113 is included in the server computing device 13 to detect one or more trigger conditions of the suggestion, the client computing device 11 may need to send user interface information to the server computing device 13, such as the width, height and / or font size of the user interface of the client computing device 11.

[0065] In various implementations, the client computing device 11 may further include an automated assistant 112, which refers to an interactive software application (also referred to as a "chatbot," "interactive personal assistant," "intelligent personal assistant," "personal voice assistant," "conversational agent," etc.). The client computing device 11 may use (or not use) the automated assistant 112 to generate or otherwise retrieve targeted content-based suggestions. For example, the client computing device 11 may retrieve one or more suggestions from the server computing device 13. As another example, the suggestion may be a suggestion for an action (e.g., a third-party action) that may be performed via a third-party application (e.g., "add milk to shopping cart A"), and the client computing device 11 may rely on the automated assistant 112 to perform one or more preemptive tasks. The one or more preemptive tasks may include, but are not limited to: determining whether the third-party application is installed or otherwise accessible to the client computing device 11, determining whether the user of the client computing device 11 has an account for the third-party application, and / or determining whether the user of the client computing device 11 is logged into an account for the third-party application.

[0066] Optionally, the suggestion may also be a suggestion of additional content that is different from the target content but derived from the target content, such as a link to a website showing an alternative recipe. Optionally, the target content detection engine 111 may be included in or otherwise accessible to the automated assistant 112.

[0067] In various implementations, the client computing device 11 may cause the suggestion (or one of the one or more suggestions) to be rendered at a user interface of the client computing device 11 that displays the portion of the document that includes the target content.

[0068] Such suggestions may be selectable. For example, a user may click (using a keyboard, finger touch, or gesture) on a suggestion to select the suggestion. As another example, a user may select a suggestion by providing an audio input via a microphone. A user may not (or may) use automated assistant 112 to interact / select a suggestion. As a non-limiting example, a user may audibly select a suggestion by providing a verbal utterance “yes please”, such as “add milk to your appA shopping cart”. In this example, the verbal utterance may be processed by automated assistant 112 so that a third-party action (e.g., adding milk to appA shopping cart) is performed. Alternatively, a user may audibly select a suggestion such as “copy the list of ingredients to my note-takingapplication” so that a first-party action (i.e., in this case, “copy the list of ingredients”) is performed by automated assistant 112.

[0069] As another non-limiting example, a user may audibly select a suggestion, such as “see substitutions for milk” by providing the spoken utterance “yes please.” In this example, the spoken utterance (e.g., “yes please”) may be processed by automated assistant 112 to confirm that the user wants to see substitutes for milk, such that automated assistant 112 may perform a search, and additional content based on the search results (e.g., one or more substitutes for milk in a recipe displayed by a web page) may be rendered to the user.

[0070] One or more components of the automated assistant 112 will be described later in this disclosure with reference to the description of the cloud-based automated assistant 19. Optionally or alternatively, reference may be made to Figure 1B , the content access application 110 may include an automated assistant component 1101 (also referred to as "automated assistant functionality"), for example, to cause changes to the interface of the content access application 110 in response to receiving voice input (also referred to as "voice command") from a user. Figure 1A and Figure 1B , the client computing device 11 may include other applications 114. The other applications 114 may include, for example, the aforementioned third-party applications, which may be shopping app A or any other application when appropriate.

[0071] Optionally, client computing device 11 may cause suggestions to be rendered at a top bar of a user interface of client computing device 11, or at a top bar of an interface of content access application 110 (which may be part of a user interface of client computing device 11). Optionally, client computing device 11 may cause suggestions to be rendered at a bottom bar of a user interface of client computing device 11, or at a bottom bar of an interface of content access application 110. Optionally, client computing device 11 may cause suggestions to be rendered as an overlay element (e.g., an HTML overlay element) on top of a user interface of client computing device 11, or as an overlay element on top of an interface of content access application 110. As a non-limiting example, the overlay element may be positioned based on a location of target content displayed at a user interface of client computing device 11.

[0072] Optionally, after receiving a voice command or spoken utterance from the user selecting a suggestion for a third-party action (i.e., “add milk to shoppingapp A”), the suggestion may disappear from the user interface of the client computing device 11. Optionally, in response to the suggestion for “add milk to shoppingapp A” disappearing after the user selecting it, a notification indicating that milk has been added to the shopping cart of shopping app A may be rendered to the user via the user interface, such as a message and / or symbol in text, graphic, or audio form. Optionally, after rendering and selecting the suggestion for “add milk to shopping app A”, shopping app A (a third-party application) may determine that one or more parameters of the third-party action (i.e., adding milk to shopping app A), such as the quantity and / or brand of milk to be added to the shopping cart, are missing. In this case, a selectable element for additional user input may be rendered to the user at the user interface, the additional user input implementing the third-party action of adding a certain quantity (e.g., half a gallon) of milk from a certain brand to the shopping cart of shopping app A.

[0073] For example, Figure 1A Automated Assistant 112 or Figure 1BThe automated assistant component 1101 in the embodiment can communicate with a cloud-based counterpart (i.e., a cloud-based automated assistant 19 that provides services in a cloud computing environment). In this case, the aforementioned environment becomes a cloud computing environment in which multiple computing devices, which may be in the order of hundreds or thousands or more, share resources through one or more networks 15. In some implementations, the cloud-based automated assistant 19 can be installed at a server computing device 13 or otherwise accessed by the server computing device to perform one or more actions that the server computing device 13 is described as being performed in the present disclosure.

[0074] The cloud-based automated assistant 19 may include, for example, an automatic speech recognition (ASR) engine 191, a natural language understanding (NLU) engine 193, and a text-to-speech (TTS) engine 195. The ASR engine 191 may process audio data that captures spoken utterances to generate recognition of spoken utterances. The NLU engine 193 may determine the semantic meaning of the audio and / or text converted from the audio by the ASR engine, and decompose the determined semantic meaning to determine the intent and / or parameters of the assistant action. For example, the NLU engine 193 may determine the intent and / or parameters of the assistant action based on the aforementioned recognition of the spoken utterances generated by the ASR engine 191.

[0075] In some implementations, the NLU engine 193 can resolve intent and / or parameters based on a single utterance of the user, and in other cases, prompts can be generated based on unresolved intent and / or parameters, those prompts rendered to the user, and user responses to those prompts utilized by the NLU engine 193 in resolving intent and / or parameters. In those cases, the NLU engine 193 can optionally work in conjunction with a dialog manager engine (not shown) that determines unresolved intent and / or parameters and / or generates corresponding prompts. The NLU engine 193 can utilize one or more NLU machine learning models to determine intent and / or parameters.

[0076] The TTS engine 195 can convert text into synthesized speech, and can rely on one or more speech synthesis neural network models to do so. For example, the TTS engine 195 can be used to convert a text response into audio data, which includes a synthesized version of the text, and the synthesized version can be audibly rendered via a hardware speaker of the client computing device 11 or another device.

[0077] As described above, the client computing device 11 may include other applications 112. For example, the other applications 112 may include social media applications, media players, and / or third-party applications installed at the client computing device 11, such as a shopping appA or a recording application.

[0078] In various implementations, the server computing device 13 may receive document information of a document that is launched or at least partially displayed at a user interface of the client computing device 11 from the client computing device 11. The server computing device 13 may determine whether a portion of the document displayed at the user interface of the client computing device 11 includes a trigger portion (e.g., target content) based on the document information. When it is determined that the portion of the document displayed at the user interface of the client computing device 11 includes a trigger portion, the server computing device 13 may further determine coordinate information of the trigger portion in the document at the user interface based on the document information and based on the user interface information, wherein at least a portion of the document (not necessarily including the trigger portion) is rendered / displayed at the user interface of the client computing device 11.

[0079] It is noted that, as described above, client computing device 11 may determine that the portion of the document displayed at the user interface of client computing device 11 includes a trigger portion. In this case, client computing device 11 may notify server computing device 13 by sending a notification message or sending an indication signal indicating that the portion of the document displayed at the user interface of client computing device 11 includes a trigger portion, and server computing device 13 may or may not need to detect whether the rest of the document includes an additional trigger portion. Alternatively, client computing device 11 may determine that the portion of the document displayed at the user interface of client computing device 11 does not include a trigger portion. In this case, server computing device 13 may detect whether the rest of the document includes a trigger portion, for example, by accessing a markup language or DOM of the entire document.

[0080] The aforementioned document information may include, for example, an address (e.g., URI, URL, or local path) of a document (e.g., a web page) so that the server computing device 13 can access the document and / or index the document. For example, the server computing device 13 may access the web page using the URL address of the web page, and index the web page using the original HTML of the web page and / or metadata extracted from the web page. The user interface information may include, for example, at least the width of the user interface of the client computing device 11, the height of the user interface of the client computing device 11, and / or the font size of the document displayed at the user interface of the client computing device 11.

[0081] In some implementations, the server computing device 13 may, for example, include the aforementioned target content detection engine 111 to access a document object model (DOM) of an entire document (e.g., a web page), wherein the DOM uses a structured data tree with multiple nodes to display the document. The multiple nodes in the structured data tree may include: a root node, one or more element nodes, one or more text nodes, and one or more attribute nodes. In this example, the server computing device 13 may determine whether the multiple nodes include a target node representing the target content to detect whether the document includes any target content. If the server computing device 13 determines that the multiple nodes include a target node representing the target content, the server computing device 13 may determine that the document includes the target content.

[0082] As described above, the server computing device 13 can determine the coordinate information of the target content in the document rendered or to be rendered at the user interface of the client computing device 11 based on the document information and based on the user interface information. For example, the server computing device 13 can determine the distance from the target content to the upper edge of the document and / or the document length from the upper edge to the lower edge of the document, and then, if the target content is not displayed when the document is loaded via the content access application 110, the percentage of the user scrolling down (or up) to see the target content is calculated. In some cases, the server computing device 13 can alternatively or additionally determine the distance from the target content to the left edge of the document and / or the document width from the left edge to the right edge of the document. Based on the distance from the target content to the left edge of the document and the width of the document, the server computing device 13 can determine that the user needs to scroll right (or left) to see the additional percentage of the target content.

[0083] Alternatively, server computing device 13 may determine a first percentage range in which a user may scroll up or down to see at least a portion of the target content. Alternatively, server computing device 13 may determine a second percentage range in which a user may scroll left or right to see at least a portion of the target content.

[0084] In various implementations, the server computing device 13 (or sometimes the client computing device 11) may determine whether and / or when to trigger a suggestion (e.g., the aforementioned first-party action or third-party action) related to the target content (or other triggering portion) based at least on the coordinate information of the target content (or based on whether a target content coordinate condition as part of the coordinate condition is satisfied). For example, when the server computing device 13 does not detect the coordinate information of the triggering portion (indicating that the document does not include the triggering portion), the server computing device 13 may determine that the suggestion does not need to be triggered.

[0085] As another example, when the coordinate information of the target content indicates that the user will need to scroll down a first percentage of, for example, approximately 35% of a document rendered at the user interface of the client computing device 11, in order to see the target content of the document, the server computing device 13 may trigger a suggestion that may be generated based on the target content in response to the user scrolling down the document by approximately 35% (or in the range of approximately 33% to 37%). Alternatively or additionally, in addition to whether the target content coordinate condition is met, the server computing device 13 (or sometimes the client computing device 11) may also determine whether and / or when to trigger a suggestion related to the target content based on historical usage data of the document. For example, the historical usage data may include a click-through rate of the suggestion, and when the click-through rate exceeds a threshold (or a predetermined rate), the suggestion may be triggered or rendered in response to the user scrolling down, for example, approximately 35% of the document in the previous example. When the click-through rate does not exceed the threshold, the suggestion will not be triggered (or rendered) in response to the user scrolling down, for example, approximately 35% of the document.

[0086] Alternatively or additionally, the server computing device 13 (or the client computing device 11) may access historical user data, such as the time period from document loading to user interaction with suggestions (e.g., click) for one or more users accessing the document, and / or a time period distribution generated using the time periods collected for one or more users ("historical distribution of selection times" or "historical statistical distribution of selection times"). The server computing device 13 (or the client computing device 11) may determine when to trigger suggestions based on the time periods (or historical distribution of selection times) for one or more users. For example, when the historical user data indicates that historical users of a document spend an average of about 2 minutes reading the document before interacting with suggestions related to the target content of the document (e.g., clicking on the suggestions), the server computing device 13 (or the client computing device 11) may trigger rendering suggestions about 2 minutes after the document is loaded or opened by the content access application 110 at the user interface of the client computing device 11.

[0087] As another example, when historical user data indicates that 90% of historical users of a document took approximately 50 to 80 seconds to read the document before clicking on a suggestion related to the target content of the document, the server computing device 13 (or client computing device 11) may trigger rendering the suggestion approximately 50 seconds after the document is loaded or opened by the content access application 110 at the user interface of the client computing device 11. Such rendering may end approximately 80 seconds after the document is loaded or opened by the content access application 110. Suggestions may be determined / generated, for example, using factors such as the device type of the client computing device 11, the type of third-party application used to perform the third-party action suggested by the suggestion, the device location of the client computing device 11 or the third-party device used to perform the third-party action, and / or the user location of the user of the client computing device 11.

[0088] Alternatively, the server computing device 13 (or the client computing device 11) may determine when to trigger a suggestion based on the location of the target content detected in the document displayed at the user interface of the client computing device 11, by parsing the document content such as the HTML content of the web page to predict the shortest browsing time for the user to reach the target content. For example, the server computing device 13 or the client computing device 11 may parse the HTML content of the web page and statistically or using one or more models predict that the user needs a minimum browsing time of about 45 seconds to reach the target content. In this example, the server computing device 13 or the client computing device 11 may cause a suggestion related to the target content about 45 seconds after the web page is loaded or opened at the client computing device 11.

[0089] Alternatively, the server computing device 13 (or the client computing device 11) may monitor one or more user actions, such as whether the user has stopped scrolling the document for more than a predetermined time period. If the user has stopped scrolling the document for more than a predetermined time period (e.g., 15 seconds), the server computing device 13 (or the client computing device 11) may determine that the portion of the document currently displayed at the user interface of the client computing device 11 includes content of interest to the user. In this case, the server computing device 13 (or the client computing device 11) may cause rendering suggestions in response to the time period during which the user stopped scrolling the document reaching or exceeding the predetermined time period. The suggestions herein may be generated based on the target content or content of interest to the user.

[0090] Figure 2A , Figure 2B , Figure 2C , Figure 2D and Figure 2E Describes the various implementations Figure 1A (or Figure 1B ) is a non-limiting example of a user interface. Figure 2A A user interface 230 of a client computing device 210 is shown, wherein the client computing device 210 can open a content access application such as a web browser in response to a user of the client computing device 210 selecting (e.g., clicking) an icon representing a web browser displayed at the interface of the client computing device 210. Figure 2A In the non-limiting example shown, a user may use a web browser to search for recipes using keywords such as “Kentucky bread pudding with bourbon” and select to open a web page 220 having a URL address “www.domain-name.com / bread-pudding-with-kentucky-bourbon” from a list of web pages (not shown) returned as search results by the web browser's search engine, which is at least partially shown at a URL field 230a of a user interface 230 of a client computing device 210.

[0091] In this example, the user interface 230 may show only a portion of the web page 220, and the user may need to use one or more scroll bars 230b to view different portions of the web page 220. The one or more scroll bars 230b may include, for example, a vertical scroll bar that allows the user to scroll up or down the web page 220 and / or a horizontal scroll bar that allows the user to scroll left or right the web page 220, thereby allowing the user to read the entire web page 220.

[0092] In this example, a web page 220 showing a recipe for "Kentucky bread pudding with bourbon" may include one or more text portions such as a first text portion 2201a and a second text portion 2201b and one or more multimedia content portions such as an image portion 2205 showing an image of a ready-to-eat "Kentucky bread pudding with bourbon." In this case, the first text portion 2201a may include a title of the web page 220 (e.g., Figure 2A ), the subtitle of web page 220 (e.g., Figure 2AThe recipe provider's information (e.g., name and contact information) may include a portion of the recipe portion 2201b (e.g., preparation time: 30 min…save the recipe | share the recipe…Ingredients), and / or information of the recipe provider (e.g., name and contact information), wherein the recipe provider's information may include contact information in a hyperlink that points to access to an email application (e.g., Gmail). The user interface 230 of the client computing device 210 (subject to limitations of the size of the user interface 230, which may depend on the type of client computing device 210) may display only a portion of the second text portion 2201b (e.g., preparation time: 30 min…save the recipe | share the recipe…Ingredients), and Figure 2A In the scenario shown, the user may need to scroll down the scroll bar 230b to read the specific ingredients needed to prepare "Kentucky bread pudding with bourbon".

[0093] In this example, the client computing device 210 may detect whether the portion of the web page 220 displayed via the user interface 230 of the client computing device 210 (i.e., the portion of the web page 220 displayed on the foreground screen of the client computing device 210) includes a trigger portion for triggering a suggestion. The trigger portion may be target content such as recipe content or ingredient content. Alternatively or additionally, the trigger portion may be other content determined based on historical user data of the web page 220 (e.g., such as Figure 2B The indicated cooking instructions section, or the conclusion / summary section at the end of the web page 220 (not shown in the figure), the historical user data indicates that historical users of the web page 220 tended (e.g., 50% of the historical users) to select the suggestion when the other content was rendered for viewing by the historical users.

[0094] Detection of the triggering portion may be accomplished by the client computing device 210 by detecting, for example, some type of pattern markup (e.g., HTML tags) that represents the target content or other content determined based on historical user data. Optionally, it should be noted that instead of the client computing device 210 detecting a portion of the web page 220, a server computing device (such as Figure 1BThe server computing device 13 in the embodiment can access the entire web page 220 and determine whether the web page 220 includes any triggering part. This can be achieved, for example, by using the server computing device to access the DOM of the web page 220 and detecting whether the DOM of the web page 220 includes a target node (or an interest node representing other content) that presents the target content.

[0095] refer to Figure 2A , after processing the portion of the web page 220 shown at the user interface 230 of the client computing device 210, the client computing device 210 may not detect a triggering portion because the target content (e.g., a specific ingredient) is not listed (or identified) at the user interface 230 and / or because other content determined from historical user data is not present at the user interface 230. In this case, the client computing device 210 does not generate a suggestion (or selectable element) to render at the user interface 230 because the triggering portion is not detected.

[0096] refer to Figure 2B , after the user scrolls the scroll bar 230b downward, a different portion of the web page 220 is displayed via the user interface 230 of the client computing device 210. In this case, the client computing device may detect that the different portion of the web page 220 currently displayed at the user interface 230 includes a trigger portion, such as target content (e.g., “6 slices bread … 1 tbsp Bourbon sauce”). In response to detecting that the different portion of the web page 220 includes the target content as a trigger portion, the client computing device 210 may cause suggestions to be rendered via the user interface 230 of the client computing device 210, such as first-party actions that may be performed by the automated assistant, or third-party actions that may be performed by a third-party application other than the automated assistant. The suggestions may be rendered, for example, at a top bar of the user interface 230, at a bottom bar of the user interface 230, or as an overlay element (e.g., an HTML overlay element) above the web page 220.

[0097] refer to Figure 2CAs a non-limiting example, the aforementioned suggestion may be a suggestion 240A that pops up on the web page 220. In this example, the client computing device 210 detects that the user interface 230 includes target content, such as "ingredients: 6 slices of bread...1 tbsp Bourbon sauce", and in response, causes rendering of a suggestion 240A generated based on the target content. The suggestion 240A may, for example, include natural language content such as "Add ingredient X to shopping app A", where ingredient X may be one of the ingredients listed in the recipe shown on the web page 220 and recommended to be purchased by the user based on the user data, and where shopping app A may be a third-party application. The user data may include, for example, historical user data showing interactions between the current user (and / or other users, “historical users”) and the client computing device 210 and / or the web page 220, user preferences collected from the current user and other users, and / or the user's personal user data for content access applications and other applications, such as a camera, email, a recording application that records the lack of ingredient milk, or a messaging application with a message expressing an intention to purchase milk, and the present disclosure is not limited thereto. For example, based on the historical user data of the web page 220 (such as the historical user of the web page 220 frequently ordering milk (or a certain milk product) using the suggestions rendered on the web page 220), a suggestion 240A may be popped up to recommend “milk” as an ingredient to be added to the shopping app A (or recommend a certain milk product).

[0098] Optionally, the client computing device 210 may detect that the user interface 230 includes a plurality of trigger parts, the plurality of trigger parts including, for example, a first trigger part and a second trigger part. Figure 2B, the client computing device 210 may detect “6 slices of bread…1 tbsp Bourbon sauce” as a first trigger portion, and detect “¾ cup brown sugar…3 tbsp Bourbon whisky” as a second trigger portion. The first target content and the second target content may be of the same type or of different types. For example, the first trigger portion may be target content, such as an ingredients section that lists ingredients such as “6 slices of bread…1 tbsp Bourbon sauce”, and the second trigger portion may be other content, such as comments by other users in a comments section, or one or more images of a main dish / food cooked using the listed ingredients in a cooking instructions section or a conclusion section, whether a semi-finished product image or a finished product image.

[0099] Optionally, the suggestion 240A may be generated and / or triggered by the server computing device based on a number of factors, including click-through rate (historical and / or predicted), the type of client computing device 210, the user's location, the availability of a third-party application (or third-party device) that implements the third-party action suggested by the suggestion 240A, etc. For example, in addition to being based on the target content of the web page 220 (e.g., the recipe content), the suggestion 240A suggesting "Add ingredient X to shopping app A" may also be generated as follows: based on the third-party application (i.e., shopping app A) being available at the client computing device 210, where the login information of the user currently using the client computing device 210 is accessible or saved; based on the services of shopping app A covering the delivery address selected by the user; and / or based on the number of times in history when the user saw such a suggestion 240A, ordering milk exceeded a certain percentage. It is noted that suggestion 240A may also suggest a first-party action that may be performed via the automated assistant, such as showing in a pop-up window “copy the list of ingredients to my notebook app” or “take a screenshot photo and save it in Photos” or simply “copy the list of ingredients” and having the automated assistant determine subsequent actions or steps, or recommending subsequent actions to the user for additional user input. For example, the automated assistant may perform the first-party action and then ask the user “The list of ingredients copied. Where do you want to paste it?” In this example, to receive additional user input, the automated assistant may display options, such as a first selectable icon representing a messaging application and a second selectable icon representing a note-taking application.

[0100] Optionally, suggestion 240A may be audibly rendered via the automated assistant to the user of client computing device 210. When suggestion 240A suggests a third-party action (“add ingredient X”) that may be performed by a third-party application (“shopping app A”), such as “Add ingredient X to shopping app A”, the automated assistant may be configured to receive user input confirming an intent to add ingredient X to shopping app A. In this case, the automated assistant may determine that one or more parameters of the third-party action (amount or brand of ingredient X, etc.) are missing, and additional user input is required from the user providing values ​​for the one or more parameters.

[0101] In some implementations, reference Figure 2D , the aforementioned suggestion may be a suggestion 240B displayed at the bottom bar of the user interface 230. The suggestion 240B may include a first natural language content, such as "Add milk to app A shoppingcart". Alternatively or additionally, the suggestion 240B may include product information, such as an item image, brand, size, and price for each product (e.g., milk) listed in the suggestion 240B. For example, suggestion 240B may list a first product Milk X1 with or without its price and size (e.g., half a gallon of milk from brand A at a price of $7.49), a second product Milk X2 with or without its price and size (e.g., 59 fluid ounces of milk from brand B at a price of $4.99), a third product Milk X3 with or without its price and size (e.g., 1 gallon of milk from brand C at a price of $5.99), and a fourth product Milk X4 with or without its price and size (e.g., 64 fluid ounces of milk from brand D at a price of $3.69), where the first, second, third, and fourth products Milk X1 are of the same type (e.g., milk).

[0102] Optionally, the item image of each of the first, second, third, and fourth products 2401B to 2407B may be configured to be selectable so that the user can select the corresponding product via a user click, touch input, audio input, or other means. Alternatively or additionally, the suggestion 240B may include a second natural language content, such as "see more options at store XX", where the second natural language content may be selectable, and when selected, an interface of app A showing items sold by store XX (of the same type as the first, second, third, and fourth products Milk X1 to X4) may be rendered via the user interface 230 of the client computing device 210. Alternatively or additionally, the suggestion 240B may include a third natural language content, which is selectable, and when selected, an interface of app A showing all stores available for shopping may be rendered to the user via the user interface 230.

[0103] In some implementations, reference Figure 2E , the aforementioned suggestion may be a suggestion 240C displayed at the bottom bar of the user interface 230. The suggestion 240C may include a first natural language content such as “Add to your app A shopping cart” and product information such as item images, brands, sizes, and prices of different ingredients listed in the recipe of the web page 220. For example, the suggestion 240C may list a first product with its price and size (e.g., 24 ounces of bread, which costs $2.79), a second product with its price and size (e.g., 0.5 gallons of milk, which costs $5.49), and a third product with its price and size (e.g., half a dozen eggs, which costs $3.49), wherein the first, second, and third products are of different types.

[0104] Optionally, the item image of each of the first, second, and third products (e.g., bread, milk, eggs) may be configured to be selectable so that the user can select the corresponding product via a user click, touch input, audio input, or other means. Optionally, the suggestion 240C may include a selectable symbol 241 that indicates that there are more products to purchase from store XX to prepare the ingredients listed in the recipe. Optionally or additionally, the suggestion 240C may include a second natural language content, such as "enter store XX", where the second natural language content may be selectable, and when selected, an interface of the shopping app A showing items sold by store XX may be rendered via the user interface 230 of the client computing device 210. Optionally or additionally, the suggestion 240C may include a third natural language content that is selectable, and when selected, an interface of the shopping app A showing all stores available for shopping may be rendered to the user via the user interface 230.

[0105] Alternatively, suggestions such as suggestion 240A, suggestion 240B, or suggestion 240C may be generated by the client computing device 210, or by one or more server computing devices in communication with the client computing device 210. In some implementations, the one or more server computing devices may generate a plurality of suggestions based on the target content of the web page 220, rank the plurality of suggestions, and send the highest ranked suggestions to the client computing device 210 for rendering to the user.

[0106] In some implementations, where a suggestion such as suggestion 240A (e.g., “add milk to shopping app A”) is audibly rendered to a user via an automated assistant application, the client computing device 210 may determine, prior to generating such a suggestion and via the automated assistant application, whether schema markup is for a document (i.e., whether the document is a web page), whether the aforementioned app A is installed at the client computing device 210 (or otherwise accessible), and / or whether the user is logged into an account of app A. In this case, the schema markup may be applied to determine whether the web page 220 includes any triggering portion (e.g., a recipe), and if a tag corresponding to a recipe is detected from the schema markup, it may be determined that the web page 220 includes a recipe as target content, such that a suggestion such as “add milk to shopping app A” may be generated and audibly rendered to the user. The user can respond to the suggestion by saying "add milk to store XX on shopping app A", and continue the conversation with the automated assistant via audio input such as "what's the status of my shopping app A order? ", "take me to the shopping app a cart", "find butter from store XX on shopping app A", "let's checkout on shopping app A", etc.

[0107] In various implementations, instead of using the client computing device 210 to process the user interface 230 that displays a portion of the web page 220, the client computing device 210 may send screen information ("user interface information") of the user interface 230 of the client computing device 210 to the server computing device. A more detailed description may be found elsewhere in this disclosure.

[0108] Optionally or alternatively, the client computing device 210 may detect that the aforementioned different parts of the web page 220 include multiple trigger parts, such as a first trigger part and a second trigger part. Figure 2B, the client computing device 210 may detect "6 slices of bread...1 tbsp Bourbon sauce" as a first trigger portion, and detect "¾ cup brown sugar...3 tbsp Bourbon whisky" as a second trigger portion. The first target content and the second target content may be of the same type (e.g., both are target content) or of different types (e.g., one is target content and the other is other content, or one is natural language content and the other is image content).

[0109] In some implementations, if one or more conditions are met, the client computing device 210 may detect whether a portion of a document currently displayed at the user interface 230 of the client computing device 210 includes a trigger portion (or target content). The one or more conditions may include, for example, a first condition indicating whether a document is loaded at the user interface 230, a second condition indicating whether a user scrolls down a vertical scroll bar, a third condition indicating whether a user scrolls up a vertical scroll bar, a fourth condition indicating whether a user scrolls left a horizontal scroll bar, and / or a fifth condition indicating whether a user scrolls right a horizontal scroll bar. However, the present disclosure is not limited thereto, and other conditions may be used to trigger detection of target content when appropriate.

[0110] It is noted that instead of having the client computing device 210 detect the triggering portion and determine when to render the suggestion based on the triggering portion, the client computing device 210 may send at least document information (e.g., web page 220) of a document at least partially displayed at the user interface 230 of the client computing device 210 to the server computing device (e.g., server computing device 13 in FIG. 1 ) for the server computing device to detect the triggering portion and determine when to render the suggestion. For example, in response to determining that the web page 220 is loaded via a web browser, the client computing device 210 may send the document information of the web page 220 and / or the user interface information of the user interface 230 of the client computing device 210 to the server computing device (e.g., server computing device 13 in FIG. 1 ).

[0111] The document information may include, for example, a URL address (www.domain-name.com / bread-pudding-with-kentucky-bourbon) of the web page 220 for the server computing device to access the content of the web page 220. As a non-limiting example, the user interface information of the user interface 230 of the client computing device 210 may include the size (width and height) and / or font size of the user interface 230. Alternatively, the size of the user interface 230 may be measured by multiplying the length of the user interface 230 in pixels by the width of the web page 220 in pixels. In various implementations, the size of the web page 220 may vary when the web page 220 is displayed at different user interfaces of different client computing devices.

[0112] The server computing device may process the document information after receiving the document information of the document from the client computing device 210 to determine whether the content of the web page 220, for example, includes a triggering portion. As a non-limiting example, when the web page 220 displayed on the client computing device 210 is about a recipe, the server computing device may retrieve the URL of the web page 220, access the content of the web page 220 using the URL, and process the content of the web page 220 to list the ingredients listed under the subheading “Ingredients” (e.g., Figure 2B As another non-limiting example, the server computing device may identify as the target content "6 slices of bread...1 tbsp Bourbon sauce" in the recipe. As another non-limiting example, the server computing device may identify as the target content a recipe content including ingredients, cooking instructions, and a story describing the creation of the recipe.

[0113] Optionally, the server computing device may identify a plurality of target contents, including at least a first target content and a second target content. Figure 2B , the server computing device may process the content of the web page 220 to identify "6 slice bread...1 tbsp Bourbon sauce" as the first target content, and "¾ cup brown sugar...3 tbsp Bourbon whisky" as the second target content. Optionally, the first target content and the second target content may be of different types.

[0114] In some implementations, the server computing device may access the document and index the document to determine whether the document includes any target content. It is noted that when indexing a document such as a web page 220, the server computing device may use the original HTML of the web page 220 and / or metadata extracted from the web page 220. In some implementations, the server computing device may generate and send an indication signal or message to the client computing device 210 indicating whether the document for which the document information sent by the client computing device 210 to the server computing device is directed includes any target content.

[0115] In various implementations, the server computing device may determine, based on the document information, coordinate information of target content in a document (e.g., web page 220) displayed at a user interface (e.g., user interface 230) of the client computing device 210. The server computing device may cause a determination, based on the coordinate information of the target content, to trigger a suggestion related to a document whose document information is received by the server computing device from the client computing device 210.

[0116] In various implementations, the server computing device may determine whether the web page 220 includes the target content, and the client computing device 210 may determine when the target content is rendered at the user interface 230 of the client computing device 210, thereby triggering the suggestion. For example, the client computing device 210 may send the URL of the web page 220 to the server computing device, wherein the server computing device may detect whether a DOM node in the DOM of the web page 220 corresponds to the target content (or other triggering portion), and send a detection signal or message indicating whether the web page 220 includes the triggering portion to the client computing device 210. If the server computing device detects a DOM node representing the triggering portion in the DOM of the web page 220, the server computing device may send information of the detected DOM node to the client computing device 210 together with the detection signal (or message).

[0117] After receiving the detection signal or detection message from the server computing device, the client computing device 210 may determine whether a DOM node (if any) detected by the server computing device is being executed when the current user interface 230 of the client computing device 210 displays a portion of the document. If the client computing device 210 determines that the node representing the target content detected by the server computing device is being executed when the user interface 230 of the client computing device 210 displays a portion of the document, the client computing device 210 may determine that the DOM condition is satisfied (i.e., the portion of the document currently displayed includes a triggering portion), and in response to satisfying the DOM condition, cause one or more suggestions to be rendered via the user interface 230 of the client computing device (the one or more suggestions may or may not be associated with the triggering portion).

[0118] In some implementations, when a document is loaded at an application, the client computing device 210 or the server computing device may collect user historical data including measured time periods for one or more users to select or otherwise interact with a suggestion, such as suggestion 240B. The user historical data may be included in document information for the document, and may include, in addition to the measured time periods, the content of the suggestions, the location of the suggestions, and the location or content of the triggering portion in the document (where the suggestions are triggered for display based at least on the triggering portion being displayed) for each measured time period. Alternatively or in addition, the historical user data may further include a distribution of measured time periods for one or more users, and / or statistics of the content or location of the suggestions and the content or location of the corresponding triggering portion.

[0119] Based on the collected historical user data, the client computing device 210 (or the server computing device) can determine when one or more trigger conditions (e.g., the aforementioned coordinate conditions, DOM conditions, time conditions, or attention tracking conditions, etc.) are met, so as to render one or more suggestions (e.g., suggestion 240B) to the current user. For example, if it is determined that it takes about 2 minutes on average from the time when the web page 220 is loaded to the time when different users (e.g., ⅔ of the historical users of the document) click or otherwise select the suggestion 240B, the client computing device 210 (or the server computing device) can render the suggestion 240B to the current user about 2 minutes after the web page 220 is loaded at the web browser.

[0120] In some other implementations, the client computing device 210 or the server computing device may parse the HTML of the web page 220 and, based on the parsing, predict the shortest browsing time it takes for the user to reach the target content. In this case, the client computing device 210 (or the server computing device) may cause the suggestions 240B to be rendered to the current user approximately when the shortest browsing time has elapsed since the web page 220 was loaded to the current user via the user interface 230.

[0121] In some other implementations, the client computing device 210 or the server computing device may determine when to render the suggestions based on the one or more navigation gesture signals. For example, the one or more navigation gesture signals may include a first navigation gesture signal that indicates that there has been no user action for a predetermined time while the document is open in the user interface (e.g., the user has stopped scrolling for at least a predetermined time, such as t milliseconds). In response to detecting the first navigation gesture signal, the client computing device 210 (or the server computing device) may cause the suggestions 240B to be rendered to the current user once it is detected that the user has stopped scrolling for a predetermined time, because in this case, stopping scrolling for at least the predetermined time indicates a high level of user interest / attention (i.e., the attention tracking condition is satisfied).

[0122] In some other implementations, in addition to identifying the trigger portion, the client computing device 210 or the server computing device may also determine whether to render the suggestion based on historical user data such as historical user click-through rates on the suggestions. For example, after the client computing device 210 (or the server computing device) detects that the web page 220 includes the trigger portion, and before rendering one or more suggestions, the client computing device 210 (or the server computing device) may determine that the one or more suggestions have a click-through rate that is below a predetermined threshold for rendering any suggestions. In this example, due to the low click-through rate, the one or more suggestions will not be rendered to the user.

[0123] Figure 3 is a flow chart illustrating an example method 300 for triggering a recommendation for a content access application according to various implementations. Figure 3As shown, method 300 includes: at box 310, the client computing device detects whether the user interface of the client computing device displays a document (e.g., a web page) in the foreground. In some implementations, the document (e.g., a web page) can be displayed at the user interface of the client computing device by a content access application such as a web browser. Depending on the size of the document at the user interface, the document can be displayed completely at the user interface, or partially at the user interface. In some implementations, the user interface of the client computing device can display multiple interfaces, where the multiple interfaces are from the same application (e.g., a content access application) or from different applications. In these cases, the client computing device can detect whether the user interface includes a document displayed by an application running in the foreground.

[0124] The method 300 may further include determining, by the client computing device, whether the first portion of the document displayed at the user interface includes the target content, at block 312. For example, the client computing device may detect whether certain types of mode tags for the target content are present / executed in the foreground of the client computing device. If the expected type of mode tags are present / executed in the foreground, the client computing device determines that the first portion of the document displayed at the user interface includes the target content.

[0125] The method 300 may further include, at box 314, in response to determining that the first portion of the document includes target content, causing, by the client computing device, rendering a suggestion at a user interface of the client computing device. In some implementations, the suggestion may be preemptively generated for the target content. In some implementations, the suggestion may be a selectable symbol and / or selectable natural language content suggesting a third-party action that may be performed by a third-party application such as shopping app A. Optionally, the suggestion may be rendered at a top bar, at a bottom bar, or as an overlay element at a user interface of the client computing device. Optionally, more than one suggestion may be rendered at the user interface.

[0126] Figure 4 4 is a flow chart illustrating another example method 400 for triggering a recommendation for a content access application according to various implementations. Figure 4 As shown, method 400 may include: at block 410, the client computing device sends document information and user interface information to the server computing device. The document information may include, for example, an address for retrieving the document, historical user data of the document, and / or the content of the document (e.g., in the format of an image or a markup language). The user interface information may include, for example, the size of the user interface of the client computing device for displaying the document and / or the font size of the natural language content in the document displayed at the user interface of the client computing device.

[0127] In various implementations, method 400 may further include: at block 412, receiving, by the client computing device, from the server computing device, a first signal or (a first message) indicating that the document includes a trigger portion (e.g., target content) and a suggestion generated based on the target content. For example, the first signal or message may be generated by the server computing device and may indicate that the document includes the target content. Optionally, the suggestion may be a suggestion for a first-party action that may be performed by the automated assistant. For example, for a web page describing an upcoming sports game, a description of a specific date (e.g., August 8, 2028, at 8:00 a.m.) or a data range ticket that will be sold for the upcoming sports game in the web page may be determined as a trigger portion or target content, and a suggestion for performing a first-party action (e.g., adding a reminder to purchase three tickets for the upcoming sports game) may be rendered to the user. When the user selects or confirms the first-party action, the automated assistant may perform the first-party action, for example, by adding a reminder to purchase three tickets in a calendar application. Optionally, the suggestion may be a suggestion for a third-party action that may be performed by a third-party application, and a repeated description is omitted herein. Alternatively, the suggestions may be generated based on the target content and / or based on device information of the client computing device. The device information of the client computing device may include whether the third-party application is installed at the client computing device, whether the user of the client computing device has set up and / or logged into the third-party application when reading the document, the location of the client computing device (indicating whether the service provided by the third-party application is available for that location), and / or other information.

[0128] In various implementations, method 400 may further include: at block 414, receiving, by the client computing device, from the server computing device, a second signal (or a second message) for triggering the suggestion. For example, the server computing device may determine one or more trigger conditions for triggering the suggestion, and determine whether any of the one or more trigger conditions for the suggestion is satisfied. In response to determining that a trigger condition in the one or more trigger conditions is satisfied, the server computing device may generate a second signal or a second message, and send the second signal or the second message to the client computing device for triggering rendering of the suggestion.

[0129] As a non-limiting example, the trigger condition may be a time condition indicating whether a first period of time has elapsed since the document was loaded (e.g., by a content access application such as a web browser). In this example, the first period of time may be determined based on a historical distribution of selection times taken by historical users / readers of the document to select suggestions to be rendered for triggering portions of the document after the document was loaded. In this example, the trigger / time condition is determined to be satisfied when the first period of time has elapsed since the document was loaded. In other words, the suggestion may be triggered or rendered after the first period of time has elapsed since the document was loaded.

[0130] As another non-limiting example, the trigger condition may indicate whether the target content is displayed in the foreground at the user interface of the client computing device. In this example, when the target content is detected as being displayed in the foreground at the user interface of the client computing device, it can be determined that the trigger condition is satisfied, so that the suggestion is triggered or rendered at the user interface. In some implementations, whether the target content is detected as being displayed in the foreground can be based on the coordinate information of the target content. For example, when the user interface of the client computing device is configured to display approximately 20% of the document at a given moment, the user will not see the target content with coordinate information indicating the following situation: the target content is located at approximately 35% of the document. In this example, the user will need to scroll down another 15% or more of the document to see the target content. When the user scrolls down another 15% or more (say 20%, when the entire target content is displayed at the top of the user interface of the client computing device), the suggestion can be triggered or rendered to the user. Optionally, one or more distances measured by pixels or inches can be used instead of percentages to represent the coordinate information, wherein the relevant description of the coordinate information can be found in other parts of this disclosure, and the repeated description is omitted here.

[0131] Alternatively or additionally, as a further non-limiting example, the trigger condition may indicate whether a navigation gesture signal is detected. In this example, a navigation gesture signal may be detected if the user has stopped scrolling for a predetermined period of time, indicating that the portion of the document displayed in the foreground at the user interface of the client computing device includes additional target content or content of interest to the user.

[0132] In various implementations, method 400 may further include causing, by the client computing device, rendering the suggestion at a user interface of the client computing device based on the second signal, at block 416. In some implementations, the second signal may further include a location for rendering the suggestion, such as a top bar, a bottom bar, or an overlay element at the user interface of the client computing device. In these cases, the suggestion may be rendered at the location indicated by the second signal.

[0133] Figure 5 is a flow chart illustrating an additional example method 500 for triggering a suggestion for a content access application based on targeted content of a document displayed by the content access application, according to various implementations. Figure 5 As shown, method 500 may include: at block 510, determining by the client computing device whether a portion of a document displayed in the foreground at a user interface of the client computing device includes target content. Optionally, the client computing device may use any of the aforementioned detection methods to detect whether the portion of the document displayed in the foreground includes target content when appropriate.

[0134] In various implementations, method 500 may include, at block 512, sending, by the client computing device, to the server computing device, a detection signal / message indicating detection of the trigger portion, document information such as the content and / or location of the trigger portion in the document, and user interface information describing the size and font size of a user interface and / or content access application of the client computing device. The document information may be the same or similar to the aforementioned document information, and the user interface information may be the same or similar to the aforementioned user interface information. A repeated description is not provided herein.

[0135] In various implementations, method 500 may include: at block 514, receiving, by the client computing device, from the server computing device, suggestions to be rendered at the user interface and / or one or more triggering conditions to trigger rendering of the suggestions. The suggestions and the one or more triggering conditions may be the same or similar to those described above, and repeated descriptions are omitted herein.

[0136] In various implementations, method 500 may include causing, by the client computing device, rendering a suggestion at a user interface for user interaction based on the one or more trigger conditions, at block 516. For example, in response to determining that a trigger condition in the one or more trigger conditions for triggering a suggestion is satisfied, the client computing device may cause rendering of the suggestion at a user interface of the client computing device. The suggestion may be a selectable element representing a third-party action or a first-party action, and the user may select the suggestion to perform a third-party action, such as adding a product to a shopping cart of shopping app A.

[0137] Figure 6 610 is a block diagram of an example computing device 610 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client computing device, a cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 610.

[0138] The computing device 610 typically includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 including, for example, a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow for user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0139] The user interface input device 622 may include a keyboard, a pointing device (such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen integrated into a display), an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 610 or onto a communication network.

[0140] The user interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 610 to a user or another machine or computing device.

[0141] The storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic to perform selected aspects of the methods disclosed herein and to implement the various components depicted in Figures 1 and 2.

[0142] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 can include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive together with associated removable media, a CD-ROM drive, an optical drive, or a removable media box. Modules that implement the functionality of certain implementations can be stored by the file storage subsystem 626 in the storage subsystem 624, or stored in other machines accessible to the processor 614.

[0143] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0144] The computing device 610 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the Figure 6 The description of computing device 610 depicted in FIG. 6 is intended only as a specific example for demonstrating some implementations. Many other configurations of computing device 610 are possible, and these configurations are similar to those of FIG. Figure 6 The computing devices depicted in FIG. 1 may have more or fewer components than those depicted in FIG.

[0145] Although several implementations have been described and shown herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be used, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or confirm many equivalents of the implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

[0146] In various implementations, a method implemented by one or more processors is provided, the method comprising: sending, for example, by a client computing device to a server computing device: document information identifying a document displayed at a user interface of the client computing device; and user interface information identifying a type and / or size information of the user interface of the client computing device, wherein the type of the user interface of the client computing device can be used to determine the size information of the user interface of the client computing device. The document can be a web page, or other public or private document, such as a PDF file. The method further comprises receiving, for example, by the client computing device from the server computing device and in response to the sending: a suggestion related to the content of the document; and a coordinate condition, wherein the coordinate condition can be generated by the server computing device based on (1) the type and / or size information of the user interface and (2) a triggering portion determined from the document for rendering the suggestion. The method can further comprise, in response to receiving the suggestion and the coordinate condition: (i) monitoring whether the coordinate condition is satisfied in the coordinate information of the triggering portion of the document relative to the user interface when the document is displayed at the user interface; and (ii) in response to determining during the monitoring that the coordinate condition is satisfied: causing the suggestion to be rendered at the user interface.

[0147] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0148] In some implementations, the coordinate condition includes: a first distance from a top edge of a document displayed at the user interface (i.e., the top edge of the user interface) to a trigger portion of the document; and / or a second distance from a side edge of the document displayed at the user interface to the trigger portion of the document. In some versions of those implementations, the coordinate condition includes the first distance, and the first distance is expressed as a first amount of pixels from the top edge of the document.

[0149] In some implementations, the suggestion is a selectable suggestion that, when selected, causes an automated assistant application accessible via a client computing device to perform one or more actions specific to the content of the document. In some versions of those implementations, the one or more actions include causing an additional application of the client computing device to open in a state specific to the content.

[0150] In some implementations, the trigger portion determined from the document for rendering the suggestions may include target content determined by the server computing device based on content in the document. In some other implementations, the trigger portion determined from the document for rendering the suggestions may include a historical portion determined by the server computing device based on historical user data associated with the document. The historical user data may include historical user interaction frequencies for one or more suggestions related to the content of the document, and wherein the historical portion of the document is determined by being a portion of the document that was historically displayed at the user interface when a suggestion having the highest historical user interaction frequency among the one or more suggestions was historically triggered at the user interface.

[0151] In some implementations, the coordinate information changes in response to a user scroll input that scrolls the document to change the content of the document displayed at the user interface. For example, when the document is just loaded at the content access application of the client computing device, the coordinate information of the trigger portion of the document may include a first distance from the upper edge of the user interface of the client computing device that displays the document (which may be a display screen or a portion thereof) to the upper edge (or lower edge, or central portion) of the trigger portion of the document, the first distance being approximately 800 pixels. In this example, the trigger portion may not be displayed to the user of the client computing device, or the user interface or display screen of the client computing device may be shorter than a size of approximately 800 pixels. When the user scrolls the scroll bar of the user interface of the client computing device, the client computing device may detect the user scroll input that scrolls the document down, say, approximately 600 pixels or so. After the user scrolls down approximately 600 pixels, the coordinate information of the trigger portion may include a modified first distance of approximately 200 pixels, which is the measured distance from the upper edge of the display screen to the trigger portion of the document. In this case, when the user interface or display screen of the client computing device is longer than 200 pixels, the triggering portion of the document can be displayed to the user via the user interface for reading.

[0152] In some implementations, different client computing devices may have user interfaces of different sizes, and when a document is displayed at different client computing devices (or when the document is displayed via different content access applications of the same client computing device), the coordinate information of the trigger portion of the document may be different. For example, when the document has just been loaded at the content access application of a first client computing device, the coordinate information of the trigger portion of the document may include a first distance from the top edge of the user interface of the first client computing device displaying the document to the trigger portion of the document, the first distance being approximately 800 pixels. When the document has just been loaded at the content access application of a second client computing device, the coordinate information of the trigger portion of the document may include a first distance from the top edge of the user interface of the second client computing device displaying the document to the trigger portion of the document, the first distance being approximately 900 pixels, where the second client computing device is different from the first client computing device.

[0153] In some implementations, when generating coordinate conditions, the server computing device uses the type and / or size information of the user interface to determine a distance metric at which the trigger portion of the document will be rendered within the user interface, and generates the coordinate conditions based on the distance metric. For example, the distance metric may include a first distance metric calculated by subtracting the length of the user interface of the client computing device from the first distance, the first distance measuring the distance from the top edge of the user interface displaying the document to the bottom edge of the trigger portion of the document. In another example, the distance metric may include a second distance metric calculated by subtracting the length of the user interface of the client computing device from the first distance, the first distance measuring the distance from the top edge of the user interface displaying the document to the top edge of the trigger portion of the document. In a further example, the distance metric may be a distance range that varies from the aforementioned first distance metric to the second distance metric.

[0154] In some implementations, causing suggestions to be rendered at the user interface includes: causing suggestions to be rendered at a bottom bar or a top bar of the user interface; and / or causing suggestions to be rendered via an HTML overlay element. Suggestions may also be rendered at other areas of the user interface, and the present disclosure is not limited thereto.

[0155] In various implementations, a method implemented by one or more processors of a server computing device is provided, and the method includes receiving from a client computing device: document information, the document information identifying a document displayed at a user interface of the client computing device; and user interface information, the user interface information identifying a type and / or size information of the user interface of the client computing device. The method may further include, in response to receiving: identifying suggestions related to the content of the document using the document information; and generating coordinate conditions. The coordinate conditions may be generated based on the type and / or size information of the user interface and based on a triggering portion of the document for rendering the suggestions. The method may further include: sending the suggestions and the coordinate conditions to the client computing device, wherein sending the suggestions and the coordinate conditions causes the client computing device to monitor whether the coordinate conditions are met and render the suggestions when the coordinate conditions are met.

[0156] In some implementations, the suggestions are pre-generated and indexed in an index based on the document information, and identifying the suggestions includes identifying the pre-generated suggestions in the index using the document information. In these implementations, the document information may include a resource identifier of the document. In these implementations, the resource identifier may be a uniform resource locator of the document.

[0157] In some implementations, the document information includes content of the document, and identifying the suggestions includes processing the content to generate the suggestions.

[0158] In various implementations, a method implemented by one or more processors is provided, and the method includes: for a document displayed at a user interface of a client computing device, identifying a historical statistical distribution of selection times, the historical statistical distribution of selection times being generated based on historical selection times of corresponding suggestions related to content of the document and rendered simultaneously with the document. The method may further include: generating a time condition for rendering suggestions related to the content based on the historical statistical distribution. In response to rendering the document at the user interface of the client computing device, the method may further include causing the client computing device to: monitor whether the time condition is met; and when the coordinate condition is met, render the suggestion.

[0159] In some implementations, the historical selection times are each related to a corresponding initial rendering of the document, and wherein the temporal condition is related to the initial rendering of the document. In some implementations, the method may further include: identifying, for the client computing device or for a user account accessing the document at the computing device, a historical metric of the suggested selections, wherein generating the temporal condition is based on the historical statistical distribution and the historical metric of the suggested selections. In these implementations, generating the temporal condition based on the historical statistical distribution and the historical metric of the suggested selections includes: selecting a time from the historical statistical distribution depending on the historical metric of the suggestion.

[0160] In some implementations, generating the time condition based on the historical statistical distribution includes: selecting a time from the historical statistical distribution, wherein the selection of the time depends on a probability defined by the historical statistical distribution; and generating the time condition based on the time. In these implementations, the suggestion is selected at the client computing device after rendering the suggestion, and the method may further include: determining an actual selection time of the selection of the suggestion at the client computing device; and updating the historical statistical distribution based on the actual selection time.

[0161] In some implementations, the historical statistical distribution at the time of selection is all document-specific. In some implementations, the historical statistical distribution at the time of selection may include selecting from a plurality of different documents determined to have one or more characteristics in common, and in response to determining that the document also has the one or more characteristics, utilizing the historical statistical distribution for the document.

[0162] In various implementations, a method implemented by one or more processors of a client computing device is provided, and the method includes: sending to a server computing device: document information, the document information identifying a document displayed at a user interface of the client computing device. The method may further include receiving, in response to sending: a suggestion related to the content of the document; and one or more conditions. The one or more conditions may include the aforementioned document object model (DOM) conditions, time conditions, and / or coordinate conditions. The method may further include, in response to receiving the suggestion and the one or more conditions: (1) when rendering the document in the user interface, monitoring whether at least one of the one or more conditions is satisfied; and (2) in response to determining that at least one of the one or more conditions is satisfied, causing the suggestion to be rendered at the user interface.

[0163] In some implementations, the one or more conditions may include a time condition and a coordinate condition, and in this case, monitoring whether at least one of the one or more conditions is satisfied includes determining whether either the time condition or the coordinate condition is satisfied.

[0164] In various implementations, a method implemented by one or more processors of a client computing device is provided, and the method includes: sending document information to a server computing device, the document information identifying a document displayed at a user interface of the client computing device; and receiving suggestions related to the content of the document, nodes of a document object model (DOM) of the document, and coordinate conditions in response to the sending. The coordinate conditions can be generated by the server computing device based on: type and / or size information of the user interface; and a trigger portion for rendering the suggestion determined from the document. The method may further include: in response to receiving the suggestion and the node of the DOM of the document, when rendering the document in the user interface, monitoring whether a markup language corresponding to the node of the DOM is actively executed. In response to determining that the markup language corresponding to the node of the DOM is actively executed, the method may further include: causing the suggestion to be rendered at the user interface.

[0165] In various implementations, a method implemented by one or more processors is provided, and the method includes: using document information of a document displayed at a user interface of a client computing device, identifying suggestions related to the content of the document. The method may further include: generating a coordinate condition, wherein the generated coordinate condition is based on the type and / or size information of the user interface and is based on a trigger portion of the document for rendering the suggestion. The method may further include, in response to identifying the suggestion and generating the coordinate condition: monitoring whether the coordinate condition is met in the coordinate information of the trigger portion of the document relative to the user interface when the document is displayed at the user interface. In response to determining that the coordinate condition is met during monitoring, the method may further include: causing the suggestion to be rendered at the user interface. In these implementations, the one or more processors are composed of a processor of a client device (or client computing device).

Claims

1. A method implemented using one or more processors of a client computing device, the method comprising: Send to the server computing device: document information identifying a document displayed at a user interface of the client computing device, and User interface information, the user interface information identifying type and / or size information of the user interface of the client computing device; Receiving in response to said sending: suggestions related to the content of said document, and The coordinate conditions generated by the server computing device are based on: the type and / or the size information of the user interface, and a trigger portion for rendering the suggestion determined from the document; as well as In response to receiving the suggestion and the coordinate conditions: monitoring whether the coordinate condition is satisfied in coordinate information of the triggering portion of the document relative to the user interface when the document is displayed at the user interface, and In response to determining during the monitoring that the coordinate condition is satisfied: The suggestion is caused to be rendered at the user interface.

2. The method of claim 1, wherein: The coordinate conditions include: a first distance from a top edge of the document displayed at the user interface to the triggering portion of the document, and / or A second distance from a side edge of the document displayed at the user interface to the triggering portion of the document.

3. The method of claim 2, wherein: The coordinate condition includes the first distance, and the first distance is expressed as a first amount of pixels from the upper edge of the document.

4. A method as claimed in any preceding claim, wherein: The document is a web page.

5. A method as claimed in any preceding claim, wherein: The suggestion is a selectable suggestion that, when selected, causes an automated assistant application accessible via the client computing device to perform one or more actions specific to the content of the document.

6. The method of claim 5, wherein: The one or more actions include causing an additional application of the client computing device to open in a state specific to the content.

7. A method as claimed in any preceding claim, wherein: The triggering portion determined from the document for rendering the suggestion includes target content determined by the server computing device based on the content in the document.

8. The method according to any one of claims 1 to 5, wherein: The triggering portion determined from the document for rendering the suggestion includes a historical portion determined by the server computing device based on historical user data associated with the document.

9. The method of claim 8, wherein: The historical user data includes historical user interaction frequencies for one or more suggestions related to the content of the document, and wherein the historical portion of the document is determined by being a portion of the document that was historically displayed at the user interface when a suggestion having a highest historical user interaction frequency among the one or more suggestions was historically triggered at the user interface.

10. A method as claimed in any preceding claim, wherein: The coordinate information changes in response to a user scroll input that scrolls the document to change content of the document displayed at the user interface.

11. A method as claimed in any preceding claim, wherein: In generating the coordinate condition, the server computing device uses the type and / or size information of the user interface to determine a distance metric within which the triggering portion of the document will be rendered, and generates the coordinate condition based on the distance metric.

12. A method as claimed in any preceding claim, wherein: Causing the suggestion to be rendered at the user interface includes: causing the suggestions to be rendered at a bottom bar or a top bar of the user interface, and / or Causes the suggestion to be rendered via an HTML overlay element.

13. A method implemented using one or more processors of a server computing device, the method comprising: Received from the client computing device: document information identifying a document displayed at a user interface of the client computing device, and User interface information, the user interface information identifying type and / or size information of the user interface of the client computing device; In response to the receiving: using the document information to identify suggestions related to content of the document; generating a coordinate condition, wherein generating the coordinate condition is based on the type and / or the size information of the user interface and based on a triggering portion of the document for rendering the suggestion; and Send to the client computing device: The recommendations and coordinate conditions, Wherein, sending the suggestion and the coordinate condition causes the client computing device to monitor whether the coordinate condition is met and render the suggestion when the coordinate condition is met.

14. The method of claim 13, wherein: The suggestions are pre-generated and indexed in an index based on the document information, and wherein identifying the suggestions includes using the document information to identify the pre-generated suggestions in the index.

15. The method of claim 14, wherein: The document information includes a resource identifier of the document.

16. The method of claim 15, wherein: The resource identifier is the uniform resource locator of the document.

17. The method of claim 13, wherein: The document information includes the content of the document, and wherein identifying the suggestion includes processing the content to generate the suggestion.

18. A method implemented using one or more processors, the method comprising: identifying, for a document displayed at a user interface of a client computing device, a historical statistical distribution of selection times, the historical statistical distribution of selection times being generated based on historical selection times of corresponding suggestions related to content of the document and rendered simultaneously with the document; generating a time condition for rendering a suggestion related to the content based on the historical statistical distribution; as well as In response to rendering the document at the user interface of the client computing device: causing the client computing device to: monitoring whether the time condition is met, and When the coordinate condition is met, the suggestion is rendered.

19. The method of claim 18, wherein: The historical selection times each relate to a corresponding initial rendering of the document, and wherein the time condition relates to the initial rendering of the document.

20. The method of claim 18 or claim 19, further comprising: identifying, for the client computing device or for a user account accessing the document at the computing device, a historical metric of suggested selections; The generating of the time condition is based on the historical statistical distribution and the historical measurement of the suggested selection.

21. The method of claim 20, wherein: Generating the time condition based on the historical statistical distribution and the historical metric of the suggested selection comprises: Depending on the historical metric of the suggestion, a time is selected from the historical statistical distribution.

22. The method of claim 18 or claim 19, wherein: Generating the time condition based on the historical statistical distribution includes: selecting a time from the historical statistical distribution, wherein selecting the time depends on a probability defined by the historical statistical distribution; and The time condition is generated based on the time.

23. The method of claim 22, wherein: The suggestion is selected at the client computing device after rendering the suggestion, and the method further comprises: determining an actual selection time of a selection of the suggestion at the client computing device; and The historical statistical distribution is updated based on the actual selection time.

24. The method of any one of claims 18 to 23, wherein: The historical statistical distribution of the selection times is all specific to the document.

25. The method of any one of claims 18 to 23, wherein: The historical statistical distribution at the time of selection includes selecting from a plurality of different documents determined to have one or more characteristics in common, and wherein the historical statistical distribution is utilized for the document in response to determining that the document also has the one or more characteristics.

26. A method implemented using one or more processors of a client computing device, the method comprising: Send to the server computing device: document information identifying a document displayed at a user interface of the client computing device, and Receiving in response to said sending: suggestions related to the content of said document, and one or more conditions, the one or more conditions comprising a document object model (DOM) condition, a time condition, and / or a coordinate condition; In response to receiving the suggestion and the one or more conditions: When rendering the document in the user interface, monitoring whether at least one of the one or more conditions is satisfied, and In response to determining that at least one of the one or more conditions is satisfied: The suggestion is caused to be rendered at the user interface.

27. The method of claim 26, wherein: The one or more conditions include the time condition and the coordinate condition, and wherein monitoring whether the at least one of the one or more conditions is satisfied includes determining whether either the time condition or the coordinate condition is satisfied.

28. A method implemented using one or more processors of a client computing device, the method comprising: Send to the server computing device: document information identifying a document displayed at a user interface of the client computing device; Receiving in response to said sending: Suggestions related to the content of said document, a node of the Document Object Model (DOM) of the document, and The coordinate conditions generated by the server computing device are based on: type and / or size information of the user interface, and a trigger portion for rendering the suggestion determined from the document; as well as In response to receiving the suggestion and the node of the DOM of the document: When rendering the document in the user interface, monitoring whether a markup language corresponding to the node of the DOM is actively executed, and In response to determining to actively execute the markup language corresponding to the node of the DOM: The suggestion is caused to be rendered at the user interface.

29. A method implemented using one or more processors, the method comprising: identifying suggestions related to content of the document using document information for a document displayed at a user interface of a client computing device; generating a coordinate condition, wherein generating the coordinate condition is based on the type and / or size information of the user interface and based on a triggering portion of the document for rendering the suggestion; In response to identifying the suggestion and generating the coordinate condition: monitoring whether the coordinate condition is satisfied in coordinate information of the triggering portion of the document relative to the user interface when the document is displayed at the user interface, and In response to determining during the monitoring that the coordinate condition is satisfied: The suggestion is caused to be rendered at the user interface.

30. The method of claim 29, wherein: The one or more processors consist of a processor of the client device.

31. A system comprising: at least one processor; as well as A memory storing instructions which, when executed, cause the at least one processor to perform operations corresponding to any one of claims 1 to 30.

32. At least one non-transitory computer-readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 30.

Images