Generate action elements that suggest content for ongoing tasks

Abstract

To provide a computing system and method which can be used to surface a selectable action element for at least one ongoing task.SOLUTION: In particular, the present disclosure provides a general pipeline for identifying potential tasks that a user has an ongoing interest in or has not yet completed so that a suggestion of a content item can be made to further advance an identified user's task. This pipeline can incorporate probabilistic transition graphs, machine-learned models, and / or historical data to determine relevance and completion of tasks that a user may desire to continue acting upon.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Priority claim This application claims priority to U.S. Application No. 17 / 368,155, filed July 6, 2021, which is incorporated herein by reference.

[0002] The present disclosure relates generally to identifying an ongoing task associated with a user, and more particularly, to generating action elements that suggest content for the identified ongoing task based on historical user web browsing data. [Background technology]

[0003] A web browser application ("browser") can fetch content from a web server and display the content on a user's device. A user can enter a uniform resource locator (URL) so that the browser can retrieve data (e.g., content) associated with the URL (e.g., by utilizing the Hypertext Transfer Protocol to communicate with the web server).

[0004] In some cases, communications between a browser and a web server may be encrypted for privacy and security purposes. Once a browser retrieves a web page, the browser's rendering engine can display the web page on the user's device, including image and video formats supported by the browser. Most browsers can use an internal cache of web page resources to improve loading times for subsequent visits to the same page. The cache can store many items, such as large images, so these items do not need to be downloaded from the server again.

[0005] Generating personalized search queries is the process of leveraging web browsing technologies to provide personalized content for users. Some standard query personalization techniques rely on aggregated user information, such as common search queries. However, existing approaches make no attempt to determine the specific tasks a particular user may be performing.

[0006] While progress has been made in the area of personalized search queries, existing approaches are typically limited to users utilizing pre-existing search bars for single, isolated queries and do not consider alternatives or overarching objectives for any given user search. Summary of the Invention [Means for solving the problem]

[0007] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments. One exemplary aspect of the present disclosure is directed to a computer-implemented method for generating task-specific action elements. The method includes: a computing system obtaining historical user data describing historical user actions taken in one or more past user online sessions; the historical user data being annotated with annotations describing attributes of content associated with the user actions; the method including the computing system identifying one or more tasks from the historical user data; the method including the computing system determining suggested content items for each of the one or more ongoing tasks; and the computing system surfacing selectable action elements for at least one ongoing task for display to a user. The selectable action elements are configured to provide access to at least one suggested content item for the at least one ongoing task.

[0008] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain relevant principles.

[0010] Detailed descriptions of embodiments directed to persons skilled in the art are set forth herein, which refer to the accompanying figures. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of an exemplary computing system according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 1 illustrates an exemplary client device including one or more processors and a user interface. [Figure 3] FIG. 2 illustrates exemplary web content displayed in a browser window within a user interface of a client device. [Figure 4] FIG. 2 is a block diagram of an exemplary content recommendation model, according to an exemplary embodiment of the present disclosure. [Figure 5] 1A-1C illustrate exemplary user interfaces illustrating how action elements may be presented to a user, according to exemplary embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates an exemplary user interface illustrating how a dedicated dashboard can be utilized to present multiple ongoing tasks to a user along with associated action elements, according to an exemplary embodiment of the present disclosure. [Figure 7]A diagram showing an exemplary user interface according to an exemplary embodiment of the present disclosure, illustrating how a user can interact with a particular ongoing task. [Figure 8] A diagram showing an exemplary user interface according to an exemplary embodiment of the present disclosure, illustrating how a user can interact with a particular individual ongoing task. [Figure 9] A flowchart diagram of an exemplary method for performing the generation of action elements that suggest content for an ongoing task according to an exemplary embodiment of the present disclosure. **DETAILED DESCRIPTION**

[0012] Reference numerals repeated across multiple figures are intended to identify the same features in various implementations.

[0013] Overview Generally, the present disclosure is directed to computing systems and methods that can be used to surface selectable action elements for at least one ongoing task, such as action elements configured to provide access to at least one suggested content item associated with the ongoing task. In particular, the present disclosure provides a general-purpose pipeline for identifying potential tasks that a user has an ongoing interest in or has not yet completed, such that suggestions of content items can be made to further the identified user tasks. This pipeline can incorporate probabilistic transition graphs, machine-learned models, and / or historical data to determine the relevance and completion of tasks that a user may wish to continue to act on.

[0014] More specifically, an exemplary computing system can identify an ongoing task. For example, the computing system can identify an ongoing task by analyzing historical user data to infer what tasks the user may be involved in in the future. By leveraging what the user has already done in the past, the computing system can determine an overarching category of activity, and from that category, the computing system can evaluate a pipeline of predicted steps the user is most likely to take with respect to the identified task. For example, if the computing system identifies that the user was engaged in a shopping activity, the computing system can generate (e.g., using a probabilistic transition graph, a machine-learned model, etc.) a most likely pipeline for shopping (e.g., inspiration, discovery, consideration, validation, purchase, post-purchase) and / or assess the user's status or position along such a pipeline. In this way, the computing system can leverage historical user data to determine which stage in the pipeline the user is most likely to be involved in next.

[0015] In some implementations, the computing system can optimize a pipeline of activities in which the user may be involved to complete a particular task. The computing system can optimize a pipeline of activities in which the user may be involved by predicting the next most likely step the user can take and identifying relevant content. The relevant content can then be presented to the user such that the user does not have to manually search for the content themselves. The relevant content can be presented to the user by surfacing selectable action elements. The selectable action elements can be displayed on a particular dashboard that the user can navigate to view a plurality of tasks determined by the computing system. As an example, the user can interact with a dedicated dashboard to indicate which of a plurality of available tasks the user desires to interface with. Further, the selectable action elements can be overlaid on the current browsing session in which the user is involved to provide suggestions that the user can interact with to nudge the user to facilitate an ongoing task.

[0016] An exemplary method for surfacing selectable action elements for at least one ongoing task can include obtaining historical user data. In particular, the historical user data can describe historical user actions taken in one or more past user online sessions. As an example, the historical user data can be annotated with annotations. In particular, the annotations can describe attributes of the content associated with the user action. For example, the computing system can obtain historical user data and attach an annotation that a portion of the historical data was related to searching for or purchasing headphones. Even more specifically, the annotations can be based on metadata associated with the content.

[0017] One or more ongoing tasks may be identified by the computing system. The one or more ongoing tasks may be identified based on historical user data (e.g., using historical user web browsing data). The ongoing tasks may be discrete tasks (e.g., purchasing an item, planning a vacation, etc.). The ongoing tasks may be continuous tasks (e.g., cooking, movies to watch, etc.). For example, the computing system may identify a discrete task such as purchasing a set of headphones.

[0018] One or more suggested content items can be determined by a computing system. For each of one or more ongoing tasks, one or more suggested content items can be determined. For example, if a user is experiencing the task of buying a set of headphones, the computing system can suggest content items related to purchasing headphones. For example, the computing system can suggest content items such as a purchasing guide for buying headphones, reviews of headphones that the user has browsed according to historical browser data, various headphones themselves, suggested search queries, comparison guides, or related products (e.g., attachable microphones, headphone cases, etc.). The suggested content items can change according to the identified user's ongoing tasks. In particular, the suggested content items can include adding the content to a user data bank (e.g., a list). For example, if the computing system instead determines that the user is experiencing an interest in cooking, the suggested content items can instead be recipes that can be added by the user to the user's recipe book in combination with other suggested content (e.g., cooking equipment, grocery lists, locations of grocery stores, etc.). As another example, if the computing system determines that the user is interested in movies, the suggested content items can include movies to be added to a watch list in combination with other suggested content (e.g., movie reviews, related movies, etc.).

[0019] The selectable action element may be raised for display to the user. The selectable action element may be related to at least one ongoing task. In particular, the selectable action element may be configured to provide access to at least one suggested content item for the at least one ongoing task. The selectable action element may include an image related to the ongoing task to which the action element pertains. As an example, a selectable action element related to a user trying to buy headphones may include an image of headphones on the action element itself.

[0020] In some implementations, historical user actions may be extracted across multiple web pages. For example, historical user actions may be extracted from multiple web pages opened by a user, regardless of whether the web pages were opened and then closed. Furthermore, historical user actions may be extracted across multiple web browser sessions. In particular, historical user actions may be extracted from multiple instances of a user initiating a search using a web browser, closing the web browser for a period of time, and then returning to the web browser to initiate another search (e.g., a different search). As an example, historical user actions may be extracted for web browser sessions over a predetermined period of time. Specifically, historical user actions within 30 days prior to the extraction of historical user actions may be extracted. A web browser session may correspond to a distinct instantiation of a web browser application (e.g., one session corresponds to the loading and unloading of an instantiation of a web browser application into device memory), use of a web browser application over a period of time (e.g., each different day of use of a web browser application corresponds to one session), web browser use during a given user's login period, different tabs within a web browser, and / or other segmentation of browser application use over time.

[0021] In some implementations, one or more ongoing tasks can be identified by a computing system. Further, one or more ongoing tasks from historical user data can be annotated. The historical user data can be annotated with annotations that describe attributes of the content associated with the user action. As an example, historical user data showing a user search for the best headphones in 2021 can be annotated to be described as related to purchasing headphones. In particular, the annotated historical user data can be placed in multiple clusters. Further, one or more ongoing tasks can be identified from the historical user data based at least in part on the multiple clusters. In one example, identifying one or more ongoing tasks can include implementing an edge threshold algorithm to identify one or more primary clusters among the multiple clusters. Examples of "edges" or "dimensions" that product image searches can match include recognition-inducing attributes such as categories (e.g., "headphones"), attributes (e.g., "noise cancellation", "gaming", etc.), or other semantic dimensions and / or machine-generated visual attributes such as machine-extracted visual characteristics or machine-generated visual embeddings, such as visual attributes like "dark color with bright color accents", "the bright color accents are thin lines that make up 40% of the entire color space".

[0022] In some implementations, one or more ongoing tasks from historical user data can be identified. In particular, the historical user data can be input into a machine-learned model. Even more specifically, the machine-learned model can generate one or more ongoing tasks from the input historical user data.

[0023] In some implementations, identifying one or more ongoing tasks from the historical user data may be based at least in part on a relevance score. In particular, the relevance score may be based at least in part on a timestamp associated with the historical user data. Even more particularly, content with a higher relevance score may be weighted more heavily than content with a lower relevance score. For example, a relevance score for a web browsing search performed within the last 24 hours may be higher than a relevance score for a web browsing search performed 30 days ago. Furthermore, the timestamp may be associated with the duration spent browsing particular content. In particular, the relevance score may be based at least in part on the duration spent browsing particular content. For example, the relevance score may be higher for content browsed over a longer period of time. Additionally, the relevance score may be based at least in part on repetition of the web browsing search. For example, the relevance score may be higher for content browsed repeatedly. Even more particularly, the relevance score may be required to meet a relevance threshold to make the related content suitable for identifying a potential user task.

[0024] In some implementations, identifying one or more in-progress tasks from historical user data can include generating a probability distribution. In particular, the probability distribution can include a set of next step intents (i.e., the intent to perform an action) that are at least partially based on the historical user data. Even more specifically, the historical user data can be extracted from multiple users. Further, the suggested next step can be identified at least partially based on a completion metric. In particular, identifying one or more in-progress tasks from historical user data can include identifying the suggested next step. Even more specifically, suggesting the next step can be based in part on a completion metric, where the completion metric indicates the user's status with respect to completing a predetermined checkpoint associated with one or more in-progress tasks. Even more specifically, the predetermined checkpoint can indicate what information the user has already consumed. For example, the identified suggested next step can vary depending on how complete the task the user is experiencing is, or which checkpoints the user has already completed. As an example, if a user is trying to buy headphones, if the user is in a preliminary stage (e.g., the user has not hit any checkpoints), the user can be mostly assisted by browsing a purchase guide. On the other hand, if the user is in a later stage of the process (e.g., the user has already hit multiple checkpoints such as browsing a purchase guide), the user may be interested in specific headphone reviews. Further, historical data can indicate a strong user interest in completing a task, but since the user may have actually recently completed this task, content can no longer surface to the user with respect to the completed task. In particular, the completion metric can be at least partially based on a URL associated with at least one predetermined checkpoint. Even more specifically, the URL associated with at least one predetermined checkpoint can be associated with one or more in-progress tasks.As an example, the URL from the purchase confirmation web page can signal that the user has already purchased a set of headphones and completed this task. Accordingly, in response, the computing system can remove the headphones from the potential ongoing tasks determined by the computing system.

[0025] In some implementations, identifying one or more ongoing tasks from historical user data can involve accessing a continuously updated probabilistic transition graph that describes the predicted pipeline of steps a user can take to complete a task. Specifically, the probabilistic transition graph can identify the next step intent for the identified task. In particular, the probabilistic transition graph can be based on multiple historical user tasks, which can span multiple different users. Even more specifically, the probabilistic transition graph can reflect different next step intents within the ongoing task needs for different categories of tasks. For example, even within a shopping task, there can be differences within the probabilistic transition graph depending on the product (e.g., TV vs. mobile phone vs. sofa). Specifically, a unique distribution can be generated for each task category and subcategory. Based on the probabilistic transition graph, the computing system can identify the next step intent for one or more ongoing tasks. In particular, the probabilistic transition graph can include a hierarchical representation of the next step intent for one or more ongoing tasks that is at least partially based on historical user data. Even more specifically, the computing system can select one or more content intents at least partially based on the hierarchical representation of the next step intent. As an example, the determined or predicted user preferences can determine the hierarchical representation of the next step intent (e.g., common product attributes, brand affinity, etc. across all products explored).

[0026] In some implementations, suggested content items for each of one or more next step intents in an ongoing task may be ranked. In particular, suggested content items for each of one or more next step intents in an ongoing task may be ranked based on at least one quality attribute. Even more particularly, the quality may be one or more of other user engagement level (e.g., the percentage of other users who interact with the suggested content item when presented as an option), reviews (e.g., comments on an article that may be provided as suggested content), freshness (e.g., how new or old the article is), or content relevance (e.g., a headphones buying guide as opposed to gaming computer reviews).

[0027] In some implementations, surfacing a selectable action element for at least one ongoing task for display to the user includes surfacing the selectable action element on a dedicated dashboard. For example, the dedicated dashboard may be configured to display multiple ongoing tasks. For example, the multiple ongoing tasks may be depicted with a heading indicating the user's explicit task unit (e.g., "headphone shopping") so that the user can interact with one of the multiple ongoing tasks to navigate to another interface that allows the user to interact with the particular ongoing task at a finer granularity.

[0028] Interacting with a particular ongoing task at a finer granularity can allow a user to not only have an even more detailed influence on the content surfaced by the computing system, but also broaden what suggestions the user can view. For example, a user can see a preview of the current suggestions as well as implicit task units such as suggested next steps. Products can be tracked and viewed, as well as interact with tracked products in ways such as turning alerts on or off for tracked products. By interfacing with particular surfaced content, a user can indicate whether the content is accurate in predicting useful content or whether the user is not interested (e.g., if the user indicates a lack of interest, the particular surfaced content can be re-ranked and alternative content can be surfaced instead). Another way a user can interact more deeply with a particular ongoing task is by interacting with a timeline of content the user has interacted with during the duration of the particular ongoing task.

[0029] The dedicated dashboard can present selectable action elements indicating suggested content items in a variety of ways. For example, the selectable action elements may be presented on a carousel (e.g., suggested movies to view), as previews of suggested web pages (e.g., a headphone buying guide), as notifications of price drops on previously viewed items (e.g., a sale on a brand of headphones), etc. Furthermore, the ranking of the suggested content items can determine how the action elements for a particular suggested content item are presented. For example, suggested content items ranked higher than a certain threshold may be presented with a larger icon (e.g., with an image depicting the content), while suggested content items ranked lower than a certain threshold may be presented with a smaller icon (e.g., below the higher-ranked content, as words only, etc.). Even more particularly, badges can be overlaid on the selectable action elements to inform the user of relevant information (e.g., streaming services that offer the suggested movie, when the user last viewed the content, etc.).

[0030] In some implementations, surfacing selectable action elements for at least one ongoing task for display to the user includes obtaining current user web browser data. The current user web browser data can include one or more of text content or image content. One or more ongoing tasks from the historical user data can be identified based at least in part on one or more of text content or image content. In particular, identifying one or more ongoing tasks from the historical user data can be based at least in part on semantically analyzing the text content. Even more particularly, the content type can be determined based at least in part on the semantically analyzed text content of the current user web browser data. Further, identifying one or more ongoing tasks from the historical user data can be based at least in part on identifying one or more compositional characteristics of the image. Even more particularly, the content type can be determined based at least in part on one or more compositional characteristics of the image of the current user web browser data. Continuing with the example from above, if the user was currently browsing an article about earbuds, the computing system can surface a selectable action element that suggests articles related to purchasing a set of headphones based at least in part on the word "earbud" appearing in the article or an image of earbuds appearing in the article.

[0031] Accordingly, the present disclosure provides computing systems and methods that can be used to surface selectable action elements for at least one ongoing task, e.g., action elements configured to provide access to at least one suggested content item. In particular, the present disclosure provides a generalized pipeline for identifying potential tasks in which a user has continuing interest or is not yet completed, so that content item suggestions can be made to further advance the identified user's task. The pipeline incorporates hierarchical graphs, machine-learned models, and historical data to determine the relevance and completion of tasks that a user may want to continue acting on and continue surfacing selectable action elements in various interfaces.

[0032] The systems and methods of the present disclosure provide several technical effects and benefits. As one exemplary technical effect, the proposed techniques can provide users with an immersive and useful experience of predicting the next steps in the tasks that the users are trying to complete or continue. In particular, providing useful suggestions regarding tasks can significantly overcome the dissatisfaction of many users when they are engaged in long-term tasks involving multiple searches without the ability to utilize previous work in an effective and productive way by removing noise. Contrary to only predicting what the user might be searching for based on the first text query input, the present disclosure shows a way to directly provide the results of historical searches to more efficiently provide the most useful content for the user. Further, the present disclosure enables users to more fully utilize historical content effectively and in combination with its future content to achieve completion of the started task in ways that would not be possible with only predictions of searches based on the first input. In particular, the proposed techniques reduce redundant work or wasted searches, thereby not only saving computing resources (e.g., processor usage, memory usage, network bandwidth, etc.) but also reducing the user's time and dissatisfaction. Specifically, by surfacing relevant content for the ongoing task, the total number of searches required to perform the task can be reduced, and as a result, computing resources such as those described above can be saved. Further, the proposed techniques surface content that is further tailored to the user and might not have surfaced based on general searches the user might have entered, thus reducing meaningless content consumption.

[0033] Next, with reference to the figures, exemplary embodiments of the present disclosure are described in more detail.

[0034] Exemplary Devices and Systems 1 illustrates a block diagram of an exemplary computing system 100 that implements personalized and / or intelligent search at least in part in response to a visual query, in accordance with an exemplary embodiment of the present disclosure. The computing system 100 includes a user computing device 102 and a web server 104 that are communicatively coupled via a network 180.

[0035] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0036] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0037] In some implementations, the web browser application 124 of the user computing device 102 retrieves content associated with objects referenced in the user input component 122 of the user computing device 102. For example, the web browser can retrieve content associated with a web page requested by the user and then display the page on the user interface 158 of the device 102.

[0038] Although the web browser application 124 is shown in FIG. 1 as being included in the device 102 , in other implementations, some or all of the functionality of the web browser application 124 may be implemented in the web server 104 .

[0039] The web server 104 includes one or more front-end servers 136 and one or more back-end servers 140. The front-end server 136 can receive user input components 122 from a user computing device, e.g., the user computing device 102 (e.g., from a web browser 124). The front-end server 136 can provide image data to the back-end server 140. The back-end server 140 can identify content associated with objects recognized in the user input data and provide that content to the front-end server 136. The front-end server 136 can then provide that content to the mobile device from which the image data was received.

[0040] The backend server 140 includes one or more processors 142 and a memory 146. The one or more processors 142 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operably connected. The memory 146 can include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc. and combinations thereof. The memory 146 can store data 148 and instructions 150 that are executed by the processor 142 to cause the web server 104 to perform operations. The backend server 140 can also include a query processing system 152.

[0041] Alternatively, the backend server 140 may not be able to access a prior file for providing to the frontend server 136. Thus, the file can be generated upon request by another program that communicates with the web server 104 and provides content 160. Next, the web server 104 and the frontend server 136 can provide the content to the mobile device from which the image data was received.

[0042] In some implementations, the web server 104 includes one or more server computing devices or, alternatively, is implemented by one or more server computing devices. In cases where the web server 104 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0043] In some implementations, query processing system 152 includes multiple processing systems. One exemplary system can enable the system to identify multiple candidate search results. For example, the system can identify multiple candidate search results when it initially receives a user input component. Alternatively, the system can identify multiple search results after further processing by the system has already occurred. Specifically, the system can identify multiple search results based on a more targeted query generated by the system. Even more specifically, when the system initially receives a user input component and then regenerates the multiple candidate search results after further processing, the system can generate multiple candidate search results based on the more targeted query generated by the system.

[0044] After content is selected, the content can be provided to the user computing device 102 from which the user input component was received, stored in the content cache 138 of the web server 104, and / or stored at the top of the memory stack of the front-end server 136. In this manner, the content can be quickly presented to the user in response to the user's request for the content. When the content is provided to the user computing device 102, the web browser 124 can store the content in the content cache 134 or other fast-access memory. For example, the web browser 124 can store content for an object with a reference to the object so that the web browser 124 can identify appropriate content for the object in response to a decision to present content for the object.

[0045] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component can be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0046] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may be carried over any type of wired and / or wireless connection using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL). 1 illustrates one exemplary computing system that may be used to implement the present disclosure. Other different distributions of components may also be used. For example, some or all of the various aspects of the visual search system may instead be located and / or implemented in the user computing device 102.

[0047] 2 shows an exemplary client device 102 that includes one or more processors 210 and a user interface 212. The client 102 may be, for example, but should not be limited to, a personal computer, a laptop computer, a personal handheld device, a mobile phone, a game console, a set-top box, an embedded system, etc. The client 102 may include any computing device that is capable of requesting web content from a web content server, such as server 104.

[0048] The client 102 includes a user interface 212, which may include various types of input and output that enable a user to interact with the client 102. Exemplary inputs may include, but are not limited to, a mouse, a keyboard, a keypad, a touchscreen, a microphone, etc. Exemplary outputs may include, but are not limited to, a display for visual output, a speaker for audible output, etc. Many, if not all, of the above example interfaces are driven, supported, or enhanced by hardware, firmware, and / or software located or operating within the client 102. For viewing web content, for example, the user interface 212 may include a web browser 214. The browser 214 includes software operating on the client 102 that enables a user to request and view web content (i.e., content that may be provided from a server, such as server 104, connected to the client 102 via one or more networks (e.g., the Internet or the World Wide Web)). Examples of browsers 214 include, but are not limited to, Chrome by Google™ Inc., Internet Explorer™ by Microsoft™, Firefox™ by Mozilla™ Corporation, Safari™ by Apple™ Inc., Opera™ by Opera Software™ ASA, and the like.

[0049] FIG. 3 shows an exemplary browser window 314 displayed to a user of client device 102. A typical browser window 314 can include a title bar 316, a menu bar 318, a toolbar 320, a URL (Uniform Resource Locator) field 324, and a body 325. The title bar 316 can include a title given to the web page being viewed by the designer of the web page. The menu bar 318 can include various drop-down menus that include tools and options related to the web page being viewed. The toolbar 320 can include various buttons 322 that represent tools and options related to the currently viewed web page and / or navigation to other web pages. The URL field 324 can include the address of the web page being viewed and can allow the user to type in the address of another web page to be viewed. The body 325 can include the content of the web page, including, for example, text 326 and images 330. The text within the body 325 can also represent hyperlinks 328 that are typically highlighted and / or underlined so as to be distinguishable from normal text. When the user selects or clicks on a hyperlink 328, a new web page can appear, for example, instead of or in addition to the currently viewed web page related to the text shown in the hyperlink 328.

[0050] When a user attempts to view a web page or particular web content on a web page (e.g., a new view of an image) by, for example, entering an address into a URL field in a browser window such as browser window 314, clicking a hyperlink (in a currently viewed web page, email, electronic document, etc.), or using menu bar 318 or toolbar 320, a request for the corresponding web content is sent over one or more networks 106 to an appropriate web content server 104. Server 104 retrieves or fetches the requested web content, for example, from database 108, and serves (sends) the web content to the requesting client device 102. Client device 102 receives the requested web content and renders it, for example, for display in browser window 314.

[0051] Example Model Configuration FIG. 4 shows a block diagram of an exemplary content recommendation model 200 according to an exemplary embodiment of the present disclosure. In some implementations, the content recommendation model 200 receives a set of input data 204 that describes historical user data and, as a result of receiving the input data 204, provides output data 206 that identifies content presented by a machine-learned model as being recommended to a user to achieve a predicted next step in an identified task. Thus, in some implementations, the content recommendation model 200 can include a task generation model 202 operable to generate one or more tasks that a user is currently engaged in. In particular, the task generation model 202 can utilize the input data 204 to determine a current task based on historical user actions determined from the historical user data. For example, the task generation model can generate annotations for the historical user data (e.g., based on metadata associated with the historical user data). Based on the annotations associated with the historical user data, the task generation model 202 can generate one or more predicted tasks that the user may be engaged in. Even more specifically, the content recommendation model can utilize the input data 204 in combination with one or more predicted tasks generated by the task generation model 202 that the user may be engaged in to generate output data 206, particularly one or more predicted content predicted to achieve a next step in a predicted task that the user may be engaged in.

[0052] Exemplary method FIG. 9 shows a flowchart diagram of an exemplary method implemented in accordance with an exemplary embodiment of the present disclosure. FIG. 9 shows steps performed in a particular order for purposes of illustration and description, but the methods of the present disclosure are not limited to the specifically shown order or arrangement. The various steps of method 900 may be omitted, reordered, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0053] At 902, the computing system can obtain historical user data that describes historical user actions taken in one or more past user online sessions. In particular, the historical user data that describes historical user actions can be annotated with annotations that describe attributes of the content associated with the user actions.

[0054] At 904, the computing system can identify one or more ongoing tasks from the historical user data.

[0055] At 906, the computing system can determine a suggested content item for each of the one or more ongoing tasks.

[0056] At 908, the computing system can surface selectable action elements for at least one ongoing task for display to the user. In particular, the selectable action elements can be configured to provide access to at least one suggested content item for at least one ongoing task.

[0057] Exemplary Application FIG. 5 illustrates a first exemplary application of the method disclosed in FIG. 9. 500 illustrates an exemplary user interface in which a user can engage in continuing a predicted task. For example, a user can engage in continuing a predicted task (e.g., prompting a task-specific dashboard of predicted content) by interacting (e.g., touching, selecting, toggling, etc.) with a selectable action element 502. In particular, the selectable action element 502 can include an icon 508 symbolizing the predicted task. For example, if the predicted task is to purchase a set of headphones, the selectable action element 502 can include an icon 508 of a set of headphones. The user interface 500 can additionally include at least one piece of content 512 that is directly recommended to the user. In particular, a user can interact with the at least one piece of content 512 that is directly recommended to the user without engaging with the selectable action element 502. For example, a user can directly engage with the recommended content 512 without prompting a task-specific dashboard of predicted content by interacting with the selectable action element 502. The user interface 500 may include an overlay 504 that describes how the directly recommended content 512 may impact the predicted task. Additionally, the user interface 500 may include tags 506 that describe keywords associated with the predicted task. Additionally, the user interface 500 may include a favorite toggle 510 that a user may engage to indicate that the user likes the directly recommended content 512. The computing system may save the directly recommended content 512 in a separate, recoverable location when the user activates the favorite toggle 510. The computing system may additionally have a button 514 that a user may interact with to launch an immersive dashboard related to the predicted task.

[0058] Figure 6 shows another exemplary application of the method disclosed in Figure 9. 600 shows an exemplary immersive dashboard user interface where a user can engage in continuing two or more predicted tasks. For example, the dashboard can use text header 602 to show two or more predicted tasks such that the user can browse which task the user desires to continue. The predicted tasks can include panels 610 on carousel function 604 that show predicted next steps for the user involved in that task. For example, panels 610 on carousel function 604 can include functions for the user to continue browsing options. Panels 610 on carousel function 604 can include predicted items of interest. Panels 610 on carousel function 604 can include user-saved content. In particular, as an example, the predicted task can be cooking. Thus, as a further example, panels 610 on carousel function 604 showing predicted items of interest can include predicted recipes of interest. As a further example, if the predicted task is to purchase headphones, panels 610 on carousel function 604 showing predicted items of interest can include headphones or earphones on sale. Panel 610 can include an information overlay 612. Information overlay 612 can show information related to the content associated with panel 610. For example, information overlay 612 can show the price or source of the content associated with panel 610. The user can indicate a desire to view more detailed suggestions related to the task by interacting with detail icon 606. By interacting with detail icon 606, the user can browse additional content related to the task. By interacting with detail icon 606, the user can secure a further dashboard that displays additional content related to one specific task. The additional content can have an information overlay 608 that shows the reason why the computing system surfaced the specific content.For example, the content can have an information overlay 608 indicating that the content has been viewed by the user in the past (e.g., "viewed four days ago"), or that the content has features similar to content saved by the user or previously viewed by the user (e.g., "inspired by saved content").

[0059] Figure 7 shows another exemplary application of the method disclosed in Figure 9. 700 shows an exemplary immersive dashboard user interface where a user can engage in continuing a particular predicted task. In particular, 700 shows how an immersive dashboard associated with a particular predicted task can enable a user to continue a particular task by increasing specificity. For example, card 712 can show predicted content related to a particular task. In particular, the designated action element 702 can be floated, and as a result, the user can interact with the designated action element 702 to trigger a predicted action tailored to the particular predicted content in combination with the particular task. For example, the designated action element 702 can connect the predicted content to other functions of the smart home (e.g., Nest, Echo, etc.). As a further example, in the case of card 712 that suggests a recipe, the designated action element 702 can include actions that the user can engage in, such as buying ingredients. Card 712 can also have a bookmark function 704 so that the user can indicate that they want to revisit the content associated with card 712. In some cases, an overlay can appear on the interface to prompt the user to show feedback in response to the predicted content. For example, the user can interface with a positive icon 706 or a negative icon 708 on the overlay to show feedback regarding the presented content. Additionally, the user can insert a note 710 associated with a particular content, and the note can include a user-written memo regarding the relevant content. The immersive dashboard user interface 700 can additionally incorporate a share icon 714, and the user can share the task with another user. The second user can be a bystander of the task as it progresses or can be added as an active contributor.When more than one user contributes to a particular task, some predictions may be attributable to a particular user, and the representation of a particular user (e.g., an image, symbol, name, etc.) can be utilized to indicate which user a particular prediction is attributable to.

[0060] Figure 8 shows another exemplary application of the method disclosed in Figure 9. 800 shows another exemplary immersive dashboard user interface where a user can engage in continuing a particular predicted task. In particular, 800 further shows how an immersive dashboard associated with a particular predicted task can enable a user to continue a particular task by increasing specificity. For example, the immersive dashboard 800 can include a related activity search 802 that can surface which keywords were included in a search. In some cases, the keywords included in the search can be hyperlinked such that interacting with those keywords initiates a full search for the user. Alternatively, the results from the related activity search 802 can be displayed in a results panel 808 alongside the related activity search 802. The results panel 808 can be displayed in a carousel fashion. The immersive dashboard user interface 800 can include a step counter 810 that indicates the number of next steps predicted by the computing system. The number of next steps shown on the step counter 810 can be the number of most strongly suggested steps (e.g., steps above a particular confidence threshold). Further, the predicted steps can be shown in a step summary panel 804 which can be hyperlinked such that a user can interact with the step summary panel 804 to initiate related steps. As an example, a predicted step can be to review a price drop of an item that the user has shown interest in (e.g., by having viewed the item in the past, marked it as a favorite, added it to a wish list, bookmarked it, etc.). Another predicted step can be to browse similar products (e.g., browse similar products below a certain cost). The immersive dashboard 800 can additionally include a user interfacing prompt 806.The user interfacing prompt 806 may allow a user to interact with the user interfacing prompt 806 to surface even more detailed predictions output by the computing system. As an example, the user interfacing prompt 806 may be associated with a step counter 810 and a step summary panel 804 so that the user can select the user interfacing prompt to view all predicted next steps, as opposed to only the most strongly suggested next step selected by the computing system.

[0061] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0062] Although the subject matter has been described in detail with respect to various specific exemplary embodiments, each example is provided by way of illustration and not limitation of the disclosure. Those skilled in the art will readily recognize that modifications, variations, and equivalents of such embodiments can be readily made. Accordingly, the disclosure is not intended to exclude such modifications, variations, and / or additions to the subject matter that will be readily apparent to those skilled in the art. For example, features shown or described as part of one embodiment can be used with another embodiment to yield further embodiments. Accordingly, the disclosure is intended to cover such modifications, variations, and equivalents.

Explanation of Signs

[0063] 100 Computing System 102 User Computing Device, Device, Client Device, Client 104 Server, Web Content Server, Web Server 106 Network 108 Database 112 Processor 114 Memory 116 Data 118 Instructions 122 User Input Component 124 Web Browser Application, Web Browser 134 Content Cache 136 Front - End Server 138 Content Cache 140 Back - End Server 142 Processor 146 Memory 148 Data 150 Instructions 152 Query Processing System 158 User Interface 160 Content 180 Network 200 Content Recommendation Model 202 Task Generation Model 204 Input Data 206 Output Data 210 Processor 212 User Interface 214 Web Browser, Browser 314 Browser Window 316 Title Bar 318 Menu Bar 320 Toolbar 322 Button 324 URL (Uniform Resource Locator) Field, URL Field 325 Main Body 326 Text 328 Hyperlink 330 Image 500 User Interface 502 Selectable Action Element 504 Overlay 506 Tag 508 Icon 510 Favorite Toggle 512 Content, Directly Recommended Content 514 Button 600 Immersive Dashboard User Interface 602 Text Header 604 Carousel Function 606 Detail Icon 608 Information Overlay 610 Panel 612 Information Overlay 700 Immersive Dashboard User Interface 702 Specified Action Element 704 Bookmark Function 706 Positive Icon 708 Negative Icon 710 Note 712 Card 714 Share Icon 800 Immersion-type dashboard user interface, immersion-type dashboard 802 Related activity search 804 Step summary panel 806 User interface prompt 808 Result panel 810 Step counter 900 Method

Claims

1. A computer-implemented method, comprising: obtaining, by a computing system, historical user data that describes historical user actions taken in one or more past user online sessions; processing, by the computing system, the historical user data to determine attributes of content associated with the user actions; determining, by the computing system, one or more ongoing tasks based on the attributes of the content associated with the user actions; determining, by the computing system, for each of the one or more ongoing tasks, a suggested content item; providing, by the computing system for display, selectable action elements for one or more ongoing tasks, the selectable action elements being configured to provide access to one or more suggested content items for the one or more ongoing tasks, the selectable action elements including an image related to the one or more ongoing tasks; A computer-implemented method comprising the above.

2. The computer-implemented method according to claim 1, wherein the attributes of the content associated with the user actions are determined based at least in part on metadata for the content.

3. The computer-implemented method according to claim 1, wherein the historical user data includes historical user web browsing data.

4. The computer-implemented method according to claim 1, wherein the one or more ongoing tasks include a product purchase task.

5. The computer-implemented method according to claim 1, wherein the one or more ongoing tasks include a cooking task.

6. receiving, by the computing system, a selection of the selectable action elements; providing, in response to receiving the selection of the selectable action elements, access to the one or more suggested content items for the one or more ongoing tasks; The computer-implemented method according to claim 1, further comprising the above.

7. A computing system comprising one or more processors, a non-transitory computer-readable storage medium storing instructions executable by the one or more processors, the instructions causing the computing system to perform operations when executed wherein the operations include obtaining historical user data that describes historical user actions taken in one or more past user online sessions, processing the historical user data to determine attributes of content associated with the user actions, determining one or more ongoing tasks based on the attributes of the content associated with the user actions, for each of the one or more ongoing tasks, determining a suggested content item, providing, for display, selectable action elements for the one or more ongoing tasks, the selectable action elements being configured to provide access to one or more suggested content items for the one or more ongoing tasks, the selectable action elements including an image related to the one or more ongoing tasks A computing system including the above. **Claim 8** Determining one or more ongoing tasks based on the attributes of the content associated with the user actions includes identifying one or more tasks from the historical user data, at least one task within the one or more tasks being ongoing, generating a list of one or more ongoing tasks by determining, for each task within the one or more tasks, whether the respective task is ongoing or completed The computing system according to claim 7, including the above. **Claim 9** Determining one or more ongoing tasks based on the attributes of the content associated with the user actions includes, for each ongoing task within the one or more ongoing tasks, accessing a continuously updated probabilistic transition graph for the respective ongoing task Identifying, at least in part based on the continuously updated probabilistic transition graph, a next step intent for each of the respective ongoing tasks; Determining, at least based on the next step intent, the suggested content items for each of the respective ongoing tasks The computing system according to claim 8, further comprising: **Claim 10** The probabilistic transition graph includes a hierarchical representation of the next step intent for the one or more ongoing tasks, at least in part based on the historical user data; The computing system according to claim 9, wherein identifying, by the computing system, the next step intent for each of the respective ongoing tasks includes the computing system selecting one or more content intents, at least in part based on the hierarchical representation of the next step intent. **Claim 11** Obtaining the historical user data that describes the historical user actions taken in the one or more past user online sessions; The computing system according to claim 7, wherein obtaining the historical user data includes extracting the historical user actions from a plurality of web pages across a plurality of web browser sessions. **Claim 12** The computing system according to claim 7, wherein the attributes of the content associated with the user action are determined at least in part based on metadata for the content. **Claim 13** The computing system according to claim 7, wherein the historical user data includes historical user web browsing data. **Claim 14** The computing system according to claim 7, wherein the one or more ongoing tasks include a product purchase task. **Claim 15** The computing system according to claim 7, wherein the one or more ongoing tasks include a cooking task. **Claim 16** Receiving, by the computing system, a selection of the selectable action element; In response to receiving the selection of the selectable action element, providing access to the one or more suggested content items for the one or more ongoing tasks The computing system according to claim 7, further comprising

17. A non-transitory computer-readable storage medium storing instructions executable by a computing system, the instructions causing the computing system to perform operations during execution, the operations comprising obtaining historical user data describing historical user actions taken in one or more past user online sessions; processing the historical user data to determine attributes of content associated with the user actions; determining one or more ongoing tasks based on the attributes of the content associated with the user actions; for each of the one or more ongoing tasks, determining a suggested content item; providing, for display, selectable action elements for the one or more ongoing tasks, the selectable action elements being configured to provide access to one or more suggested content items for the one or more ongoing tasks, the selectable action elements including images related to the one or more ongoing tasks A non-transitory computer-readable storage medium including

18. The non-transitory computer-readable storage medium according to claim 17, wherein providing the selectable action elements for display includes floating the selectable action elements on a dedicated dashboard.

19. Providing the selectable action elements for display includes obtaining current user web browser data, the current user web browser data including one or more of text content or image content currently presented to the user; selecting, from the historical user data, one or more ongoing tasks based at least in part on the one or more of the text content or the image content; floating, for display to the user, the selectable action elements for the one or more ongoing tasks overlaid on or adjacent to the current user web browser data The non-transitory computer-readable storage medium according to claim 17, comprising **Claim 20** The non-transitory computer-readable storage medium according to claim 17, wherein the attribute of the content associated with the user action is determined based at least in part on metadata for the content.

Images