Using language models to generate user interface components

Abstract

An example computing system retrieves context information from one or more background applications and applies a language model to the context information to identify one or more tasks. The computing system determines, for each of the one or more tasks, one or more associated applications, in which each of the one or more associated applications includes one or more functions for performing a respective task. The computing system assigns a respective priority score to each of the one or more tasks and generates instructions for generating a graphical user interface. The graphical user interface includes graphical components, in which each graphical component is associated with a respective task, and each graphical component is arranged within the graphical user interface based on the priority score assigned to the respective task.

Description

Docket No.: 1333-882WO01USING LANGUAGE MODELS TO GENERATE USER INTERFACE COMPONENTSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of US Provisional Patent Application No. 63 / 701,230, filed September 30, 2024, which is incorporated by reference herein in its entirety.BACKGROUND

[0002] Applications executed on computing devices may provide a wide variety of functionality to users, which may help them perform various tasks. However, users must typically interact with multiple user interface elements and / or screens of multiple applications before they are able to access such functionality and complete such tasks. Furthermore, users may find it challenging and / or time-consuming to navigate through entire applications, and may find it difficult to complete tasks due to information being stored across multiple different applications.SUMMARY

[0003] In general, techniques of this disclosure are directed to techniques for dynamically generating a custom user interface for performing tasks associated with one or more applications by at least applying a language model to context information from the one or more applications. A remote computing device (e.g., a smartphone) may include a number of applications, such as messaging applications, social media applications, web browser applications, banking applications, etc. Only after receiving explicit consent from a user, a computing system may retrieve, using one or more application programming interfaces, context information from the one or more applications, such as information relating to user actions performed within an application, notifications generated by an application, etc., while the user is away and / or while the application is running in the background of a user’s device (i.e., the application is idle, minimized, inactive, or the like). As an example, the retrieved context information may indicate a message was received from a certain contact, a draft social media post has not yet been published, a video is paused, and a funds transfer has been requested while the user has been away from their device.

[0004] The computing system may apply a machine learning model, such as a large language model, to the context information to identify one or more tasks. For example, the machineDocket No.: 1333-882WO01 learning model may identify tasks such as “finish watching video,” “send $20 to John,” “schedule the meeting with Jane,” “publish the draft social media post,” etc. The computing system may determine, for each task, at least one associated application, in which the at least one associated application includes one or more functions for performing the task. The computing system may further assign a priority score to each of the one or more tasks. That is, the computing system may determine a level of importance for each task based on, for example, user data, historical user data, and / or context information (e.g., whether the task was identified based on a notification or alert, when the notification was received, etc.). The computing system may generate instructions for generating a first graphical user interface (GUI) that includes graphical components (e.g., “widgets”) for each task. The “task widgets” may be arranged within the first GUI based on their respective priority score. That is, the first GUI, which may be scrollable, may include a plurality of task widgets, in which each task widget is arranged, for example, in an order from highest priority to lowest priority. In some examples, responsive to a user interacting with a particular task widget, additional GUIs (e.g., overlay GUIs) for the one or more associated applications may be displayed, in which the additional GUIs may include graphical components that are associated with functionality for completing the respective task.

[0005] In one example, the disclosure is directed toward a method that includes retrieving, by a computing system, context information from one or more applications, and applying, by the computing system, a machine learning model to the context information to identify one or more tasks. The method further includes determining, by the computing system, and for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task. The method further includes assigning, by the computing system, a respective priority score to each task from the one or more tasks, and generating, by the computing system, instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

[0006] In another example, the disclosure is directed toward a computing system comprising one or more processors, and one or more storage devices that store instructions. The instructions, when executed by the one or more processors, cause the one or more processors to retrieve context information from one or more applications, and apply a machine learningDocket No.: 1333-882WO01 model to the context information to identify one or more tasks. The instructions further cause the one or more processors to determine, for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task. The instructions further cause the one or more processors to assign a respective priority score to each task from the one or more tasks, and generate instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

[0007] In another example, the disclosure is directed toward a non-transitory computer- readable storage medium encoded with instructions that, when executed by one or more processors, cause one or more processors to retrieve context information from one or more applications, and apply a machine learning model to the context information to identify one or more tasks. The instructions further cause the one or more processors to determine, for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task. The instructions further cause the one or more processors to assign a respective priority score to each task from the one or more tasks, and generate instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

[0008] In another example, the disclosure is directed toward a computer program product for generating custom user interfaces for performing tasks associated with background applications. The computer program product comprises one or more instructions that, when executed by at least one processor, cause the at least one processor to retrieve context information from one or more applications, and apply a machine learning model to the context information to identify one or more tasks. The one or more instructions further cause the at least one processor to determine, for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task. The one or more instructions further cause the at least one processor to assign a respective priority score to each task from the oneDocket No.: 1333-882WO01 or more tasks, and generate instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

[0009] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE FIGURES

[0010] FIG. 1 is a conceptual diagram illustrating an example computing system for dynamically generating custom graphical user interfaces for performing one or more tasks identified based on retrieved context information, in accordance with one or more techniques of this disclosure.

[0011] FIG. 2 is a block diagram illustrating another example computing system configured to apply a machine learning module to retrieved context information to dynamically generate custom graphical user interfaces, in accordance with one or more techniques of this disclosure.

[0012] FIG. 3 A is a conceptual diagram illustrating an example training process for a machine learning module, in accordance with one or more techniques of this disclosure.

[0013] FIG. 3B is a conceptual diagram illustrating an example trained machine learning module, in accordance with one or more techniques of this disclosure.

[0014] FIG. 3C is a conceptual diagram illustrating a machine learning module configured to parse natural language input to identify tasks and determine associated applications for completing tasks, in accordance with one or more techniques of this disclosure.

[0015] FIGS. 4A-4J are conceptual diagrams illustrating examples of graphical user interfaces and graphical components for performing tasks, in accordance with one or more techniques of this disclosure.

[0016] FIG. 5 is a flowchart illustrating an example operation for dynamically generating custom graphical user interfaces for performing one or more tasks identified based on retrieved context information, in accordance with one or more techniques of this disclosure.DETAILED DESCRIPTIONDocket No.: 1333-882WO01

[0017] FIG. 1 is a conceptual diagram illustrating an example computing system for dynamically generating custom graphical user interfaces for performing one or more tasks identified based on retrieved context information, in accordance with one or more techniques of this disclosure. In the example of FIG. 1, a user 120 interacts with computing device 112 that is in communication with computing system 100. In some examples, some or all of the components and / or functionality attributed to computing system 100 may be implemented or performed by computing device 112.

[0018] In some examples, computing system 100 may be implemented on a plurality of computing devices that may include, but are not limited to, portable, mobile, or other devices, such as mobile phones (including smartphones), laptop computers, desktop computers, tablet computers, smart television platforms, server computers, mainframes, etc. In some examples, computing system 100 may represent a cloud computing system that provides one or more services via network 101. That is, in some examples, computing system 100 may be a distributed computing system.

[0019] In examples in which computing system 100 may be a distributed system, such as in the example of FIG. 1, computing system 100 may communicate with computing device 112 via network 101. Network 101 may include any public or private communication network, such as a cellular network, Wi-Fi network, a direct cell-to-satellite communication network, or other type of network for transmitting data between computing system 100 and computing device 112. In some examples, network 101 may represent one or more packet switched networks, such as the Internet. Computing device 112 may send and receive data to and from computing system 100 across network 101 using any suitable communication techniques. For example, computing system 100 and computing device 112 may each be operatively coupled to network 101 using respective network links. Network 101 may include network hubs, network switches, network routers, etc., that are operatively inter-coupled thereby providing for the exchange of information between computing device 112 and computing system 100. In some examples, network links of network 101 may be Ethernet, ATM or other network connections. Such connections may include wireless and / or wired connections.

[0020] As shown in the example of FIG. 1, computing device 112 includes one or more user interface (UI) components (“UI components 102”). UI components 102 of computing device 112 may be configured to function as input devices and / or output devices for computing device 112. UI components 102 may be implemented using various technologies. For instance, UI components 102 may be configured to receive input from user 120 through tactile, audio, and / or video feedback. Examples of input devices include a presence-sensitiveDocket No.: 1333-882WO01 display, a presence-sensitive or touch-sensitive input device (such as that shown in FIG. 1), a mouse, a keyboard, a voice responsive system, video camera, microphone or any other type of device for detecting a command from user 120. In some examples, a presence-sensitive display includes a touch-sensitive or presence-sensitive input screen, such as a resistive touchscreen, a surface acoustic wave touchscreen, a capacitive touchscreen, a projective capacitive touchscreen, a pressure sensitive screen, an acoustic pulse recognition touch screen, or another presence-sensitive technology. That is, UI components 102 of computing device 112 may include a presence-sensitive device that may receive tactile input from user 120. UI components 102 may receive indications of the tactile input by detecting one or more gestures from user 120 (e.g., when user 120 touches or points to one or more locations of UI components 102 with a finger or a stylus pen).

[0021] UI components 102 may additionally or alternatively be configured to function as an output device by providing output to user 120 using tactile, audio, or video stimuli. Examples of output devices include a sound card, a video graphics adapter card, or any of one or more display devices, such as a liquid crystal display (LCD), dot matrix display, light emitting diode (LED) display, microLED, miniLED, organic light-emitting diode (OLED) display, e- ink, or similar monochrome or color display capable of outputting visible information to user 120. Additional examples of an output device include a speaker, a haptic device, or other device that can generate intelligible output to a user. For instance, UI components 102 may present output to user 120 as a graphical user interface that may be associated with functionality provided by computing device 112. In this way, UI components 102 may present various user interfaces of applications executing at or accessible by computing device 112 (e.g., an electronic message application, an Internet browser application, etc.). User 120 may interact with a respective user interface of an application to cause computing device 112 to perform operations relating to a function provided by the application.

[0022] In some examples, UI components 102 of computing device 112 may detect two- dimensional and / or three-dimensional gestures as input from user 120. For instance, a sensor of UI components 102 may detect the user's movement (e.g., moving a hand, an arm, a pen, a stylus, etc.) within a threshold distance of the sensor of UI components 102. UI components 102 may determine a two- or three-dimensional vector representation of the movement and correlate the vector representation to a gesture input (e.g., a hand-wave, a pinch, a clap, a pen stroke, etc.) that has multiple dimensions. In other words, UI components 102 may, in some examples, detect a multidimensional gesture without requiring the user to gesture at or near a screen or surface at which UI components 102 output information for display. Instead, UIDocket No.: 1333-882WO01 components 102 may detect a multi-dimensional gesture performed at or near a sensor which may or may not be located near the screen or surface at which UI components 102 output information for display.

[0023] In the example of FIG. 1, computing system 100 includes user interface (UI) module 104. UI module 104 may perform operations described herein using hardware, software, firmware, or a mixture thereof residing in and / or executing at computing system 100. Computing system 100 may execute UI module 104 with one processor or with multiple processors. In some examples, computing system 100 may execute UI module 104 as a virtual machine executing on underlying hardware. UI module 104 may execute as one or more services of an operating system or computing platform or may execute as one or more executable programs at an application layer of a computing platform.

[0024] UI module 104, as shown in the example of FIG. 1, may be operable by computing system 100 to perform one or more functions, such as receive input and send indications of such input to other components associated with computing system 100. UI module 104 may also receive data from components associated with computing system 100. Using the data received, UI module 104 may cause other components associated with computing system 100, such as UI components 102, to provide output based on the data. For instance, UI module 104 may send data to UI components 102 of computing device 112 to display a GUI, such as GUI 116.

[0025] In general, user 120 may be provided with an opportunity to provide input to control whether programs or features of computing device 112 and / or computing system 100 can collect and make use of user information (e.g., user 120’s personal data, information about user 120’s current location, location history, activity, etc.), or to dictate whether and / or how computing device 112 and / or computing system 100 may receive content that may be relevant to user 120, such as user information retrieved from one or more applications installed at computing device 112. Other user information may include data that includes the context of user usage, either obtained from an application itself or from other sources. Examples of usage context may include breadth of share (sharing publicly, or with a large group, or privately, or a specific person), context of share, etc. When permitted by the user, additional data can include the state of the device, e.g., the location of the device, the apps running on the device, etc. In addition, certain data may be treated in one or more ways before it is stored or used by computing device 112 and / or computing system 100 so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined about the user, or a user’sDocket No.: 1333-882WO01 geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, user 120 may have control over how information is collected about them and used by computing device 112 and / or computing system 100. For example, user 120 may be prompted by computing device 112 to provide explicit consent for computing device 112 and / or computing system 100 to retrieve and / or store any or all of user 120’ s data, including the context information described herein. In some examples, an action log executed on computing device 112 may provide user 120 a ledger of activity, which may show any automations or applications running in the background of computing device 112, as well as an accurate log of all UI generator module 108 activity.

[0026] In the example of FIG. 1, graphical user interface (GUI) 116 may be an example representation of a mobile phone home screen. GUI 116 may include a plurality of user interface elements. For example, as shown in FIG. 1, GUI 116 includes user interface elements 118A-1181, which may be referred to as “widgets”. A widget may be a smaller GUI or GUI element that provides specific functionality or access to a larger application. For example, GUI 116 includes widgets 118A-1181, which may provide user 120 access to one or more applications. For example, widget 118A may be a widget for a first banking application, in which, responsive to user 120 clicking on widget 118A, computing device 112 may open the first banking application for user 120. Widget 118B, for example, may be a widget for a second banking application. Widget 118C, for example, may be a widget for a calendar application, widget 118D may be a widget for a third banking application, and widget 118E may be a widget for a messaging application. As such, computing device 112 may include one or more applications, i.e., one or more applications may be installed at computing device 112, in which the one or more applications may be accessed via one or more widgets displayed on GUI 116.

[0027] In general, an application from the one or more applications may include “one or more functions for performing a respective task.” The one or more functions may refer to functions, or functionality, e.g., capabilities or features, that are provided by the values, settings, or other data that are directly embedded into the source code of the application, rather than those that are dynamically generated or configurable at runtime. An application may include functionality provided by values, logic, etc. that are fixed, e.g., “hard-coded”, in an application’s source code, and cannot be easily changed without modifying the code itself. The one or more functions may be considered statically defined functions, or functions that are predefined at compile time or build time and do not change during execution. As such,Docket No.: 1333-882WO01“the graphical components that are associated with functionality for completing the respective task” may be considered graphical components that are associated with the statically defined capabilities or features of an application.

[0028] Computing system 100 may retrieve information associated with application functionality, e.g., via an API, from the one or more applications installed on a computing device, such as computing device 112. For example, an application may include an API that enables external applications or modules to interact with and use the data stored by the application. As such, API module 106 may retrieve data associated with the predefined or statically defined functionality of the one or more applications, e.g., an API response. As an example, a banking application may include predefined or statically defined functionality for a button that a user may interact with to have funds transferred from their bank account. API module 106 may use the banking application API to retrieve the information associated with the banking application’s functions, which may include, for example, instructions for generating the button GUI element, and a value for the current balance of the user’s bank account, but may not include all of the predefined or statically defined functionality or logic for determining and displaying the value for the current balance of the user’s bank account.

[0029] As such, the one or more applications may be considered to include a plurality of predefined functions. For example, a calculator application may include predefined functionality for performing various arithmetic and mathematical operations, a browser application may include predefined functionality for accessing and browsing the Internet, a banking application may include predefined functionality for transferring funds, etc. As such, many applications executed on computing devices may include predefined functionality for performing various tasks, such as responding to messages, scheduling appointments, booking reservations, browsing the Internet, etc. As an example, if a user wants to book a dinner reservation, they may use a dining application to reserve a table at a particular restaurant. However, the user may also need to use a calendar application to determine what date and time they can book the reservation for, use a map application to find local restaurants, use a web browser application to find reviews for a restaurant, use a messaging application to determine if any friends would like to join the dinner reservation, etc. As such, just to complete a single task, such as booking a dinner reservation, a user may have to navigate through multiple applications, which may be time-consuming and frustrating for a user.

[0030] Furthermore, users may have multiple tasks to complete at any given time, some of which may or may not be “known” to a user. That is, while a user is away from their device (e.g., while the user is asleep), or while a user is interacting with a foreground or currentDocket No.: 1333-882WO01 application, one or more applications may still run in the background of the user’s device (i.e., one or more applications may be idle, minimized, inactive, or the like). These background applications may store data pertaining to a user’s various tasks. For example, a background messaging application may receive a message from a certain contact, a background social media application may store a draft social media post that has not yet been published, a background video platform application may pause a video that the user was previously watching, etc. While some user devices may generate notifications when data is received by a single application (e.g., a notification may be generated when a message is received by a messaging application), users may benefit from custom graphical user interfaces and graphical components that help them complete tasks in a more organized, “short-cut” manner, such as when performing a task that requires functionality and context information from more than one application.

[0031] In general, with explicit consent from a user, computing system 100 may retrieve, using API module 106, context information from one or more applications, systems, modules, files, data stores, cloud services, etc. included in and / or associated with computing system 100, and / or included in and / or associated with computing device 112 in communication with computing system 100. For example, the context information may be retrieved from one or more installed applications, operating system(s), hardware modules, system settings and preferences, system logs and diagnostic tools, configuration files, metadata, associated cloud services, and the like. As such, the "context information" (retrieved with explicit user consent) described herein may refer to various types of information that provide context for how, where, and when a user may interact with a computing device to complete their tasks. That is, the "context information" described herein may include, but is not limited to, application data, application usage data, application permissions, application metadata, user data, historical user data, user preference data, user feedback data, location data, system data, environmental data, time data (e.g., when data is received by an application, timestamped data, etc.), event data, notification data (e.g., notifications generated by an application), security data, device data, device metadata, network information, connectivity information, device battery data, sensor data, and the like. For example, the retrieved context information may include data pertaining to messaging data, calendar invites, birthdays, special events, user notes, news, weather, stocks, traffic, and the like that may be relevant to a user. In some examples, context information may be retrieved by computing system 100 continuously or periodically. In some examples, the context information may be retrieved during a period of time in which the user is away from their device (e.g., the device was turned off, the user wasDocket No.: 1333-882WO01 sleeping, etc.). That is, in some examples, the computing device may be in an "active state," in which a user is actively interacting with the computing device, or in a "inactive state," in which the user is not actively interacting with the computing device. In some examples, the retrieved context information may include data that pertains to a time period in which the computing device was in an inactive state, e.g., data for application notifications that the user received while sleeping.

[0032] In accordance with techniques of this disclosure, computing system 100 may include a user interface generator module 108 configured to dynamically generate custom user interfaces and graphical components for performing tasks associated with one or more applications. In general, with explicit consent from user 120, computing system 100 may retrieve, via application programming interface (API) module 106, context information from the one or more applications, such as the applications associated with widgets 118A-1181 on GUI 116. In general, with explicit consent from user 120, user interface generator module 108 may run continuously and be configured to monitor the content of one or more applications and / or user activity. For example, with explicit consent from user 120, user interface generator module 108 may run continuously in the background of computing device 112 and be configured to monitor the content of one or more applications executing at computing device 112 (e.g., in the background and / or foreground of computing device 112) and / or user activity within computing device 112. As such, API module 106 receives explicit consent from user 120 to gather information from user 120 and the one or more applications installed at computing device 112. In general, user interface generator module 108 may continuously retrieve and analyze context information from computing device 112, again provided that user 120 has given explicit permission for computing system 100 to do so.

[0033] In general, API module 106, which can be considered an API library, may include multiple APIs that can be used to access one or more application APIs. In some examples, API module 106 may provide information about user interface elements, events, and actions to assistive technologies (e.g., screen readers, magnification gestures, switch devices, etc.) provided by computing system 100 and / or computing device 112. In some examples, API module 106 may be configured to enable the exchanging of data in a standardized format. For example, API module 106 may support REST (Representational State Transfer), which is a widely-used architectural style for building APIs that use HTTP (Hypertext Transfer Protocol) to exchange data between applications.

[0034] In some examples, API module 106 may be configured to generate a stream of accessibility events as the user interacts with computing device 112 and applications executedDocket No.: 1333-882WO01 on computing device 112. In some examples, these events may represent actions and changes in a user interface, such as button presses, text changes, and screen transitions. With explicit consent from user 120, user interface generator module 108 may receive and analyze these events to better understand how user 120 interacts with applications installed on computing device 112.

[0035] API module 106 may be configured to retrieve accessibility actions from applications executed on computing device 112. “Accessibility actions” may refer to different types of inputs that can be detected at a location associated with a UI component 102, such as mechanical inputs (e.g., a clicking of a button, a swiping of a screen, etc.), audio input (e.g., verbal command), or gesture control (e.g., triple tapping on a screen, hand wave, assistive gestures, etc.). As such, accessibility actions may provide users the ability to interact with an application or user interface element in multiple ways according to their needs. In some examples, with explicit consent from user 120, computing system 100 may determine which accessibility actions are frequently performed by user 120 when interacting with a GUI or application such that new user interfaces and graphical components generated by user interface generator module 108 can be better tailored for user 120’s needs. In some examples, the information retrieved by API module 106 from computing device 112 may be stored by computing system 100 to identify potential accessibility issues and / or better understand how user 120 interacts with computing device 112. In some examples, user interface generator module 108 may use information retrieved from computing device 112 to determine the format, size, color scheme, accessibility features, or any other features to include in the instructions (e.g., code) for generating new graphical user interfaces and / or components for performing tasks. In some examples, user interface generator module 108 may also provide users the ability to configure various accessibility and / or display options according to their needs. For example, user 120 may be able to adjust the user interface elements of a GUI, such as text size, enable color correction, set up magnification gestures, and configure gesturebased navigation.

[0036] In some examples, the context information retrieved by API module 106 includes application data, such as a set of instructions (e.g., code, data, information, etc.) associated with one or more functions (e.g., application functionality). The context information may include user data, such as user account data, information relating to user actions performed within an application, etc. The context information may include system data, environmental data, time data (e.g., when data is received by an application, timestamped data, etc.), event data, notification data (e.g., notifications generated by an application), security data,Docket No.: 1333-882WO01 application and / or device metadata, etc. As an example, the retrieved context information may indicate a message was received from a certain contact by a background messaging application, a draft social media post has not yet been published on a background social media application, a video is paused in a background video platform application, and the device battery percentage has dropped below a threshold. In some examples, the retrieved context information may be pre-processed by computing system 100. In some examples, the retrieved context information may be in a data format that can be parsed by a machine learning model, such as a language model (e.g., the data may be in a structured or semistructured data format).

[0037] In general, user interface generator module 108 may send information (e.g., the retrieved context information) to machine learning module 110 only if computing system 100 receives permission from the user of computing device 112 to send the information. For example, in situations discussed in which computing system 100 and / or computing device 112 may collect, transmit, or may make use of personal information about a user (e.g., location information, financial information, etc.), the user may be provided with an opportunity to control whether programs or features of computing system 100 can collect user information (e.g., information about a user’s social network, a user’s social actions or activities, a user’s profession, a user’s preferences, or a user’s current location), or to control whether and / or how computing system 100 and / or computing device 112 may store and share user information. Thus, the user may have control over how information is collected about the user and stored, transmitted, and / or used in accordance with techniques of this disclosure.

[0038] User interface generator module 108 may apply machine learning module 110, which may include a language model configured to perform natural language processing techniques, to the context information retrieved by API module 106 to identify one or more tasks. In general, machine learning module 110 may parse through input including any amount of data, i.e., machine learning module 110 may identify any number of tasks in the retrieved context information. For example, machine learning module 110 may identify tasks such as “finish watching video,” “send $20 to John,” “schedule meeting with Jane,” and “publish the draft social media post” based on the retrieved context information. User interface generator module 108 may determine, for each task, at least one associated application, in which the at least one associated application includes one or more functions for performing the task.

[0039] In some examples, user interface generator module 108 may also use the context information retrieved by API module 106 to interpret and understand the functionality provided by the one or more applications installed at computing device 112. For example,Docket No.: 1333-882WO01 user interface generator module 108 may use at least a portion of the retrieved context information to contextualize other portions of the retrieved context information. As one example, context information retrieved from a messaging application may include a message received from John D. that states, “Hi, can you send me $20?”. User interface generator module 108 may apply machine learning module 110 to the retrieved context information, in which machine learning module 110 may identify the “send $20 to John” task. User interface generator module 108 may also use other retrieved context information to interpret and understand functionality provided by other applications, so as to determine which application(s) provide functions that can perform the identified tasks. In this example, user interface generator module 108 may determine, based on the retrieved context information, that computing device 112 includes a banking application that provides functions associated with the task of sending money. As such, user interface generator module 108 may further contextualize the text message from John D. and / or the identified task of sending $20 to John D., as user interface generator module 108 may determine that performing the identified task involves actions such as transferring funds from user 120’s bank account to John D.’s bank account via the banking application. In this way, user interface generator module 108 may determine, for each of the one or more tasks, one or more associated applications, in which each of the one or more associated applications includes one or more functions for performing a respective task. Continuing the example, user interface generator module 108 may determine the at least one associated application to be the banking application, in which the banking application includes functionality for performing the task of sending $20 to John D. Furthermore, in some examples, with explicit consent from user 120, user interface generator module 108 may use the context information retrieved from the one or more applications to generate suggested data, prepopulate entry fields, etc. for performing tasks. As an example, computing system 100 may generate instructions for generating an overlay GUI for the banking application, in which the GUI entry fields may be prepopulated with data based on the retrieved context information. For instance, a text entry field may be prepopulated with John D.’s username “John Doe” (e.g., based on a list of user 120’s trusted contacts stored within the banking application), and another text entry field may be prepopulated with “$20.00” (e.g., based on the content of the text message received from John. D).

[0040] In general, user interface generator module 108 may generate instructions for generating a central or main GUI (e.g., a “task list GUI”) that includes graphical components for each task (e.g., a “task widget”). For example, continuing the example above, userDocket No.: 1333-882WO01 interface generator module 108 may generate a GUI (e.g., a “task list GUI”) that includes a task widget for the task of sending $20 to John D. In some examples, user interface generator module 108 may further assign a priority score to each of the one or more tasks. That is, user interface generator module 108 may determine a level of importance for each task based on, for example, user data, historical user data (e.g., system feedback data), and / or the retrieved context information (e.g., whether the task was identified based on a notification or alert, when the notification was received, etc.). Thus, in some examples, an arrangement of the task widgets within the task list GUI may be based on their respective priority scores.

[0041] In some examples, responsive to user 120 interacting with a particular task widget, additional GUIs for the one or more associated applications may be displayed, in which the additional GUIs may include graphical components that are associated with functionality for completing the respective task. For example, user 120 may interact with the “Send $20 to John” task widget to have an overlay GUI for the banking application be displayed. The user may then interact with the graphical components included in the overlay GUI, such as a button associated with functionality for initiating a funds transfer.

[0042] As such, with explicit user consent, computing system 100 may help users perform tasks using context information retrieved from computing device 112, in which computing system 100 may provide users with a mechanism to “shortcut” the complexity of performing various actions for various tasks. That is, instead of requiring users to navigate through multiple user interfaces of multiple applications to complete a task, computing system 100 may generate instructions for generating a GUI that presents and organizes tasks in a way that is similar to a “to-do list.” Additionally, the tasks identified by computing system 100 may extend beyond tasks based on notification data; for example, some example identified tasks may be “resume watching video,” “continue reading blog post,” “schedule meeting that occurs bi-weekly,” (e.g., based on historical data retrieved from a calendar application), “catch up with Joe M.,” (e.g., based on Joe M. being a contact that a user frequently messages), etc. In this way, computing system 100 may help a user in completing tasks that may otherwise be forgotten by the user, such as when the task pertains to an application that is running in the background, is idle, minimized, inactive, does not generate notifications, etc. Furthermore, by linking associated applications to identified tasks and by providing suggested data, a user may complete their tasks more easily and efficiently.

[0043] In this respect, various aspects of the techniques described in this disclosure may facilitate better user experience with applications executing on user devices. Specifically, a main task list GUI with smaller and more organized task widgets that provide users access toDocket No.: 1333-882WO01 functionality of larger applications may reduce the amount of time and effort required by a user to access such functionality when trying to complete tasks. The techniques described may also provide more assistance to users with disabilities when interacting with devices and applications. Additionally, users may find that organizing tasks based on priority is helpful for completing tasks in a less convoluted manner.

[0044] FIG. 2 is a block diagram illustrating another example computing system configured to apply a machine learning module to retrieved context information to dynamically generate custom graphical user interfaces, in accordance with one or more techniques of this disclosure. As shown in the example of FIG. 2, computing system 200 includes processors 224, one or more communication channels 230, one or more user interface components (UIC) 232, one or more communication units 228, and one or more storage devices 238. Storage devices 238 of computing system 200 may include user interface module 204, and user interface generator module 208. As shown in the example of FIG. 2, user interface generator module 208 further includes API module 206, machine learning module 210, and instructions storage 222.

[0045] Some or all of the components and / or functionality attributed to computing system 200 may be implemented or performed by a computing device that may be in communication with computing system 200. In other examples, computing system 200 may be considered a computing device, such as a user computing device (e.g., a mobile phone). Computing system 200, user interface module 204, user interface generator module 208, API module 206, machine learning module 210, and user interface (UI) components 202 may be similar if not substantially similar to computing system 100, user interface module 104, user interface generator module 108, API module 106, machine learning module 110, and user interface (UI) components 102 of FIG. 1, respectively.

[0046] The one or more communication units 228 of computing system 200, for example, may communicate with external devices by transmitting and / or receiving data at computing system 200, such as to and from remote computer systems or computing devices. Example communication units 228 include a network interface card (e.g., such as an Ethernet card), an optical transceiver, a radio frequency transceiver, or any other type of device that can send and / or receive information. Other examples of communication units 228 may be devices configured to transmit and receive Ultrawideband®, Bluetooth®, GPS, 3G, 4G, and Wi-Fi®, etc. that may be found in computing devices, such as mobile devices and the like.

[0047] As shown in the example of FIG. 2, communication channels 230 may interconnect each of the components as shown for inter-component communications (physically,Docket No.: 1333-882WO01 communicatively, and / or operatively). In some examples, communication channels 230 may include a system bus, a network connection (e.g., to a wireless connection), one or more inter-process communication data structures, or any other components for communicating data between hardware and / or software locally or remotely.

[0048] One or more I / O devices 234 of computing system 200 may receive inputs and generate outputs. Examples of inputs are tactile, audio, kinetic, and optical input, to name only a few examples. Input devices of I / O devices 234, in one example, may include a touchscreen, a touchpad, a mouse, a keyboard, a voice responsive system, a video camera, buttons, a control pad, a microphone or any other type of device for detecting input from a human or machine. Output devices of I / O devices 234, may include, a sound card, a video graphics adapter card, a speaker, a display, or any other type of device for generating output to a human or machine.

[0049] User interface module 204, user interface generator module 208, API module 206, machine learning module 210, and instructions storage 222 (hereinafter “modules 204-222”) may perform operations described herein using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and executing on computing system 200 or at one or more other computing devices (e.g., a cloud-based application - not shown). For example, some or all of modules 204-222 may be included in and executable on a local computing device, such as computing device 112 of FIG. 1. As such, the techniques described herein may all be implemented locally on a computing device.

[0050] Computing system 200 may execute one or more of modules 204-222, with one or more processors 224 or may execute any or part of one or more of modules 204-222 as or within a virtual machine executing on underlying hardware. One or more of modules 204-222 may be implemented in various ways, for example, as a downloadable or pre-installed application, remotely as a cloud application, or as part of the operating system of computing system 200. Other examples of computing system 200 that implement techniques of this disclosure may include additional components not shown in FIG. 2.

[0051] In the example of FIG. 2, one or more processors 224 may implement functionality and / or execute instructions within computing system 200. For example, one or more processors 224 may receive and execute instructions that provide the functionality of UIC 232, communication units 228, one or more storage devices 238 and an operating system to perform one or more operations as described herein. For example, one or more processors 224 may receive and execute instructions that provide the functionality of some or all of modules 204-222 to perform one or more operations and various functions described herein.Docket No.: 1333-882WO01The one or more processors 224 include a central processing unit (CPU). Examples of CPUs include, but are not limited to, a digital signal processor (DSP), a general-purpose microprocessor, a tensor processing unit (TPU); a neural processing unit (NPU); a neural processing engine; a core of a CPU, VPU, GPU, TPU, NPU or another processing device, an application specific integrated circuit (ASIC), a field programmable logic array (FPGA), or other equivalent integrated or discrete logic circuitry, or other equivalent integrated or discrete logic circuitry.

[0052] One or more storage devices 238 within computing system 200 may store information, such as information retrieved from a user computing device, or other data discussed herein, for processing during the operation of computing system 200. In some examples, one or more storage devices of storage devices 238 may be a volatile or temporary memory. Examples of volatile memories include random access memories (RAM), dynamic random-access memories (DRAM), static random-access memories (SRAM), and other forms of volatile memories known in the art. Storage devices 238, in some examples, may also include one or more computer-readable storage media. Storage devices 238 may be configured to store larger amounts of information for longer terms in non-volatile memory than volatile memory. Examples of non-volatile memories include magnetic hard disks, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage devices 238 may store program instructions and / or data associated with the modules 204-222 of FIG. 2.

[0053] In general, with explicit consent from a user, computing system 200 may retrieve, using API module 206, context information from one or more applications, systems, modules, files, data stores, cloud services, etc. included in and / or associated with computing system 200, and / or included in and / or associated with computing device(s) in communication with computing system 200. For example, the context information may be retrieved from one or more installed applications, operating system(s), hardware modules, system settings and preferences, system logs and diagnostic tools, configuration files, metadata, associated cloud services, and the like. As such, the context information (retrieved with explicit user consent) may include, but is not limited to, application data, application usage data, application permissions, user data, user preference data, user feedback data, location data, system data, device data, network information, connectivity information, device battery data, sensor data, environmental data, time data, event data, notification data, and security data. The retrievedDocket No.: 1333-882WO01 context information may be referred to herein as “input data” that may be processed, stored, analyzed, transformed, etc. by computing system 200.

[0054] UI module 204 may receive information and instructions from one or more associated platforms, operating systems, applications, and / or services executing at the computing device (e.g., user interface generator module 208) for generating one or more files each comprising a set of instructions. In some examples, a set of instructions may include instructions for generating a GUI, such as a task list GUI, including a first plurality of graphical components, in which each graphical component from the first plurality of graphical components is associated with a respective task, e.g., the first plurality of graphical components may be considered a plurality of task widgets. In some examples, a set of instructions may include instructions for generating one or more additional graphical user interfaces, such as overlay GUIs for associated application(s) determined by computing system 200 to include function(s) for performing the identified task(s). In some examples, UI module 204 may act as an intermediary between the one or more associated platforms, operating systems, applications, and / or services executing at the computing device and various output devices of the computing device (e.g., speakers, LED indicators, vibrators, etc.) to produce output (e.g., graphical, audible, tactile, etc.) with the computing device.

[0055] In some examples, user interface generator module 208 may be implemented on a computing device in various ways. For example, user interface generator module 208 may be implemented as a downloadable or pre-installed application or “app.” In another example, user interface generator module 208 may be implemented as part of an operating system of a computing device.

[0056] Instructions storage 222 is a storage repository that may store, with explicit user consent, context information retrieved by API module 206. In general, the context information may include API response data. For example, the context information may be retrieved from one or more applications, in which the context information may include information associated with the one or more functions included in the one or more applications, e.g., the statically defined capabilities or features of an application. For example, an application may include an API that enables external applications or modules to interact with and use the data stored by the application. As such, API module 206 may retrieve data associated with the functionality of the one or more applications, e.g., an API response. As an example, a banking application may include functionality for displaying a current balance of a user’s bank account. API module 206 may use the banking application API to retrieve the information associated with the functionality, which may include, forDocket No.: 1333-882WO01 example, a value for the current balance of the user’s bank account, but may not include all of the predefined or statically defined functionality or logic for determining and displaying the value for the current balance of the user’s bank account. The context information may include various types of information associated with the application, e.g., the context information may include notification data from the banking application. In some examples, the context information may additionally or alternatively include system data, environmental data, time data (e.g., when data is received by an application, timestamped data, etc.), event data, notification data (e.g., notifications generated by an application), security data, application and / or device metadata, etc. Information may be stored in instructions storage 222 for use by other modules of user interface generator module 208, such as machine learning module 210. In some examples, instructions storage 222 may operate, at least in part, as a cache for instructions retrieved from a computing device (e.g., using one or more communication units 228) or other computing devices. In general, instructions storage 222 may be configured as a database, flat file, table, or other data structure stored within storage device 238. In some examples, instructions storage 222 is shared between various modules executing at computing system 200 (e.g., between one or more of modules 204-222 or other modules not shown in FIG. 2). In other examples, a different data repository is configured for a module executing at computing system 200 that requires a data repository. Each data repository may be configured and managed by different modules and may store data in a different manner. In some examples, computing system 200 may receive and store information, such as the context information, from a computing device over a specified period of time.

[0057] In general, machine learning module 210 may be configured to interpret input data, such as the context information, received or retrieved by computing system 200, so as to identify one or more tasks. The retrieved context information may be in various data formats that may or may not be readable to machine learning module 210 (e.g., a language model included in machine learning module 210). In some examples, the retrieved context information may be in data formats including, but not limited to, JavaScript Object Notation (JSON), extensible Markup Language (XML), Ain’t Markup Language (YAML), INI files, plain text, Comma-Separated Values (CSV), Structured Query Language (SQL), and Non- Structured Query Language (NoSQL). In some examples, the context information may be in binary formats, database records, highly specialized formats, etc. that may not be immediately readable to machine learning module 210. In these examples, the context information may be converted, manipulated, transformed, etc. into a readable format, such as structured or semi -structured text, and / or metadata may be used to interpret the contextDocket No.: 1333-882WO01 information. For example, machine learning module 210 may convert any input or context information to XML, or other structured text types, such as, but not limited to, HTML, JSON, CSV, INI Files, etc. In this way, the context information and any other input received by user interface generator module 208 can be provided to ML module 210 in a standardized and / or readable format. Furthermore, in some examples, machine learning module 210 may determine the type of information to include in the structured text representation. More specifically, machine learning module 210 may analyze various application functionality, capabilities, and attributes included in the context information stored in instructions storage 222, such as content descriptions, roles, states, actions, and / or other relevant properties of user interface elements.

[0058] As such, in some examples, the retrieved context information may be preprocessed. Preprocessing techniques may include extracting one or more additional features from raw data. For example, feature extraction techniques may be applied to the user input or retrieved instructions to generate one or more new, additional features.

[0059] In general, machine learning module 210 may employ a large language model (LLM) that can interpret the context information and identify one or more tasks. In some examples, machine learning module 210 may implement other machine-learned models that may be used in place of or in conjunction with an LLM model, such as those described with respect to FIGS. 3A, 3B, and 3C. Machine learning module 210 may employ an LLM that can infer indications of natural language input (e.g., natural language text retrieved from a messaging application). In some examples, machine learning module 210 may analyze portions of the retrieved context information to interpret and understand other portions of the retrieved context information. Machine learning module 210 may analyze the context information to interpret and understand the functionality included in computing system 200 and / or included in a device in communication with computing system 200, so as to determine one or more actions for completing an identified task. For example, machine learning module 210 may determine that for an identified task of responding to a text message received by a messaging application, a first action may be to check a calendar application, and a second action may be to send a message back through the messaging application. In this example, the first action may be determined based on the content of the received text message. As such, machine learning module 210 may determine, for each of the one or more tasks, one or more associated applications, in which each of the one or more associated applications includes one or more functions for performing a respective task. Continuing the example, machineDocket No.: 1333-882WO01 learning module 210 may determine, for the task of responding to a text message, an associated calendar application and an associated messaging application.

[0060] The techniques of the present disclosure may be implemented by or otherwise executed on one or more computing devices (e.g., computing device 112 of FIG. 1). Examples of such computing devices include user computing devices (e.g., laptops, desktops, and mobile computing devices such as tablets, smartphones, wearable computing devices, etc.); embedded computing devices (e.g., devices embedded within a vehicle, camera, image sensor, industrial machine, satellite, gaming console or controller, or home appliance such as a refrigerator, thermostat, energy meter, home energy manager, smart home assistant, etc.); other computing devices; or combinations thereof. Computing system 200 and / or a computing device that implements machine learning module 210 or other aspects of the present disclosure may include a number of hardware components that enable the performance of the techniques described herein.

[0061] Machine learning module 210 may further assign a priority score to each of the one or more tasks. For example, machine learning module 210 may assign priority scores to tasks based on the input data pertaining to each task, which may include, but is not limited to, application data, historical user data, user feedback data, user preference data, notification data, time data, CPU usage data, memory usage data, device battery data, etc. For example, the priority score may be based on, but is not limited to, the content of a message (e.g., based on text such as “Urgent,” “Emergency,” etc.), when the task was determined (e.g., when a message was received), historical user data (e.g., data indicating which tasks or types of tasks a user historically completed in an amount of time less than a threshold, such as within 5 minutes, 30 minutes, 1 hour, 1 day, etc. after the task was identified), user feedback data (e.g., a user interacting with lower priority task widgets before interacting with higher priority task widgets), whether a particular task will consume a threshold amount of computational power, memory, battery power, network bandwidth, etc. (e.g., tasks that require memory usage over a predetermined threshold may be assigned a lower priority score), and the like, and / or combinations thereof. In some examples, the respective priority scores assigned to the different tasks may be determined based on different types of context information (i.e., the priority score assigned to a specific task may be based on the context information pertaining to that specific task).

[0062] As another example, a priority score for a task of responding to a text message may be based on whether the text message was received by a contact that a user frequently messages. More specifically, in some examples, the context information may include application dataDocket No.: 1333-882WO01 indicative of one or more user contacts, in which machine learning module 210 may determine, based on historical user data, one or more user contacts associated with a messaging frequency greater than a threshold. For example, machine learning module 210 may determine, based on historical user data retrieved from a messaging application, that the user messages John Doe with a frequency greater than a predetermined threshold. In some examples, the threshold may be determined based on the context information, e.g., based on the top ten contacts that the user most frequently messages. For example, machine learning module 210 may determine that the user messages John Doe on average twice a day, in which John Doe is a top ten contact that the user most frequently messages. As such, any tasks identified based on messages received from John Doe may be assigned higher priority scores. In some examples, an arrangement of a task widget within a task list GUI may be based on the respective priority score assigned to the respective task. Thus, continuing the example, the task widget for responding to a text message received from John Doe may be arranged at a top portion of the task list GUI, e.g., the task widget for responding to John Doe may be arranged above task widgets for responding to text messages received from non-top ten contacts. In some examples, computing system 200 may generate instructions for generating the task list GUI, in which the task list GUI includes at least one graphical component associated with the one or more user contacts associated with the messaging frequency greater than the threshold. That is, in some examples, the task list GUI may include a widget including cards or icons for the user’s top ten contacts. In some examples, the list of contacts associated with the messaging frequency greater than the threshold, and / or any graphical components associated with a user’s top contacts, may be editable by the user (e.g., one or more contacts may be added or removed by the user).

[0063] As described in the example above, in some examples, an arrangement of a task widget within the task list GUI may be based on the respective priority score assigned to the respective task. In general, however, the GUIs and / or graphical components generated by user interface generator module 208 may be user-configurable, e.g., a user may rearrange, reorder, re-position, re-size, etc. any GUIs and / or graphical components described herein. In some examples, data indicative of edits made to the GUIs and / or graphical components may be provided as feedback data to computing system 200, so as to better understand the priorities of different tasks, user display preferences, etc.

[0064] In some examples, UI components 232 of computing system 200 may receive an indication of a gesture detected at a location of I / O device(s) 234 that corresponds to a graphical component from the first plurality of graphical components, e.g., a user may tap aDocket No.: 1333-882WO01 display screen to select a particular task widget. Based on the indication of the gesture, e.g., the particular task widget selected, computing system 200 may generate instructions for generating one or more additional GUIs, in which each of the one or more additional GUIs is associated with one of the one or more associated applications determined for the particular task. For example, based on a user selecting a task widget for responding to a message from John Doe, computing system 200 may generate instructions for generating an overlay GUI for the messaging application. An “overlay GUI” may be considered a graphical element or GUI that appears on top of (e.g., “overlays”) a main GUI (e.g., the dimensions of the overlay GUI may be smaller than the dimensions of the main GUI), in which the overlay GUI may provide a “window” for an associated application. That is, in some examples, an overlay GUI may be a minimized application GUI. In some examples, the overlay GUI may display information and graphical components associated with application functionality, such that a user may access the functionality without having to navigate away from the current screen of the main GUI. As such, each of the one or more additional GUIs (e.g., overlay GUIs) may include a subset of graphical components from a second plurality of graphical components associated with the one or more functions for performing a respective task. That is, computing system 200 may generate instructions for generating a main task list GUI, which may further include task widgets for the identified tasks that are arranged based on the respective priority scores assigned to the identified tasks. Computing system 200 may determine one or more associated applications for each task. As such, when a user interacts with a particular task widget, overlay GUIs for the one or more associated applications may be presented, in which the overlay GUIs may include graphical components associated with application functionality that can perform the particular task. Furthermore, computing system 200 may apply machine learning module 210 to the context information to determine suggested data, in which the graphical components associated with the application functionality may include the suggested data. For example, text entry fields included in an overlay GUI may be prepopulated with suggested data, suggested data may be presented as buttons that a user can select, etc.

[0065] In some examples, multiple task list GUIs may be generated, in which each task GUI may correspond to a specific category of tasks. For example, a first task list GUI may correspond to a "Catch Up" category, and may include task widgets generated based on, e.g., context information associated with one or more of conversation data, notification data, calendar invites, birthdays, special events, journaling, news, weather, stocks, traffic, and the like. In some examples, the first task list GUI and task widgets included therein may beDocket No.: 1333-882WO01 generated based on information received during a period of time in which the user was away from their device (e.g., the device was turned off, the user was sleeping, etc. As another example, a second task list GUI may correspond to an "App Sprint" category, and may include task widgets generated based on applications that a user may frequently navigate through to complete various tasks (e.g., applications that a user may navigate through frequently to complete daily tasks). In some examples, a task widget and any associated application overlay GUI(s) may be designed to guide a user through the application to complete a regular, or daily, task. In some examples, the second task list GUI and task widgets included therein may be generated based on context information and / or historical user data indicative of the applications that a user frequently interacts with (e.g., applications a user interacts with daily or weekly). As another example, a third task list GUI may correspond to an "Automate" category, and may include task widgets generated based on identified user routines. That is, the third task list GUI and task widgets included therein may be generated based on context information and / or historical user data indicative of actions that a user frequently performs. As one example, based on the retrieved context information that indicates the user has a "date night" scheduled on their calendar, a task widget may be generated that provides suggested actions for the user, such as to play romantic music via an associated music streaming application, and dim the user's kitchen lights via a home controls application. As another example, the retrieved context information and / or historical user data may indicate that a user frequently turns on their phone's low power mode. In this example, a task widget may be generated to remind the user to turn on their phone's low power mode, and, responsive to the user interacting with the task widget, a phone settings application overlay GUI may be displayed, in which the user may easily turn on low power mode. As another example, a fourth task list GUI may correspond to a "Planner" category, and may include task widgets generated based on signals indicating a follow-up task may need to be performed. That is, the fourth task list GUI and task widgets included therein may be generated based on context information indicative of various events. For example, the retrieved context information may indicate a user received a ticket to an event via their email. In this example, a task widget may be generated to remind the user of the event, and / or may suggest action items that help the user plan the event. For example, responsive to the user interacting with the task widget, a rideshare application overlay GUI may be displayed, in which the user may easily schedule transportation to and from the event.

[0066] In this way, the techniques described herein may dynamically generate user interfaces and graphical components for performing various tasks, in which the user interfaces andDocket No.: 1333-882WO01 graphical components may be tailored to a user’s preferences and needs. By displaying graphical components and information for the various tasks in a main GUI based on priority, the techniques described herein may help users navigate their tasks in a more organized manner. Furthermore, by providing associated application overlay GUIs and suggested data, the techniques described herein may reduce the amount of time it takes for users to complete a single task, as user may no longer be required to navigate through multiple user interfaces of multiple applications to access relevant information and / or application functionality, and instead may complete their tasks through a few simple UI interactions.

[0067] FIG. 3 A is a conceptual diagram illustrating an example training process for a machine learning module, in accordance with one or more techniques of this disclosure. In some examples, computing device 112 of FIG. 1 may store and implement machine learning module 310 locally (i.e., on-device). Thus, in some examples, machine learning module 310 can be stored at and / or implemented locally by an embedded device or a user computing device such as a mobile device. Output data obtained through local implementation of machine learning module 310 at the embedded device or the user computing device can be used to improve performance of the embedded device or the user computing device (e.g., an application implemented by the embedded device or the user computing device). Machine learning module 310 described herein can be trained at a training computing system, and then provided for storage and / or implementation at one or more computing devices, such as computing device 112 of FIG. 1. In some examples, training process 340 executes locally at computing system 100 of FIG. 1. However in some examples, training process 340 can be included in or separate from any computing system that implements machine learning module 310.

[0068] In general, machine learning module 310 may be or include one or more inference models, i.e., one or more trained machine learning models that can be used to make predictions based on new, unseen data. Machine learning module 310 may “infer” conclusions or outputs, which may be predictions, classifications, recommendations, or other types of decision-making. Machine learning module 310 may be trained according to one or more of various different training types or techniques. For example, in some examples, machine learning module 310 may be trained by training process 340 of FIG. 3 A.

[0069] As further shown in the example of FIG. 3 A, in some examples, machine learning module 310 may be trained on training data 331 that may include input data 333 that has labels 337. The training process shown in FIG. 3A is one example training process; other training processes may be used as well. In general, during training process 340, machineDocket No.: 1333-882WO01 learning module 310 may learn patterns from training data 331, and training process 340 may optimize parameters for machine learning module 310 to minimize prediction errors.

[0070] Training data 331 can include, upon user permission for use of such data for training, anonymized usage logs of sharing flows, e.g., content items that were shared together, bundled content pieces already identified as belonging together, e.g., from entities in a knowledge graph, etc. In some examples, training data 331 can include examples of input data 333 that have been assigned labels 337 that correspond to output data 335.

[0071] In some examples, machine learning module 310 can be trained by optimizing an objective function, such as objective function 339. For example, in some examples, objective function 339 may be or include a loss function that compares (e.g., determines a difference between) output data generated by the model from the training data and labels (e.g., groundtruth labels) associated with the training data. For example, the loss function can evaluate a sum or mean of squared differences between output data 335 and the labels. In some examples, objective function 339 may be or include a cost function that describes a cost of a certain outcome or output data. Other examples of objective function 339 can include marginbased techniques such as, for example, triplet loss or maximum-margin training.

[0072] One or more of various optimization techniques can be performed to optimize objective function 339. For example, the optimization technique(s) can minimize or maximize objective function 339. Example optimization techniques include Hessian-based techniques and gradient-based techniques, such as, for example, coordinate descent; gradient descent (e.g., stochastic gradient descent); subgradient methods; etc. Other optimization techniques include black box optimization techniques and heuristics.

[0073] In some examples, backward propagation of errors can be used in conjunction with an optimization technique (e.g., gradient based techniques) to train machine learning module 310 (e.g., when a machine-learned model is a multi-layer model such as an artificial neural network). For example, an iterative cycle of propagation and model parameter (e.g., weights) update can be performed to train machine learning module 310. Example backpropagation techniques include truncated backpropagation through time, Levenberg- Marquardt backpropagation, etc.

[0074] In some examples, machine learning module 310 described herein can be trained using unsupervised learning techniques. Unsupervised learning can include inferring a function to describe hidden structure from unlabeled data. For example, a classification or categorization may not be included in the data. Unsupervised learning techniques can be usedDocket No.: 1333-882WO01 to produce machine-learned models capable of performing clustering, anomaly detection, learning latent variable models, or other tasks.

[0075] Machine learning module 310 can be trained using semi-supervised techniques which combine aspects of supervised learning and unsupervised learning. Machine learning module 310 can be trained or otherwise generated through evolutionary techniques or genetic algorithms. In some examples, machine learning module 310 described herein can be trained using reinforcement learning. In reinforcement learning, an agent (e.g., model) can take actions in an environment and learn to maximize rewards and / or minimize penalties that result from such actions. Reinforcement learning can differ from the supervised learning problem in that correct input / output pairs are not presented, nor sub-optimal actions explicitly corrected.

[0076] In some examples, one or more generalization techniques can be performed during training to improve the generalization of machine learning module 310. Generalization techniques can help reduce overfitting of machine learning module 310 to the training data. Example generalization techniques include dropout techniques; weight decay techniques; batch normalization; early stopping; subset selection; stepwise selection; etc.

[0077] In some examples, machine learning module 310 described herein can include or otherwise be impacted by a number of hyperparameters, such as, for example, learning rate, number of layers, number of nodes in each layer, number of leaves in a tree, number of clusters; etc. Hyperparameters can affect model performance. Hyperparameters can be hand selected or can be automatically selected through application of techniques such as, for example, grid search; black box optimization techniques (e.g., Bayesian optimization, random search, etc.); gradient-based optimization; etc. Example techniques and / or tools for performing automatic hyperparameter optimization include Hyperopt; Auto-WEKA; Spearmint; Metric Optimization Engine (MOE); etc.

[0078] In some examples, various techniques can be used to optimize and / or adapt the learning rate when the model is trained. Example techniques and / or tools for performing learning rate optimization or adaptation include Adagrad; Adaptive Moment Estimation (ADAM); Adadelta; RMSprop; etc.

[0079] In some examples, transfer learning techniques can be used to provide an initial model from which to begin training of machine learning module 310 described herein. In some examples, transfer learning involves reusing a model and its model parameters obtained while solving one problem and applying it to a different but related problem. Models trained on very large data sets may be retrained or fine-tuned on additional data. Often, all modelDocket No.: 1333-882WO01 designs and their parameters on a source model are copied except output layer(s). The output layers(s) are often called the head, and other layers are often called the base. The source parameters may be considered to contain the knowledge learned from the source dataset and this knowledge may also be applicable to a target dataset. Fine-tuning may include updating the head parameters with the body parameters being fixed or updated in a later step.

[0080] In some examples, machine learning module 310 may be trained in an offline fashion or an online fashion. In offline training (also known as batch learning), machine learning module 310 is trained on the entirety of a static set of training data. In online learning, machine learning module 310 is continuously trained (or re-trained) as new training data becomes available (e.g., while the model is used to perform inference).

[0081] In some examples, training process 340 may involve centralized training of machine learning module 310 (e.g., based on a centrally stored dataset). In other implementations, decentralized training techniques such as distributed training, federated learning, or the like can be used to train, update, or personalize machine learning module 310.

[0082] Machine learning module 310 described herein can be trained according to one or more of various different training types or techniques. For example, in some examples, machine learning module 310 can be trained by training process 340 using supervised learning, in which machine learning module 310 is trained on a training dataset that includes instances or examples that have labels. The labels can be manually applied by experts, generated through crowd-sourcing, or provided by other techniques (e.g., by physics-based or complex mathematical models). In some examples, if the user has provided consent, the training examples can be provided by the user computing device. In some examples, this process can be referred to as personalizing the model.

[0083] In some examples, machine learning module 310 includes a language model that may be trained (e.g., pre-trained, fine-tuned, etc.) by training process 340. For example, training process 340 may pre-train a language model on a large and diverse corpus of text. As such, in some examples, training data 331 may include a dataset that covers a wide range of topics and domains to ensure machine learning module 310 learns diverse linguistic patterns and contextual relationships. Training process 340 may train a language model to optimize objective function 339. Objective function 339 may be or include a loss function, such as cross-entropy loss, that compares (e.g., determines a difference between) output data generated by the model from training data 331 and labels 337 (e.g., ground-truth labels) associated with training data 331. For example, objective function 339 for a language modelDocket No.: 1333-882WO01 may be to correctly predict the next word in a sequence of words or correctly fill in missing words as much as possible.

[0084] In some examples, training process 340 may use techniques such low-rank adaptation (LoRA) to train or fine-tune language models (LLMs) implemented by machine learning module 310. In general, LoRA may reduce the number of trainable parameters by freezing pre-trained weights of an LLM and injecting small, trainable low-rank matrices that adapt the model for specific tasks. LoRa may be useful when a model needs to be adapted to multiple tasks with limited task-specific data. That is, training process 340 may use LoRA for taskspecific fine-tuning. In some examples, training process 340 may use techniques such as retrieval-augmented generation (RAG), which is a hybrid framework that combines information retrieval with text generation. RAG may be used to fine-tune a generative model implemented by machine learning module 310 by retrieving relevant information from an external database or dataset (e.g., a large and diverse corpus of text) and using that information to generate output that is more accurate and informative. RAG may be useful for generating more factually accurate and contextually relevant summaries and responses to questions.

[0085] In some examples, training process 340 may continuously or periodically train a language model included in machine learning module 310. In some examples, training process 340 may fine-tune a language model by using feedback in the training process. For example, UI component 202 of FIG. 2 may receive a user input via a computing device that selects feedback (e.g., thumbs up, thumbs down, etc.) relating to the generated application functionality and associated GUIs that are presented to the user on the computing device. In some examples, the feedback may indicate whether the generated application functionality and associated GUIs are accurate or inaccurate, correct or incorrect, high quality or low quality, etc. UI module 204 may receive this feedback and may send it to user interface generator module 208. User interface generator module 208 may transmit the feedback to machine learning module 310 (specifically to training process 340), in which training process 340 uses the feedback for training. For example, training process 340 may convert the feedback into labeled data for supervised training. Additionally or alternatively, training process 340 may fine-tune a language model by monitoring the relationship between the performance of the language model and user feedback, and iterate the fine-tuning process as necessary (e.g., to receive more positive user feedback and less negative user feedback). In this way, the techniques of this disclosure may establish a feedback loop that continuously improves the quality of output data 335 (e.g., an instructions file) of a language model.Docket No.: 1333-882WO01

[0086] FIG. 3B is a conceptual diagram illustrating an example trained machine learning module, in accordance with one or more techniques of this disclosure. In some examples, computing device 112 of FIG. 1 may store and implement machine learning module 310 locally (i.e., on-device). Thus, in some examples, machine learning module 310 can be stored at and / or implemented locally by an embedded device or a user computing device such as a mobile device. Output data obtained through local implementation of machine learning module 310 at the embedded device or the user computing device can be used to improve performance of the embedded device or the user computing device (e.g., an application implemented by the embedded device or the user computing device). Machine learning module 310 of FIG. 3B may be trained at a computing system, such as computing system 100 of FIG. 1, and then provided for storage and / or implementation at one or more computing devices, such as computing device 112 of FIG. 1. In some examples, machine learning module 310 executes locally at computing system 100 of FIG. 1. In some examples, computing system 100 may perform machine learning as a service.

[0087] As illustrated in FIG. 3B, in some examples, machine learning module 310 is trained (e.g., via training process 340 of FIG. 3A) to receive input data 333, which may be of one or more types and, in response, provide output data 335, which may be of one or more types. Thus, FIG. 3B illustrates machine learning module 310 performing inference, in which machine learning module 310 may use learned patterns to make predictions or decisions on new data, e.g., input data 333. Machine learning module 310 may include one or more machine-learned models trained by training process 340 of FIG. 3 A.

[0088] Input data 333 may include one or more features that are associated with an instance or an example. In some examples, the one or more features associated with the instance or example can be organized into a feature vector. In some examples, output data 335 can include one or more predictions. Predictions can also be referred to as inferences. Thus, given features associated with a particular instance, machine learning module 310 can output a prediction for such instance based on the features.

[0089] Machine learning module 310 can be or include one or more of various different types of machine-learned models. In particular, in some examples, machine learning module 310 may perform NLP tasks. Machine learning module 310 may summarize, translate, or organize input data 333. Machine learning module 310 may use recurrent neural networks (RNNs) and / or transformer models (self-attention models). Example models may include, but are not limited to, GPT-3, BERT, Gemini (e.g., Gemini Ultra, Gemini Pro, Gemini Flash, Gemini Nano), Android AlCore, and T5. In some examples, machine learning module 310Docket No.: 1333-882WO01 may perform classification, summarization, name generation, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0090] In some examples, machine learning module 310 can perform various types of classification based on input data 333. For example, machine learning module 310 can perform binary classification or multiclass classification. In binary classification, output data 335 can include a classification of input data 333 into one of two different classes. In multiclass classification, output data 335 can include a classification of input data 333 into one (or more) of more than two classes. The classifications can be single label or multi-label. Machine learning module 310 may perform discrete categorical classification in which input data 333 is simply classified into one or more classes or categories.

[0091] In some examples, machine learning module 310 can perform classification in which machine learning module 310 provides, for each of one or more classes, a numerical value descriptive of a degree to which it is believed that input data 333 should be classified into the corresponding class. In some instances, the numerical values provided by machine learning module 310 can be referred to as “confidence scores” that are indicative of a respective confidence associated with classification of the input into the respective class. In some examples, the confidence scores can be compared to one or more thresholds to render a discrete categorical prediction. In some examples, only a certain number of classes (e.g., one) with the relatively largest confidence scores can be selected to render a discrete categorical prediction.

[0092] Machine learning module 310 may output a probabilistic classification. For example, machine learning module 310 may predict, given a sample input, a probability distribution over a set of classes. Thus, rather than outputting only the most likely class to which the sample input should belong, machine learning module 310 can output, for each class, a probability that the sample input belongs to such class. In some examples, the probability distribution over all possible classes can sum to one. In some examples, a Softmax function, or other type of function or layer can be used to squash a set of real values respectively associated with the possible classes to a set of real values in the range (0, 1) that sum to one.

[0093] In some examples, the probabilities provided by the probability distribution can be compared to one or more thresholds to render a discrete categorical prediction. In some examples, only a certain number of classes (e.g., one) with the relatively largest predicted probability can be selected to render a discrete categorical prediction.

[0094] In cases in which machine learning module 310 performs classification, machine learning module 310 may be trained using supervised learning techniques. For example,Docket No.: 1333-882WO01 machine learning module 310 may be trained on a training dataset that includes training examples labeled as belonging (or not belonging) to one or more classes.

[0095] In some examples, machine learning module 310 can perform regression to provide output data in the form of a continuous numeric value. The continuous numeric value can correspond to any number of different metrics or numeric representations, including, for example, currency values, scores, or other numeric representations. As examples, machine learning module 310 can perform linear regression, polynomial regression, or nonlinear regression. As examples, machine learning module 310 can perform simple regression or multiple regression. As described above, in some examples, a Softmax function or other function or layer can be used to squash a set of real values respectively associated with two or more possible classes to a set of real values in the range (0, 1) that sum to one.

[0096] Machine learning module 310 may perform various types of clustering. For example, machine learning module 310 can identify one or more previously-defined clusters to which input data 333 most likely corresponds. Machine learning module 310 may identify one or more clusters within input data 333. That is, in instances in which input data 333 includes multiple objects, documents, or other entities, machine learning module 310 can sort the multiple entities included in input data 333 into a number of clusters. In some examples in which machine learning module 310 performs clustering, machine learning module 310 can be trained using unsupervised learning techniques.

[0097] Machine learning module 310 may perform anomaly detection or outlier detection. For example, machine learning module 310 can identify input data that does not conform to an expected pattern or other characteristic (e.g., as previously observed from previous input data). As examples, the anomaly detection can be used for fraud detection or system failure detection.

[0098] In some examples, machine learning module 310 can provide output data in the form of one or more recommendations. For example, machine learning module 310 can be included in a recommendation system or engine. As an example, given input data that describes previous outcomes for certain entities (e.g., a score, ranking, or rating indicative of an amount of success or enjoyment), machine learning module 310 can output a suggestion or recommendation of one or more additional entities that, based on the previous outcomes, are expected to have a desired outcome (e.g., elicit a score, ranking, or rating indicative of success or enjoyment). As one example, given input data descriptive of a context of a computing device, such as computing device 112 of FIG. 1, a recommendation system canDocket No.: 1333-882WO01 output a suggestion or recommendation of an application that the user might enjoy or wish to download to computing device 112.

[0099] Machine learning module 310 may, in some cases, act as an agent within an environment. For example, machine learning module 310 can be trained using reinforcement learning, which will be discussed in further detail below.

[0100] In some examples, machine learning module 310 can be a parametric model while, in other implementations, machine learning module 310 can be a non-parametric model. In some examples, machine learning module 310 can be a linear model while, in other implementations, machine learning module 310 can be a non-linear model.

[0101] As described above, machine learning module 310 can be or include one or more of various different types of machine-learned models. Examples of such different types of machine-learned models are provided below for illustration. One or more of the example models described below can be used (e.g., combined) to provide output data 335 in response to input data 333. Additional models beyond the example models provided below can be used as well.

[0102] In some examples, machine learning module 310 can be or include one or more classifier models such as, for example, linear classification models; quadratic classification models; etc. Machine learning module 310 may be or include one or more regression models such as, for example, simple linear regression models; multiple linear regression models; logistic regression models; stepwise regression models; multivariate adaptive regression splines; locally estimated scatterplot smoothing models; etc.

[0103] In some examples, machine learning module 310 can be or include one or more decision tree-based models such as, for example, classification and / or regression trees; iterative dichotomiser 3 decision trees; C4.5 decision trees; chi-squared automatic interaction detection decision trees; decision stumps; conditional decision trees; etc.

[0104] Machine learning module 310 may be or include one or more kernel machines. In some examples, machine learning module 310 can be or include one or more support vector machines. Machine learning module 310 may be or include one or more instance-based learning models such as, for example, learning vector quantization models; self- organizing map models; locally weighted learning models; etc. In some examples, machine learning module 310 can be or include one or more nearest neighbor models such as, for example, k- nearest neighbor classifications models; k- nearest neighbors regression models; etc. Machine learning module 310 can be or include one or more Bayesian models such as, for example, naive Bayes models; Gaussian naive Bayes models; multinomial naive BayesDocket No.: 1333-882WO01 models; averaged one-dependence estimators; Bayesian networks; Bayesian belief networks; hidden Markov models; etc.

[0105] In some examples, machine learning module 310 can be or include one or more artificial neural networks (also referred to simply as neural networks). A neural network can include a group of connected nodes, which also can be referred to as neurons or perceptrons. A neural network can be organized into one or more layers. Neural networks that include multiple layers can be referred to as “deep” networks. A deep network can include an input layer, an output layer, and one or more hidden layers positioned between the input layer and the output layer. The nodes of the neural network can be connected or non-fully connected.

[0106] Machine learning module 310 can be or include one or more feed forward neural networks. In feed forward networks, the connections between nodes do not form a cycle. For example, each connection can connect a node from an earlier layer to a node from a later layer.

[0107] In some instances, machine learning module 310 can be or include one or more recurrent neural networks. In some instances, at least some of the nodes of a recurrent neural network can form a cycle. Recurrent neural networks can be especially useful for processing input data that is sequential in nature. In particular, in some instances, a recurrent neural network can pass or retain information from a previous portion of input data 333 sequence to a subsequent portion of input data 333 sequence through the use of recurrent or directed cyclical node connections.

[0108] In some examples, sequential input data can include time-series data (e.g., sensor data versus time or imagery captured at different times). For example, a recurrent neural network can analyze sensor data versus time to detect or predict a swipe direction, to perform handwriting recognition, etc. Sequential input data may include words in a sentence (e.g., for natural language processing, speech detection or processing, etc.); notes in a musical composition; sequential actions taken by a user (e.g., to detect or predict sequential application usage); sequential object states; etc.

[0109] Example recurrent neural networks include long short-term (LSTM) recurrent neural networks; gated recurrent units; bi-direction recurrent neural networks; continuous time recurrent neural networks; neural history compressors; echo state networks; Elman networks; Jordan networks; recursive neural networks; Hopfield networks; fully recurrent networks; sequence-to- sequence configurations; etc.

[0110] In some examples, machine learning module 310 can be or include one or more convolutional neural networks. In some instances, a convolutional neural network can includeDocket No.: 1333-882WO01 one or more convolutional layers that perform convolutions over input data using learned filters.[OHl] Filters can also be referred to as kernels. Convolutional neural networks can be especially useful for vision problems such as when input data 333 includes imagery such as still images or video. However, convolutional neural networks can also be applied for natural language processing.

[0112] In some examples, machine learning module 310 can be or include one or more generative networks such as, for example, generative adversarial networks. Generative networks can be used to generate new data such as new images or other content.

[0113] Machine learning module 310 may be or include an autoencoder. In some instances, the aim of an autoencoder is to learn a representation (e.g., a lower- dimensional encoding) for a set of data, typically for the purpose of dimensionality reduction. For example, in some instances, an autoencoder can seek to encode input data 333 and then provide output data that reconstructs input data 333 from the encoding. Recently, the autoencoder concept has become more widely used for learning generative models of data. In some instances, the autoencoder can include additional losses beyond reconstructing input data 333.

[0114] Machine learning module 310 may be or include one or more other forms of artificial neural networks such as, for example, deep Boltzmann machines; deep belief networks; stacked autoencoders; etc. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

[0115] One or more neural networks can be used to provide an embedding based on input data 333. For example, the embedding can be a representation of knowledge abstracted from input data 333 into one or more learned dimensions. In some instances, embeddings can be a useful source for identifying related entities. In some instances, embeddings can be extracted from the output of the network, while in other instances embeddings can be extracted from any hidden node or layer of the network (e.g., a close to final but not final layer of the network). Embeddings can be useful for performing auto suggest next video, product suggestion, entity or object recognition, etc. In some instances, embeddings can be useful inputs for downstream models. For example, embeddings can be useful to generalize input data (e.g., search queries) for a downstream model or processing system.

[0116] Machine learning module 310 may include one or more clustering models such as, for example, k-means clustering models; k-medians clustering models; expectation maximization models; hierarchical clustering models; etc.Docket No.: 1333-882WO01

[0117] In some examples, machine learning module 310 can perform one or more dimensionality reduction techniques such as, for example, principal component analysis; kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Sammon mapping; multidimensional scaling; projection pursuit; linear discriminant analysis; mixture discriminant analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; autoencoding; etc.

[0118] In some examples, machine learning module 310 can perform or be subjected to one or more reinforcement learning techniques such as Markov decision processes; dynamic programming; Q functions or Q-learning; value function approaches; deep Q-networks; differentiable neural computers; asynchronous advantage actor-critics; deterministic policy gradient; etc.

[0119] In some examples, machine learning module 310 can be an autoregressive model. In some instances, an autoregressive model can specify that output data 335 depends linearly on its own previous values and on a stochastic term. In some instances, an autoregressive model can take the form of a stochastic difference equation. One example autoregressive model is WaveNet, which is a generative model for raw audio.

[0120] In some examples, machine learning module 310 can include or form part of a multiple model ensemble. As one example, bootstrap aggregating can be performed, which can also be referred to as “bagging.” In bootstrap aggregating, a training dataset is split into a number of subsets (e.g., through random sampling with replacement) and a plurality of models are respectively trained on the number of subsets. At inference time, respective outputs of the plurality of models can be combined (e.g., through averaging, voting, or other techniques) and used as the output of the ensemble.

[0121] One example ensemble is a random forest, which can also be referred to as a random decision forest. Random forests are an ensemble learning method for classification, regression, and other tasks. Random forests are generated by producing a plurality of decision trees at training time. In some instances, at inference time, the class that is the mode of the classes (classification) or the mean prediction (regression) of the individual trees can be used as the output of the forest. Random decision forests can correct for decision trees' tendency to overfit their training set.

[0122] Another example ensemble technique is stacking, which can, in some instances, be referred to as stacked generalization. Stacking includes training a combiner model to blend or otherwise combine the predictions of several other machine-learned models. Thus, a pluralityDocket No.: 1333-882WO01 of machine-learned models (e.g., of same or different type) can be trained based on training data. In addition, a combiner model can be trained to take the predictions from the other machine-learned models as inputs and, in response, produce a final inference or prediction. In some instances, a single-layer logistic regression model can be used as the combiner model.

[0123] Another example of an ensemble technique is boosting. Boosting can include incrementally building an ensemble by iteratively training weak models and then adding to a final strong model. For example, in some instances, each new model can be trained to emphasize the training examples that previous models misinterpreted (e.g., misclassified). For example, a weight associated with each of such misinterpreted examples can be increased. One common implementation of boosting is AdaBoost, which can also be referred to as Adaptive Boosting. Other example boosting techniques include LPBoost; TotalBoost; BrownBoost; xgboost; MadaBoost, LogitBoost, gradient boosting; etc. Furthermore, any of the models described above (e.g., regression models and artificial neural networks) can be combined to form an ensemble. As an example, an ensemble can include a top level machine- learned model or a heuristic function to combine and / or weight the outputs of the models that form the ensemble.

[0124] In some examples, multiple machine-learned models (e.g., that form an ensemble can be linked and trained jointly (e.g., through backpropagation of errors sequentially through the model ensemble). However, in some examples, only a subset (e.g., one) of the jointly trained models is used for inference.

[0125] In some examples, machine learning module 310 can be used to preprocess input data 333 for subsequent input into another model. For example, machine learning module 310 can perform dimensionality reduction techniques and embeddings (e.g., matrix factorization, principal components analysis, singular value decomposition, word2vec / GLOVE, and / or related approaches); clustering; and even classification and regression for downstream consumption.

[0126] As discussed above, machine learning module 310 can be trained or otherwise configured to receive input data 333 and, in response, provide output data 335. Input data 333 can include different types, forms, or variations of input data. As examples, in various implementations, input data 333 can include features that describe the content (or portion of content) initially selected by the user, e.g., content of user-selected document or image, links pointing to the user selection, links within the user selection relating to other files available on device or cloud, metadata of user selection, etc. Additionally, with user permission, input data 333 includes the context of user usage, either obtained from the app itself or from otherDocket No.: 1333-882WO01 sources. Examples of usage context include breadth of share (sharing publicly, or with a large group, or privately, or a specific person), context of share, etc. When permitted by the user, additional input data can include the state of the device, e.g., the location of the device, the apps running on the device, etc.

[0127] In some examples, machine learning module 310 can receive and use input data 333 in its raw form. In some examples, the raw input data can be preprocessed. Thus, in addition or alternatively to the raw input data, machine learning module 310 can receive and use the preprocessed input data.

[0128] In some examples, preprocessing input data 333 can include extracting one or more additional features from the raw input data. For example, feature extraction techniques can be applied to input data 333 to generate one or more new, additional features. Example feature extraction techniques include edge detection; corner detection; blob detection; ridge detection; scale-invariant feature transform; motion detection; optical flow; Hough transform; etc.

[0129] In some examples, the extracted features can include or be derived from transformations of input data 333 into other domains and / or dimensions. As an example, the extracted features can include or be derived from transformations of input data 333 into the frequency domain. For example, wavelet transformations and / or fast Fourier transforms can be performed on input data 333 to generate additional features.

[0130] In some examples, the extracted features can include statistics calculated from input data 333 or certain portions or dimensions of input data 333. Example statistics include the mode, mean, maximum, minimum, or other metrics of input data 333 or portions thereof.

[0131] In some examples, as described above, input data 333 can be sequential in nature. In some instances, the sequential input data can be generated by sampling or otherwise segmenting a stream of input data. As one example, frames can be extracted from a video. In some examples, sequential data can be made non-sequential through summarization.

[0132] As another example preprocessing technique, portions of input data 333 can be imputed. For example, additional synthetic input data can be generated through interpolation and / or extrapolation.

[0133] As another example preprocessing technique, some or all of input data 333 can be scaled, standardized, normalized, generalized, and / or regularized. Example regularization techniques include ridge regression; least absolute shrinkage and selection operator (LASSO); elastic net; least-angle regression; cross-validation; LI regularization; L2 regularization; etc. As one example, some or all of input data 333 can be normalized byDocket No.: 1333-882WO01 subtracting the mean across a given dimension’s feature values from each individual feature value and then dividing by the standard deviation or other metric.

[0134] As another example preprocessing technique, some or all or input data 333 can be quantized or discretized. In some cases, qualitative features or variables included in input data 333 can be converted to quantitative features or variables. For example, one hot encoding can be performed.

[0135] In some examples, dimensionality reduction techniques can be applied to input data 333 prior to input into machine learning module 310. Several examples of dimensionality reduction techniques are provided above, including, for example, principal component analysis; kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Sammon mapping; multidimensional scaling; projection pursuit; linear discriminant analysis; mixture discriminant analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; autoencoding; etc.

[0136] In some examples, during training, input data 333 can be intentionally deformed in any number of ways to increase model robustness, generalization, or other qualities. Example techniques to deform input data 333 include adding noise; changing color, shade, or hue; magnification; segmentation; amplification; etc.

[0137] In response to receipt of input data 333, machine learning module 310 can provide output data 335. Output data 335 can include different types, forms, or variations of output data. As examples, in various implementations, output data 335 can include content, either stored locally on the user device or in the cloud, that is relevantly shareable along with the initial content selection.

[0138] As discussed above, in some examples, output data 335 can include various types of classification data (e.g., binary classification, multiclass classification, single label, multilabel, discrete classification, regressive classification, probabilistic classification, etc.) or can include various types of regressive data (e.g., linear regression, polynomial regression, nonlinear regression, simple regression, multiple regression, etc.). In other instances, output data 335 can include clustering data, anomaly detection data, recommendation data, or any of the other forms of output data discussed above.

[0139] In some examples, output data 335 can influence downstream processes or decision making. As one example, in some examples, output data 335 can be interpreted and / or acted upon by a rules-based regulator.Docket No.: 1333-882WO01

[0140] Any of the different types or forms of input data described herein can be combined with any of the different types or forms of machine-learned models described herein to provide any of the different types or forms of output data described herein.

[0141] The systems and methods of the present disclosure can be implemented by or otherwise executed on one or more computing devices. Example computing devices include user computing devices (e.g., laptops, desktops, and mobile computing devices such as tablets, smartphones, wearable computing devices, etc.); embedded computing devices (e.g., devices embedded within a vehicle, camera, image sensor, industrial machine, satellite, gaming console or controller, or home appliance such as a refrigerator, thermostat, energy meter, home energy manager, smart home assistant, etc.); server computing devices (e.g., database servers, parameter servers, file servers, mail servers, print servers, web servers, game servers, application servers, etc.); dedicated, specialized model processing or training devices; virtual computing devices; other computing devices or computing infrastructure; or combinations thereof. A computing system that implements machine learning module 310 or other aspects of the present disclosure may include a number of hardware components that enable the performance of the techniques described herein.

[0142] In some instances, output data 335 obtained through machine learning module 310 at a computing system or device can be used to improve other device tasks or can be used by other non-user devices to improve services performed by or for such other non-user devices. For example, output data 335 can improve other downstream processes performed by a server device for a computing device of a user or embedded computing device. In other instances, output data 335 obtained through implementation of machine learning module 310 at a computing system or device can be sent to and used by a user computing device, an embedded computing device, or some other client device. In some examples, computing system 200 of FIG. 2 may perform machine learning as a service.

[0143] In yet other implementations, different respective portions of machine learning module 310 can be stored at and / or implemented by some combination of a user computing device; an embedded computing device; a server computing device; etc. In other words, portions of machine learning module 310 may be distributed in whole or in part amongst a client device (e.g., computing device 112 of FIG. 1) and a computing system (e.g., computing system 100 of FIG. 1).

[0144] A computing device such as computing device 112 of FIG. 1 may perform graph processing techniques or other machine learning techniques using one or more machineDocket No.: 1333-882WO01 learning platforms, frameworks, and / or libraries, such as, for example, TensorFlow, Caffe / Caffe2, Theano, Torch / Py Torch, MXnet, CNTK, etc.

[0145] In some examples, multiple instances of machine learning module 310 can be parallelized to provide increased processing throughput. For example, the multiple instances of machine learning module 310 can be parallelized on a single processing device or computing device or parallelized across multiple processing devices or computing devices.

[0146] A computing device that implements machine learning module 310 or other aspects of the present disclosure can include a number of hardware components that enable performance of the techniques described herein. For example, a computing device can include one or more memory devices that store some or all of machine learning module 310. For example, machine learning module 310 can be a structured numerical representation that is stored in memory. The one or more memory devices can also include instructions for implementing machine learning module 310 or performing other operations. Example memory devices include RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof.

[0147] A computing device can also include one or more processing devices that implement some or all of machine learning module 310 and / or perform other related operations. Example processing devices include one or more of: a central processing unit (CPU); a visual processing unit (VPU); a graphics processing unit (GPU); a tensor processing unit (TPU); a neural processing unit (NPU); a neural processing engine; a core of a CPU, VPU, GPU, TPU, NPU or other processing device; an application specific integrated circuit (ASIC); a field programmable gate array (FPGA); a co-processor; a controller; or combinations of the processing devices described above. Processing devices can be embedded within other hardware components such as, for example, an image sensor, accelerometer, etc.

[0148] Hardware components (e.g., memory devices and / or processing devices) can be spread across multiple physically distributed computing devices and / or virtually distributed computing systems.

[0149] In some examples, machine learning module 310 described herein can be included in different portions of computer-readable code on a computing device. In one example, machine learning module 310 can be included in a particular application or program and used (e.g., exclusively) by such a particular application or program. Thus, in one example, a computing device can include a number of applications and one or more of such applications can contain its own respective machine learning library and machine-learned model(s).Docket No.: 1333-882WO01

[0150] In another example, machine learning module 310 described herein can be included in an operating system of a computing device (e.g., in a central intelligence layer of an operating system) and can be called or otherwise used by one or more applications that interact with the operating system. In some examples, each application can communicate with the central intelligence layer (and model(s) stored therein) using an application programming interface (API) (e.g., a common, public API across all applications).

[0151] In some examples, the central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device. The central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some examples, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0152] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination.

[0153] Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0154] In addition, the machine learning techniques described herein are readily interchangeable and combinable. Although certain example techniques have been described, many others exist and can be used in conjunction with aspects of the present disclosure.

[0155] Further to the descriptions above, a user may be provided with controls that enable the user to make an election as to both if and when systems, programs or features described herein may enable collection of user information (e.g., information about a user’s social network, social actions or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIPDocket No.: 1333-882WO01 code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0156] FIG. 3C is a conceptual diagram illustrating a machine learning module configured to parse natural language input to identify tasks and determine associated applications for completing tasks, in accordance with one or more techniques of this disclosure. Machine learning module 310 of FIG. 3C may be an example of machine learning module 310 of FIGS. 3 A and 3B. In general, ML module 310 can be or include one or more transformerbased neural networks, such as a large language model module 342. In general, language model module 342 may apply an LLM to retrieved context information to identify one or more tasks. In some examples, language model module 342 may apply an LLM to the retrieved context information to determine, for each of the one or more tasks, one or more associated applications, in which each of the one or more associated applications includes one or more functions for performing a respective task.

[0157] Language model module 342 may implement, for example, the Pathways Language Model developed by Google. Transformer-based neural networks may refer to a type of deep learning architecture specifically designed for handling sequential data, such as text or time series. In other words, transformer-based neural networks like LLMs may be configured to perform natural language processing (NLP) tasks, such as question-answering, machine translation, text summarization, and sentiment analysis. Language model module 342 may be configured to perform tasks such as classification, sentiment analysis, entity extraction, extractive question answering, summarization, re-writing text in a different style, ad copy generation, and concept ideation.

[0158] Transformer-based neural networks may utilize a self-attention mechanism, which allows the model to weigh the importance of different elements in a given input sequence relative to each other. The self-attention mechanism may help language model module 342 effectively capture long-range dependencies and complex relationships between elements, such as words in a sentence.

[0159] Language model module 342 may include an encoder and a decoder that operate to process and generate sequential data, such as structured text. Both the encoder and decoder may include one or more of self-attention mechanisms, position-wise feedforward networks, layer normalization, or residual connections. In some examples, the encoder may process an input sequence and create a representation that captures the relationships and context among the elements in the sequence. The decoder may then obtain the representation generated byDocket No.: 1333-882WO01 the encoder and produce an output sequence. In some examples, the decoder may generate the output one element at a time (e.g., one word at a time), using a process called autoregressive decoding, where the previously generated elements are used as input to predict the next element in the sequence.

[0160] In some examples, language model module 342 may determine a set of information types included in the input. An information type may be or otherwise include a topic, theme, point, subject, purpose, intent, keyword, etc. In some examples, language model module 342 may determine the information type by leveraging a self-attention mechanism to capture the relationships and dependencies between words in the input sequence. For example, language model module 342 may tokenize (e.g., split) a sequence of words or subwords, which language model module 342 may convert into vectors (e.g., numerical representations) that language model module 342 can process. Language model module 342 may use the selfattention mechanism to weigh the importance of each token in relation to the others. In this way, language model module 342 may identify patterns and relationships between the tokens, and in turn the words corresponding to the tokens, that indicate one or more information types.

[0161] In general, language model module 342 may excel at performing NLP tasks, such as generating text and other content (e.g., new code that generates GUIs, graphical components, and / or functionality for performing one or more tasks). However, with respect to specific types of content (e.g., specific information types), language model module 342 may have an increased likelihood of generating false, inaccurate, or bad quality information. To address this issue, language model module 342 may be configured to exclude the generation of content or code relating to a set of excluded information types. For example, the set of excluded information types may include one or more of phone numbers, addresses, web addresses, functionality prohibited by an application, sensitive data (e.g., full bank account information), etc. Thus, input information may be passed in language model module 342 with certain prerequisites, prompts, or “rules” that can be stored in rules storage 344. Machine learning module 310 may apply these prerequisites, prompts, or rules when generating the set of instructions for generating the GUIs and graphical components associated with the functionality for performing the identified tasks.

[0162] For example, machine learning module 310 may implement a rule such as, “Do not include user’s sensitive information” when generating instructions for generating an associated banking application overlay GUI that includes pre-populated text entry (e.g., instead of including a user’s full bank account number, the pre-populated text entry mayDocket No.: 1333-882WO01 include a string such as, “Bank Account ending in 1234”). In some examples, machine learning module 310 may use accessibility information when generating new code for GUIs and graphical components, such that the user can easily interact with the GUIs and graphical components. In some examples, the rules may be text inputs such as, for example, “Do not display more than 25 characters of a message within a task widget.” As such, rules storage 354 may store a plurality of text inputs and / or other data that further specify how instructions file 346 should be generated by machine learning module 310. For example, language model module 342 may be applied to the context information in accordance with the one or more predefined rules stored in rules storage 344, which may include, for example, unauthorized terms, unauthorized class names, unauthorized dimensions of the graphical user interface, unauthorized application functionality, etc. Because language model module 342 can interpret the rules along with the input, the computing system may provide more accurate instructions for generating GUIs, graphical components, associated application overlay GUIs, and / or suggested data for performing identified tasks. In this way, the computing system may be able to interpret context information to identify a user’s tasks, and then write or generate, at machine speed, new, robust, working code that can render new graphical user interfaces and / or components performing the identified tasks.

[0163] While language model module 342 may be a transformer-based neural network in some examples, in some other examples, language model module 342 may be or otherwise include one or more other types of neural networks. For example, language model module 342 may be or include an autoencoder. In some examples, the aim of an autoencoder is to learn a representation (e.g., a lower- dimensional encoding) for a set of data, typically for the purpose of dimensionality reduction. For example, in some examples, an autoencoder can seek to encode the input data and then provide output data that reconstructs the input data from the encoding. In some examples, the autoencoder can include additional losses beyond reconstructing the input data. Language model module 342 may be or include one or more other forms of artificial neural networks such as, for example, deep Boltzmann machines, deep belief networks, stacked autoencoders, etc. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

[0164] Generally, large language models can be slow and expensive in terms of carbon, energy usage, and financial cost. Thus, in some examples, machine learning module 310 may minimize how often language model module 342 is invoked by caching generated instructions, or new code, in instructions cache 348. For example, in some examples, language model module 342 may use a prompt including the context information retrieved byDocket No.: 1333-882WO01 the computing system. At runtime, more specific details may be gathered (e.g., via the API), such that the generated instructions or code may be reused. Specifically, machine learning module 310 may be configured to perform instruction embedding in which a representation (i.e., embedding) of frequently used or critical instructions are stored in instructions cache 348.

[0165] In various examples, instructions file 346 may be generated based on the instructions stored in instructions cache 348 and any additional instructions, information, or updates retrieved by the API that are not present in instructions cache 348. For example, instructions storage 222 of FIG. 2 or any other local memory may store these additional instructions, information, or updates retrieved by API module 206. Machine learning module 310 may query instructions storage 222 or other local memory to gather these additional instructions, information, or updates and use them with the cached instructions at runtime to generate instructions file 346. As an example, the instructions for generating the main task list GUI may be stored in instructions cache 348, and may be updated when the computing system determines, based on one or more of a received indication of a gesture and additional context information, that a task was performed (in which the instructions for generating the main task list GUI may only be updated to indicate the respective task was performed, e.g., by removing the task widget for the completed task). As another example, if the computing system determines a task is reoccurring, such as a task for scheduling a bi-weekly meeting with Jane, language model module 342 may generate instructions for generating a “schedule meeting with Jane” task widget, and store the instructions in instructions cache 348. In this example, the computing system may update the main task list GUI instructions stored in instructions cache 348 to include the “schedule meeting with Jane” task widget instructions, e.g., on a time basis, such as every other week. However, if the computing system retrieves context information pertaining to the task of scheduling the meeting with Jane, such as a text message from Jane stating, “I need to cancel next week’s meeting,” or “Can we have our meetings every other Tuesday instead?”, machine learning module 310 may use this additional context information with the cached instructions at runtime to generate instructions file 346, in which instructions file 346 may include updated instructions for generating the “schedule meeting with Jane” task widget. That is, based on the additional context information, the “schedule meeting with Jane” task widget may not be included in instructions file 346 this week (as the user may not be tasked with scheduling the meeting this week), or the overlay GUI of the calendar application associated with the “schedule meeting with Jane” task widget may be updated to display the schedule for every other Tuesday. AsDocket No.: 1333-882WO01 such, in general, instructions file 346 may include cached instructions and / or additional instructions, information, or updates for generating GUIs, graphical components, associated application overlay GUIs, and suggested data. Furthermore, in some examples, instructions file 346 may be updated and / or sent to a computing device on a time basis, e.g., periodically, based on when one or more tasks recur, based on when additional context information is retrieved, etc.

[0166] By storing frequently used or critical instructions in instructions cache 348, machine learning module 310 may reuse the frequently used or critical instructions without having to invoke language model module 342 on data other than what is included in new context information or input (e.g., language model module 342 may not have to re-apply the large language model to all stored context information). In some examples, machine learning module 310 may apply code caching to both compiled and interpreted languages. Machine learning module 310 may implement various types of caching, such as, for example, Just-In- Time (JIT) compilation, Ahead-Of-Time (AOT) compilation, and bytecode caching.

[0167] In some examples, instructions file 346 may include all data collected or used by the computing system to generate instructions file 346. For example, instructions file 346 may include details for how the user's natural language was resolved into working code. In some examples, users may be able to view or “inspect” instructions file 346. In other words, a user may be provided various controls to clarify, inspect, or stop a task to ensure that the computing system is following the user’s intent. Thus, the generated user interfaces and / or graphical components may be inspectable, in which users can, for example, interact with widgets to see the associated data, code or instructions (e.g., instructions file 346), or pinch to expand widgets to reveal more controls. Furthermore, a user may be able to edit instructions file 346. For example, a user may edit the parameters used by machine learning module 310, and the code included in instructions file 346 may update to reflect the edits. Furthermore, in some examples, users may interact with the GUIs and / or graphical components to add or delete GUIs and / or graphical components, directly edit parameters, edit the order of the GUIs, the arrangement of the graphical components, change, add, or delete visual effects, etc. As such, any predetermined or suggested data determined by machine learning module 310, the instructions for generating the GUIs, graphical components, associated application overlay GUIs, and any other data included in instructions file 346 may be customizable or user configurable. However, it should be noted that in some examples, certain instructions may not be inspectable and / or editable by users, such as those pertaining to certain graphical elements included in associated application overlay GUIs (e.g., trademarked symbols), andDocket No.: 1333-882WO01 one or more functions included in the associated applications (e.g., a user may not edit a banking application’s functionality for transferring funds).

[0168] By leveraging one or more of the machine learning techniques described herein, and by leveraging code caching, the user interface generation provided by the computing system may require less time and / or computational resources to create new GUIs and graphical components for performing a user’s identified tasks. Furthermore, instead of users having to remember multiple tasks and navigate through multiple applications and user interfaces to access relevant information and functionality for performing their multiple tasks, the techniques of this disclosure may provide users the ability to quickly have their tasks organized in a manner similar to a “to-do list,” in which users may quickly perform their tasks by providing a few simple user interface interactions.

[0169] FIGS. 4A-4I are conceptual diagrams illustrating examples of graphical user interfaces and graphical components for performing tasks, in accordance with one or more techniques of this disclosure. FIGS. 4A-4I may be described with respect to computing system 100 and computing device 112 of FIG.1 Some or all of the components and / or functionality attributed to computing system 100 may be implemented or performed by computing device 112. That is, in some examples, the techniques described herein may be implemented or performed locally, e.g., “on-device.”

[0170] In the example of FIG. 4 A, with explicit consent from user 120 operating computing device 112, user interface generator module 108 may retrieve, using API module 106, context information from one or more applications (e.g., the applications associated with widgets 118A-1181 on GUI 116 of FIG. 1). For example, the context information may include data from a messaging application, e.g., a text message received from John Doe such as, “Hi, can you send me $20?” User interface generator module 108 may apply machine learning module 110 to the retrieved context information to identify one or more tasks. Continuing the example, machine learning module 110 may identify, based on at least a portion of the context information, a task of sending $20 to John Doe. In some examples, machine learning module 110 may further determine, e.g., based on the context information, and for each of the one or more tasks, one or more associated applications, in which each of the one or more associated applications includes one or more functions for performing a respective task. For example, machine learning module 110 may determine, based on the retrieved context information, that computing device 112 includes a banking application that provides functions associated with the task of sending money. As such, machine learning module 110 may further contextualize the text message from John Doe and / or the identified task ofDocket No.: 1333-882WO01 sending $20 to John Doe, as machine learning module 110 may determine that performing the identified task involves actions such as transferring funds from user 120’ s bank account to John Doe’s bank account via the banking application. As such, machine learning module 110 may determine the at least one associated application to be the banking application, in which the banking application includes functionality for performing the task of sending $20 to John Doe.

[0171] In some examples, machine learning module 110 may assign a priority score to each of the one or more tasks. For example, a respective priority score for a task involving a particular contact of user 120 may be based on historical user data that indicates that user 120 messages one or more contacts with a frequency greater than a threshold, that user 120 frequently responds to family member contacts within shorter time frames, that user 120 frequently performs actions within applications that involve certain contacts, etc. User interface generator module 108 may generate instructions for generating GUI 419A, which may be considered a main GUI for organizing a user’s identified tasks, or a “task list GUI.” As shown in the example of FIG. 4 A, GUI 419A may include a first plurality of graphical components, such as task widgets 450A-450F, in which each graphical component from the first plurality of graphical components is associated with a respective task. Furthermore, an arrangement of each graphical component from the first plurality of graphical components within GUI 419A is based on the respective priority score assigned to the respective task. As shown in the example of FIG. 4A, task widget 450D may be associated with the task of sending $20 to John. In this example, machine learning module 110 may assign a respective priority score to the task of sending $20 to John that is less than the priority scores assigned to the tasks associated with task widgets 450A, 450B, and 450C. However, the respective priority score assigned to the task of sending $20 to John may be greater than the tasks associated with task widgets 450E and 450F; as such, task widget 450D may be arranged below task widgets 450A, 450B, and 450C, but above task widgets 450E and 450F. In some examples, GUI 419A may be scrollable, and may include any number of task widgets associated with any number of tasks.

[0172] Task widget 450D, as shown, may include text such as “John D.” to indicate that John D. is the contact associated with the task, “1 day ago,” to indicate when the message from John D. was received, at least a portion of the text message received from John D. (e.g., “Hi, can you send me $20?”), “Send money” button 449, and associated application widgets 418A and 418B, which may be similar to application widgets 118A and 118B of FIG. 1, respectively. That is, in this example, associated application widget 418A may be theDocket No.: 1333-882WO01 application widget for a first banking application installed at computing device 112, and associated application widget 418B may be the application widget for a second banking application installed at computing device 112, as machine learning module 110 determined the two banking applications to include functionality associated with the task of sending $20 to John Doe.

[0173] In some examples, computing system 100 may receive an indication of a gesture detected at a location of a UI component 102 that corresponds to a task widget from the plurality of task widgets included in GUI 419A. For example, user 120 may interact with “Send money” button 449 of task widget 450D, and computing system 100 may receive an indication of the detected interaction. Based on the indication of the gesture, computing system 100 may generate instructions for generating one or more additional graphical user interfaces, such as overlay GUI 451 A, in which each of the one or more additional graphical user interfaces is associated with one of the one or more associated applications determined for a respective task, and each of the one or more additional graphical user interfaces includes a subset of graphical components from a second plurality of graphical components associated with the one or more functions for performing the respective task. Furthermore, in some examples, machine learning module 110 may determine, based on the context information, suggested data, in which the second plurality of graphical components associated with the one or more functions for performing the respective task includes the suggested data. As shown in the example of FIG. 4A, overlay GUI 451 A includes text entry field 452A (prepopulated with suggested text “$20”), text entry field 453 A (prepopulated with suggested text “JD” to represent John D. as the recipient), and “Send” button 454A, with which user 120 may interact to perform the task of sending $20 to John D. In some examples, overlay GUI 451 A may be considered a smaller, more concise overlay GUI or widget for the first banking application corresponding to associated application widget 418A.

[0174] In examples in which machine learning module 110 determines two or more associated applications for a single task, such as in the example of FIG. 4 A, machine learning module 110 may further rank each associated application, e.g., based on historical user data indicating which associated application user 120 is most likely to interact with to complete a specific task. In this example, machine learning module 110 may determine that user 120 is more likely to send money through the first banking application corresponding to associated application widget 418A than the second banking application corresponding to associated application widget 418B; as such, when user 120 interacts with “Send money” button 449, overlay GUI 451 A associated with the first banking application may be displayed first. InDocket No.: 1333-882WO01 some examples, however, user 120 may be provided the option to toggle to an overlay GUI for the second banking application, so as to complete the task using the second banking application.

[0175] Overlay GUI 45 IB of FIG. 4B may be another view of overlay GUI 451 A of FIG. 4A. That is, in some examples, user 120 may interact with task widget 450D, such as with associated application widget 418A or another graphical component included in the task widget, and based on the indication of the interaction, computing system 100 may generate instructions for generating overlay GUI 45 IB, which may be an expanded version of overlay GUI 451 A. Thus, overlay GUI 45 IB may be considered a larger overlay GUI or widget for the first banking application corresponding to associated application widget 418A, and may include more graphical components than overlay GUI 451 A, such as header 460 (which may include text pertaining to the task, such as “John Doe - $20.00) text entry field 455 (which may be a search bar that enables a user to search for other users of the first banking application), text entry field 456 (prepopulated with suggested text “Bank Account ending in 1234”), and “Request” button 462. As shown, overlay GUI 45 IB may also include expanded versions of the graphical components of overlay GUI 451 A, such as text entry field 452B (prepopulated with suggested text “$20.00”), text entry field 453B (prepopulated with suggested text “John Doe,” which may be John D.’s username determined based on a list of user 120’s trusted contacts stored within the first banking application), and “Send” button 454B, with which user 120 may interact to perform the task of sending $20.00 to John Doe through the first banking application. In some examples, the one or more additional user interfaces may be displayed along with the one or more associated application widgets. As shown in the example of FIG. 4B, associated application widgets 418A, 418B, and 418D may be presented such that user 120 may interact with each to toggle between the different associated applications. For example, while overlay GUI 45 IB may be an overlay GUI for the first banking application corresponding to associated application widget 418A, user 120 may interact with associated application widget 418B to switch to an overlay GUI for the second banking application, or interact with associated application widget 418D (which may be similar to application widget 118D of FIG. 1) to switch to an overlay GUI for a third banking application.

[0176] Overlay GUI 451C of FIG. 4C may be another view of overlay GUI 45 IB of FIG. 4B, in which user 120 has performed the task of sending $20 to John D. Specifically, in some examples, computing system 100 may receive an indication of a gesture detected at a location of a UI component 102 that corresponds to a graphical component from the second pluralityDocket No.: 1333-882WO01 of graphical components associated with the one or more functions for performing the respective task. For example, a user may interact with “Send” button 454B of FIG. 4B that is associated with the functionality included in the first banking application that performs the task of sending $20 to John D. Computing system 100 may determine, based on one or more of the indication of the gesture and additional context information, whether the respective task was performed. For example, after sending $20 to John D. via overlay GUI 45 IB, overlay GUI 45 IB may transition to overlay GUI 451C, which may include text output 464 “$20.00 sent to John Doe.” Furthermore, user 120 may, for example, send a text message to John D. such as, “Just sent.” In this example, based on the indication of the interaction with “Send” button 454B, and / or additional retrieved context information including text output 464 and content of the subsequent text message sent to John D. from user 120, machine learning module 110 may determine that the task of sending $20 to John D. was performed. Responsive to determining that the respective task was performed, user interface generator module 108 may update the instructions for generating GUI 419A to indicate the respective task was performed. For example, user interface generator module 108 may update the instructions to remove task widget 450D from GUI 419A, updating task widget 450D to include, for example, a check mark icon, etc.

[0177] In this way, rather than having to navigate through multiple applications to perform a single task, a user may perform the task in a quick and organized manner, e.g., by simply interacting with associated application overlay GUIs 451 A or 45 IB, which may even be prepopulated with suggested data needed to complete the task. Therefore, the graphical components, e.g., task widgets and associated application overlay GUIs, may provide users a “shortcut” for completing tasks. For example, instead of requiring a user to remember the task of sending money to John D., task widget 450D may serve as an automatic notification or reminder for the task of sending money to John D. Furthermore, instead of requiring a user to navigate through a messaging application to find out how much money to send to John D., and navigate through a banking application to search for John D.’s banking account username, initiate a new transfer, and manually enter in all relevant information, associated application overlay GUIs 451 A or 45 IB may provide the functionality for performing all of the aforementioned actions through a few simple user interface interactions, e.g., a few button clicks. As such, the user may find it easier to complete tasks, and may enjoy an overall improved user experience.

[0178] Responsive to determining that a respective task was performed, such as the task associated with task widget 450D of FIG. 4A, computing system 100 may update theDocket No.: 1333-882WO01 instructions for generating GUI 419A to indicate the respective task was performed. As shown in the example of FIG. 4D, GUI 419B may be an updated version of GUI 419A, in that GUI 419B no longer includes task widget 450D as the task associated with task widget 450D was performed. In the example of FIG. 4D, user 120 interacts with task widget 450B, which may be a task identified based on the content of a text message received 14 minutes ago from Ryan K., a portion of which reads, “Game night at mine Fri, want to. ..” For example, the task associated with task widget 450B may be a task of replying to Ryan K.’s message based on user 120’s schedule. As such, in some examples, a respective task may include a plurality of actions. To determine the one or more associated applications for the respective task, machine learning module 110 may identify, based on a first subset of actions from the plurality of actions, a first associated application that includes one or more functions for performing the first subset of actions. For example, in the example of FIG. 4D, the full text message received from Ryan K. may be “Game night at mine Fri, want to come around 7?” Machine learning module 110 may analyze the content of the message and determine a first subset of actions to include scheduling. Furthermore, machine learning module 110 may identify a calendar application installed on computing device 112 that includes one or more functions for performing the scheduling actions. Based on a second subset of actions from the plurality of actions, machine learning module 110 may further identify a second associated application that includes one or more functions for performing the second subset of actions. Continuing the example, machine learning module 110 may determine a second subset of actions to include replying to Ryan K.’s message. As such, machine learning module 110 may identify the messaging application installed on computing device 112 that includes one or more functions for performing the messaging actions. As shown in the example of FIG. 4D, task widget 450B may include associated application widget 418C (which may be similar to application widget 118C of FIG. 1) corresponding to the associated calendar application, and associated application widget 418E (which may be similar to application widget 118E of FIG. 1) corresponding to the associated messaging application.

[0179] In some examples, a first additional GUI from the one or more additional GUIs is associated with a first associated application, and a second additional GUI from the one or more additional GUIs is associated with a second associated application. In some examples, any number of additional GUIs may be associated with any number of associated applications. In some examples, the first additional GUI includes a first subset of graphical components from the second plurality of graphical components, in which the first subset of graphical components is associated with the one or more functions for performing the firstDocket No.: 1333-882WO01 subset of actions. In the example of FIG. 4E, user 120 may interact with task widget 450B to have overlay GUI 467 displayed. In this example, overlay GUI 467 may be an overlay GUI for the associated calendar application identified by machine learning module 110, and may include “Game Night” header 466 and add calendar event button 468. Furthermore, as shown in the example of FIG. 4E, overlay GUI 467 may display user 120’s schedule for Friday, October 14th, which may be based on the content of Ryan K.’s text message. That is, machine learning module 110 may analyze the retrieved context information and use at least a portion of the context information to generate GUIs, graphical components, suggestions, etc. Therefore, the content of overlay GUI 467 may be based on the content of Ryan K.’s text message. Overlay GUI 467, as shown, indicates that user 120 has a scheduled “Lunch at Restaurant A" at 12 PM, and a scheduled “Meeting with Jane” between 3 and 4 PM. The context information retrieved from the associated calendar application, such as the data pertaining to user 120’ s schedule on Friday, October 14th, may be used by machine learning module 110 to generate suggested data, such as suggested summary 469, which states, “You are available from 4 PM this Friday.” The instructions for generating GUI 419B and text summary 469 may also include instructions for generating “Schedule” button 470, in which “Schedule” button 470 may correspond to a suggested action of scheduling the 7 PM game night event referred to in Ryan K.’s text message. Thus, the graphical components included in overlay GUI 467 are associated with functionality for performing the first subset of actions, e.g., the scheduling actions. User 120 may interact with associated application widgets 418C and 418E to toggle or switch between overlay GUIs for the associated calendar application and the associated messaging application.

[0180] In the example of FIG. 4F, user 120 has interacted with associated application widgets 418C and 418E to switch to GUI 472 A, which is an overlay GUI for the associated messaging application. In some examples, the second additional GUI includes a second subset of graphical components from the second plurality of graphical components, in which the second subset of graphical components is associated with the one or more functions for performing the second subset of actions. As shown, GUI 472A includes “Game Night” header 466, text message 474 from Ryan K. that states, “Game night at mine Fri. Want to come around 7?”, and text message entry field 476. Furthermore, the instructions for generating GUI 472A include instructions for generating “Going” button 477 and “Not Going” button 478, which may each be associated with a suggested reply to text message 474. In the example of FIG. 4F, user 120 may interact with “Going” button 477 to select the suggested data. In some examples, the selected suggestion may be automatically sent as aDocket No.: 1333-882WO01 reply. For example, in another example, user 120 may select a button that reads, “I’ll be there!”, in which a text message containing the message “I’ll be there!” may be automatically sent as a reply to text message 474 from Ryan K. As such, user 120 may not be required to manually enter text into text entry field 476 to reply to text message 474.

[0181] In some examples, the suggested data may correspond to one or more application capabilities, and / or actions that a user may perform within the application and / or beyond the application. That is, an overlay GUI may present graphical elements such as buttons that a user may select to provide input and / or access application capabilities.

[0182] As shown in the example of FIG. 4G, in some examples, responsive to user 120 selecting a suggestion, one or more additional suggestions may be generated by machine learning module 110 based on the selected suggestion. For instance, GUI 472B, also associated with the messaging application, may be a version of GUI 472A that has been updated based on user 120 interacting with “Going” button 477 of FIG. 4H to select the suggested data. In this example, machine learning module 110 may generate additional suggested text 479A, which may be a suggested reply to text message 474 that reads, “Game night! That’s great!” In some examples, user 120 may edit suggested text 479A. For example, user 120 may interact with draggable circle 481 to emphasize a word in suggested text 479A. User 120 may interact with “Send” button 480 to automatically send suggested text 479A.

[0183] GUI 472C, also associated with the messaging application, may be a version of GUI 472B that is also updated based on user input. For example, user 120 may interact with draggable circle 481 to emphasize the word “great” in suggested text 479A of FIG. 4G. As shown in the example of FIG. 4H, suggested text 479A may be updated as suggested text 479B, which reads, “Game night! That’s greaaaaat!”. As such, one or more letters of a particular word in the suggested text may be repeated based on a distance that user 120 drags draggable circle 481. In some examples, the transition from suggested text 479A to suggested text 479B, and / or any transitions from any GUI and / or graphical component described herein to another GUI and / or graphical component, may include visual effects. For example, the visual effects may include dynamically changing font size, color, etc. User 120 may interact with “Send” button 480 to automatically send suggested text 479B.

[0184] GUI 472D, also associated with the messaging application, may be a version of GUI 472C that has also been updated based on user input. As shown in the example of FIG. 41, responsive to user 120 interacting with “Send” button 480 of FIG. 4H, suggested text 479B may be sent as text message 479C in reply to text message 474. Furthermore, as shown, responsive to computing system 100 determining that the task of replying to Ryan K.’sDocket No.: 1333-882WO01 message was performed, the instructions for generating GUI 419B and 472 may be updated to include instructions for generating check mark icon 483, so as to indicate that the task was performed. Furthermore, in some examples, GUI 419B may be updated to no longer include task widget 450B provided that the associated task was performed. In other examples, however, task widget 450B may be updated as a reminder (e.g., similar to a calendar reminder) to user 120 that user 120 will be attending game night at Ryan K.’s house at 7 PM on Friday, October 14th.

[0185] As shown in the example of FIG. 4 J, in some examples, the overlay GUIs may be presented in various formats. As an example, in some examples, GUI 484, which may be another example overlay GUI, may be divided into a number of different portions, e.g., a first portion 485 for a fixed header (such as header 466), a second portion 486 for an "activity viewer" (which may display a smaller “window” of the calendar application, and / or interactive application graphical components such as calendar event button 468), and a third portion 487 for an "Artificial Intelligence (Al) action module" (which may display intelligent suggestions). In some examples, second portion 486, e.g., the "activity viewer", and third portion 487, e.g., the "Al action module", may be expanded. That is, user 420 may interact with a portion to expand the height of the portion, such as to view additional information. Furthermore, in some examples, user 420 may interact with one or more graphical components to have a full screen GUI of an associated application be displayed.

[0186] In some examples, while an overlay GUI associated with a particular task widget is being displayed, user 420 may navigate to a different overlay GUI associated with a different task widget by swiping their screen in a vertical (e.g., up or down) direction. That is, in some examples, a user may "scroll" through the different overlay GUIs associated with different task widgets to perform their tasks. In some examples, user 420 may navigate to different overlay GUIs associated with the same task widget, and / or navigate to different portions of an overlay GUI by swiping their screen horizontally in a left or right direction, e.g., in horizontal direction 488. For example, third portion 487 may first display suggestions pertaining to the calendar application, such as suggested summary 469, "Schedule" button 470, and associated messaging application widget 418E of FIG. 4E. As shown in FIG. 4J, user 420 may interact with third portion 487, e.g., swipe horizontally from right to left in horizontal direction 488, to have third portion 487 display “Going” button 477 and “Not Going” button 478. That is, while second portion 486 displays content for the calendar application, user 420 may interact with third portion 487 to have other suggested outputs corresponding to other associated applications be displayed. In this example, while user 420Docket No.: 1333-882WO01 is still viewing the calendar application content in second portion 486, user 420 may select one of “Going” button 477 and “Not Going” button 478. That is, user 420 may select a suggested reply to text message 474 of FIG. 4F received by the messaging application, and thus send a reply in the messaging application without GUI 472 A of FIG. 4F even being displayed.

[0187] In some examples, first portion 485 may display a global prompt that can be edited by a user and applied globally, i.e., can be provided as input to a number of associated applications. That is, for an example overlay GUI for a rideshare application, first portion 485 may display the user’s desired route, e.g., “Home to School.” The “Home to School” prompt may be provided as input to a first associated application, such as the rideshare application, as well as other associated applications, such as an alternative rideshare application, and geographic information system (e.g., mapping) application. In some examples, the suggestions, graphical components, etc. displayed in third portion 487 may correspond to one or more application capabilities, and / or actions that a user may perform within the application and / or beyond the application. In the example of the rideshare application overlay GUI, third portion 487 may display buttons corresponding to different types of transportation that a user may select. Furthermore, in some examples, third portion 487 may display an “Al chat,” or text generated by the computing system that provides relevant information and / or live updates for the task being performed.

[0188] As such, in general, computing system 100 may generate instructions for GUIs, graphical components, and suggested data that can help complete a user’s tasks in a “shortcut” manner, as the GUIs and graphical components may provide functionality from one or more applications that computing system 100 has identified as being pertinent to performing respective tasks. Furthermore, suggested data determined by computing system 100 may be fine-tuned and customized by the user; as such, the user may quickly and easily change suggested input parameters for actions pertaining to their tasks. Furthermore, the design and layout of the GUIs (e.g., the main task list GUI and the associated application overlay GUIs) and graphical components (e.g., the task widgets and the specific graphical components of the associated application overlay GUIs) may present relevant task information and actions in a more concise, less busy, and more organized manner, which may reduce the overwhelmingness of performing tasks. Lastly, provided that computing system 100 may retrieve context information from background applications (e.g., data such as application data, user data, historical user data, time data, event data, notification data, etc.) and from a device and / or the system itself (e.g., data such as battery data, sensor data,Docket No.: 1333-882WO01 security data, user feedback data, system data, device data, environmental data, etc.), computing system 100 may identify tasks that extend beyond tasks easily identified based on, for example, notification data. That is, because of the variety of context information that computing system 100 may retrieve and analyze, computing system 100 may more intuitively determine a user’s various tasks. In this way, the techniques described herein may reduce the mental load, complexity, and time required for users to complete their various tasks, and therefore may provide an overall improved user experience when operating user devices.

[0189] FIG. 5 is a flowchart illustrating an example operation for dynamically generating custom graphical user interfaces for performing one or more tasks identified based on retrieved context information, in accordance with one or more techniques of this disclosure. The example of FIG. 5 is described with respect to FIGS. 1-41.

[0190] Computing system 100 retrieves, via API module 106, context information from one or more applications (590). In some examples, the one or more applications are one or more background applications. In some examples, the one or more applications are installed at computing device 112. For example, in some examples, API module 106 retrieves context information from the applications associated with applications widgets 118A-1181. In some examples, the context information includes one or more of application data, user data, historical user data, user feedback data, system data, device data, environmental data, time data, event data, notification data, device battery data, sensor data, and security data. In some examples, computing system 100 stores the retrieved context information in instructions storage 222.

[0191] Computing system 100 applies machine learning module 310 to the context information to identify one or more tasks (592). In some examples, machine learning module 310 applies language model module 342 to the context information to identify the one or more tasks. Machine learning module 310 determines, for each of the one or more tasks, one or more associated applications (594). In some examples, each of the one or more associated applications includes one or more functions for performing a respective task. Machine learning module 310 assigns a priority score to each of the one or more tasks (596).

[0192] User interface generator module 108 generates instructions file 346 for generating GUI 419A including a plurality of graphical components, such as task widgets 450A-450F (598). In some examples, each task widget is associated with a respective task. In some examples, an arrangement of each task widget within GUI 419A is based on the respective priority score assigned to the respective task. In some examples, the context information retrieved by API module 106 includes application data indicative of one or more userDocket No.: 1333-882WO01 contacts. In these examples, machine learning module 310 determines, based on historical user data, one or more user contacts associated with a messaging frequency greater than a threshold. In these examples, user interface generator module 108 generates instructions file 346 for generating GUI 419A, in which GUI 419A may include at least one graphical component associated with the one or more user contacts associated with the messaging frequency greater than the threshold.

[0193] In some examples, computing system 100 receives an indication of a gesture detected at a location of a UI component 102 that corresponds to a graphical component from the first plurality of graphical components, such as task widget 450D. In some examples, computing system 100 generates, based on the indication of the gesture, instructions for generating one or more additional graphical user interfaces, in which each of the one or more additional graphical user interfaces is associated with one of the one or more associated applications determined for a respective task. For example, computing system 100 generates, based on the indication that corresponds to task widget 450D, instructions for generating associated banking application overlay GUIs 451 A, 45 IB, and 451C. In some examples, each of the one or more additional GUIs includes a subset of graphical components from a second plurality of graphical components associated with the one or more functions for performing the respective task. For example, associated banking application overlay GUI 451 A includes graphical components such as text entry field 452A, text entry field 453A, and “Send” button 454A, associated banking application overlay GUI 45 IB includes graphical components such as text entry field 452B, text entry field 453B, “Send” button 454B, header 460, text entry field 455, text entry field 456, and “Request” button 462, and associated banking application overlay GUI 451C includes text output 464.

[0194] In some examples, computing system 100 applies machine learning module 310 to the context information to determine suggested data, in which the second plurality of graphical components associated with the one or more functions for performing the respective task includes the suggested data. For example, text entry field 453B, text entry field 452B, and text entry field 456 of associated banking application overlay GUI 45 IB include suggested input text data “John_Doe,” “$20.00,” and “Bank Account ending in 1234,” respectively.

[0195] In some examples, computing system 100 receives an indication of a gesture detected at a location of a UI component 102 that corresponds to a graphical component from the second plurality of graphical components associated with the one or more functions for performing the respective task, such as “Send” button 454B of associated banking application overlay GUI 45 IB. In some examples, machine learning module 310 may determine, basedDocket No.: 1333-882WO01 on one or more of the indication of the gesture and additional context information, whether the respective task was performed. In some examples, responsive to determining that the respective task was performed, user interface generator module 108 updates instructions file 346 for generating GUI 419A to indicate the respective task was performed. For example, GUI 419A may be updated to GUI 419B, in which GUI 419B no longer includes task widget 450D.

[0196] In some examples, a task includes a plurality of actions. In these examples, to determine the one or more associated applications for the task, machine learning module 310 identifies, based on a first subset of actions from the plurality of actions, a first associated application that includes one or more functions for performing the first subset of actions. For example, to determine the one or more associated applications for the task associated with task widget 450B, which may be a task of responding to party invite from Ryan K. sent over text message, machine learning module 310 may identify a calendar application that includes one or more functions for performing scheduling actions. Machine learning module 310 further identifies, based on a second subset of actions from the plurality of actions, a second associated application that includes one or more functions for performing the second subset of actions. Continuing the example, machine learning module 310 may identify a messaging application that includes one or more functions for performing messaging actions. In some examples, a first additional GUI from the one or more additional GUIs is associated with the first associated application, and a second additional GUI from the one or more additional GUIs is associated with the second associated application. For example, instructions file 346 may include instructions for generating overlay GUI 467 associated with the calendar application, and may include instructions for generating GUI 472 associated with the messaging application.

[0197] In some examples, the first additional GUI includes a first subset of graphical components from the second plurality of graphical components, in which the first subset of graphical components is associated with the one or more functions for performing the first subset of actions. In some examples, the second additional GUI includes a second subset of graphical components from the second plurality of graphical components, in which the second subset of graphical components is associated with the one or more functions for performing the second subset of actions. For example, overlay GUI 467 associated with the calendar application may include add calendar event button 468, text summary 469, and “Schedule” button 470. GUI 472A associated with the messaging application may include text message 474, text entry field 476, “Going” button 477, and “Not Going” button 478.Docket No.: 1333-882WO01GUI 472B, also associated with the messaging application, may be a version of GUI 472A updated based on user input, and may include suggested text message 479A, draggable circle 481, and “Send” button 480. GUI 472C, also associated with the messaging application, may be a version of GUI 472B updated based on user input, and may include suggested text message 479B.

[0198] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over, as one or more instructions or code, a computer- readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media, which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that may be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0199] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection is properly termed a computer- readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usuallyDocket No.: 1333-882WO01 reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0200] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0201] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, various units may be combined in a hardware unit or provided by a collection of intraoperative hardware units, including one or more processors, in conjunction with suitable software and / or firmware.

[0202] It is to be recognized that, depending on the example, certain acts or events of any of the techniques described herein may be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi -threaded processing, interrupt processing, or multiple processors, rather than sequentially.

[0203] In some examples, a computer-readable storage medium comprises a non-transitory medium. The term “non-transitory” indicates that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).

[0204] Example 1 : A method includes retrieving, by a computing system, context information from one or more applications; applying, by the computing system, a machine learning model to the context information to identify one or more tasks; determining, by the computing system, and for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task; assigning, by the computing system, a respective priority score to each task from the one or more tasks; and generating, by the computingDocket No.: 1333-882WO01 system, instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

[0205] Example 2: The method of example 1, wherein the plurality of graphical components is a first plurality of graphical components, the method further includes receiving, by the computing system, an indication of a gesture detected at a location of an input device that corresponds to a graphical component from the first plurality of graphical components; and generating, by the computing system, and based on the indication of the gesture, instructions for generating one or more additional graphical user interfaces, wherein each of the one or more additional graphical user interfaces is associated with one of the one or more associated applications determined for a respective task, and wherein each of the one or more additional graphical user interfaces includes a subset of graphical components from a second plurality of graphical components associated with the one or more functions for performing the respective task.

[0206] Example 3 : The method of example 2, wherein the respective task includes a plurality of actions, and wherein determining the one or more associated applications for the respective task further comprises: identifying, by the computing system, and based on a first subset of actions from the plurality of actions, a first associated application that includes one or more functions for performing the first subset of actions; and identifying, by the computing system, and based on a second subset of actions from the plurality of actions, a second associated application that includes one or more functions for performing the second subset of actions, wherein a first additional graphical user interface from the one or more additional graphical user interfaces is associated with the first associated application, and wherein a second additional graphical user interface from the one or more additional graphical user interfaces is associated with the second associated application.

[0207] Example 4: The method of example 3, wherein the first additional graphical user interface includes a first subset of graphical components from the second plurality of graphical components, wherein the first subset of graphical components is associated with the one or more functions for performing the first subset of actions, wherein the second additional graphical user interface includes a second subset of graphical components from the second plurality of graphical components, and wherein the second subset of graphicalDocket No.: 1333-882WO01 components is associated with the one or more functions for performing the second subset of actions.

[0208] Example 5: The method of any of examples 2 through 4, the method further includes applying, by the computing system, the machine learning model to the context information to determine suggested data, wherein the second plurality of graphical components associated with the one or more functions for performing the respective task includes the suggested data.

[0209] Example 6: The method of any of examples 2 through 5, further includes receiving, by the computing system, an indication of a gesture detected at a location of the input device that corresponds to a graphical component from the second plurality of graphical components associated with the one or more functions for performing the respective task; determining, by the computing system, and based on one or more of the indication of the gesture and additional context information, whether the respective task was performed; and responsive to determining that the respective task was performed, updating, by the computing system, the instructions for generating the graphical user interface to indicate the respective task was performed.

[0210] Example 7: The method of any of examples 1 through 6, wherein the context information includes one or more of application data, user data, historical user data, user feedback data, system data, device data, environmental data, time data, event data, notification data, device battery data, sensor data, and security data.

[0211] Example 8: The method of example 7, wherein the context information includes application data indicative of one or more user contacts, the method further includes determining, by the computing system, and based on the historical user data, one or more user contacts associated with a messaging frequency greater than a threshold; and generating, by the computing system, the instructions for generating the graphical user interface, wherein the graphical user interface includes at least one graphical component associated with the one or more user contacts associated with the messaging frequency greater than the threshold.

[0212] Example 9: The method of any of examples 1 through 8, wherein the machine learning model is a language model.

[0213] Example 10: The method of any of examples 1 through 9, wherein the one or more applications are one or more background applications.

[0214] Example 11 : The method of example 10, wherein the one or more applications are one or more background applications operating in the background of a computing device, the method further includes sending, by the computing system and to the computing device, the instructions.Docket No.: 1333-882WO01

[0215] Example 12: A computing system includes one or more processors; and one or more storage devices that store instructions, that, when executed by the one or more processors, cause the one or more processors to: retrieve context information from one or more applications; apply a machine learning model to the context information to identify one or more tasks; determine, for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task; assign a respective priority score to each task from the one or more tasks; and generate instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

[0216] Example 13: The computing system of example 12, wherein the plurality of graphical components is a first plurality of graphical components, and wherein the instructions further cause the one or more processors to: receive an indication of a gesture detected at a location of an input device that corresponds to a graphical component from the first plurality of graphical components; and generate, based on the indication of the gesture, instructions for generating one or more additional graphical user interfaces, wherein each of the one or more additional graphical user interfaces is associated with one of the one or more associated applications determined for a respective task, and wherein each of the one or more additional graphical user interfaces includes a subset of graphical components from a second plurality of graphical components associated with the one or more functions for performing the respective task.

[0217] Example 14: The computing system of example 13, wherein the respective task includes a plurality of actions, and wherein to determine the one or more associated applications for the respective task, the instructions further cause the one or more processors to: identify, based on a first subset of actions from the plurality of actions, a first associated application that includes one or more functions for performing the first subset of actions; and identify, based on a second subset of actions from the plurality of actions, a second associated application that includes one or more functions for performing the second subset of actions, wherein a first additional graphical user interface from the one or more additional graphical user interfaces is associated with the first associated application, and wherein a second additional graphical user interface from the one or more additional graphical user interfaces is associated with the second associated application.Docket No.: 1333-882WO01

[0218] Example 15: The computing system of example 14, wherein the first additional graphical user interface includes a first subset of graphical components from the second plurality of graphical components, wherein the first subset of graphical components is associated with the one or more functions for performing the first subset of actions, wherein the second additional graphical user interface includes a second subset of graphical components from the second plurality of graphical components, and wherein the second subset of graphical components is associated with the one or more functions for performing the second subset of actions.

[0219] Example 16: The computing system of any of examples 13 through 15, wherein the instructions further cause the one or more processors to: apply the machine learning model to the context information to determine suggested data, wherein the second plurality of graphical components associated with the one or more functions for performing the respective task includes the suggested data.

[0220] Example 17: The computing system of any of examples 13 through 16, wherein the instructions further cause the one or more processors to: receive an indication of a gesture detected at a location of the input device that corresponds to a graphical component from the second plurality of graphical components associated with the one or more functions for performing the respective task; determine, based on one or more of the indication of the gesture and additional context information, whether the respective task was performed; and responsive to determining that the respective task was performed, update the instructions for generating the graphical user interface to indicate the respective task was performed.

[0221] Example 18: The computing system of any of examples 12 through 17, wherein the context information includes one or more of application data, user data, historical user data, user feedback data, system data, device data, environmental data, time data, event data, notification data, device battery data, sensor data, and security data.

[0222] Example 19: The computing system of example 18, wherein the context information includes application data indicative of one or more user contacts, wherein the instructions further cause the one or more processors to: determine, based on the historical user data, one or more user contacts associated with a messaging frequency greater than a threshold; and generate the instructions for generating the graphical user interface, wherein the graphical user interface includes at least one graphical component associated with the one or more user contacts associated with the messaging frequency greater than the threshold.

[0223] Example 20: The computing system of any of examples 12 through 19, wherein the machine learning model is a language model.Docket No.: 1333-882WO01

[0224] Example 21 : The computing system of any of examples 12 through 20, wherein the one or more applications are one or more background applications.

[0225] Example 22: The computing system of example 21, wherein the one or more applications are one or more background applications operating in a background of a computing device, wherein the instructions further cause the one or more processors to: send, to the computing device, the instructions.

[0226] Example 23 : A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors, cause one or more processors to: retrieve context information from one or more applications; apply a machine learning model to the context information to identify one or more tasks; determine, for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task; assign a respective priority score to each task from the one or more tasks; and generate instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

[0227] Example 24: The non-transitory computer-readable storage medium of example 23, wherein the plurality of graphical components is a first plurality of graphical components, and wherein the instructions further cause the one or more processors to: receive an indication of a gesture detected at a location of an input device that corresponds to a graphical component from the first plurality of graphical components; and generate, based on the indication of the gesture, instructions for generating one or more additional graphical user interfaces, wherein each of the one or more additional graphical user interfaces is associated with one of the one or more associated applications determined for a respective task, and wherein each of the one or more additional graphical user interfaces includes a subset of graphical components from a second plurality of graphical components associated with the one or more functions for performing the respective task.

[0228] Example 25: The non-transitory computer-readable storage medium of example 24, wherein the respective task includes a plurality of actions, and wherein to determine the one or more associated applications for the respective task, the instructions further cause the one or more processors to: identify, based on a first subset of actions from the plurality of actions, a first associated application that includes one or more functions for performing the firstDocket No.: 1333-882WO01 subset of actions; and identify, based on a second subset of actions from the plurality of actions, a second associated application that includes one or more functions for performing the second subset of actions, wherein a first additional graphical user interface from the one or more additional graphical user interfaces is associated with the first associated application, and wherein a second additional graphical user interface from the one or more additional graphical user interfaces is associated with the second associated application.

[0229] Example 26: The non-transitory computer-readable storage medium of example 25, wherein the first additional graphical user interface includes a first subset of graphical components from the second plurality of graphical components, wherein the first subset of graphical components is associated with the one or more functions for performing the first subset of actions, wherein the second additional graphical user interface includes a second subset of graphical components from the second plurality of graphical components, and wherein the second subset of graphical components is associated with the one or more functions for performing the second subset of actions.

[0230] Example 27: The non-transitory computer-readable storage medium of any of examples 24 through 26, wherein the instructions further cause the one or more processors to: apply the machine learning model to the context information to determine suggested data, wherein the second plurality of graphical components associated with the one or more functions for performing the respective task includes the suggested data.

[0231] Example 28: The non-transitory computer-readable storage medium of any of examples 24 through 27, wherein the instructions further cause the one or more processors to: receive an indication of a gesture detected at a location of the input device that corresponds to a graphical component from the second plurality of graphical components associated with the one or more functions for performing the respective task; determine, based on one or more of the indication of the gesture and additional context information, whether the respective task was performed; and responsive to determining that the respective task was performed, update the instructions for generating the graphical user interface to indicate the respective task was performed.

[0232] Example 29: The non-transitory computer-readable storage medium of any of examples 23 through 28, wherein the context information includes one or more of application data, user data, historical user data, user feedback data, system data, device data, environmental data, time data, event data, notification data, device battery data, sensor data, and security data.Docket No.: 1333-882WO01

[0233] Example 30: The non-transitory computer-readable storage medium of example 29, wherein the context information includes application data indicative of one or more user contacts, wherein the instructions further cause the one or more processors to: determine, based on the historical user data, one or more user contacts associated with a messaging frequency greater than a threshold; and generate the instructions for generating the graphical user interface, wherein the graphical user interface includes at least one graphical component associated with the one or more user contacts associated with the messaging frequency greater than the threshold.

[0234] Example 31 : The non-transitory computer-readable storage medium of any of examples 23 through 30, wherein the machine learning model is a language model.

[0235] Example 32: The non-transitory computer-readable storage medium of any of examples 23 through 31, wherein the one or more applications are one or more background applications.

[0236] Example 33: The non-transitory computer-readable storage medium of example 32, wherein the one or more applications are one or more background applications operating in a background of a computing device, wherein the instructions further cause the one or more processors to: send, to the computing device, the instructions.

[0237] Example 34: A computer program product for generating custom user interfaces for performing tasks associated with background applications, the computer program product comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to: retrieve context information from one or more applications; apply a machine learning model to the context information to identify one or more tasks; determine, for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task; assign a respective priority score to each task from the one or more tasks; and generate instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

[0238] Example 35: The computer program product of example 34, wherein the plurality of graphical components is a first plurality of graphical components, and wherein the one or more instructions further cause the at least one processor to: receive an indication of a gesture detected at a location of an input device that corresponds to a graphical component from theDocket No.: 1333-882WO01 first plurality of graphical components; and generate, based on the indication of the gesture, instructions for generating one or more additional graphical user interfaces, wherein each of the one or more additional graphical user interfaces is associated with one of the one or more associated applications determined for a respective task, and wherein each of the one or more additional graphical user interfaces includes a subset of graphical components from a second plurality of graphical components associated with the one or more functions for performing the respective task.

[0239] Example 36: The computer program product of example 35, wherein the respective task includes a plurality of actions, and wherein to determine the one or more associated applications for the respective task, the one or more instructions further cause the at least one processor to: identify, based on a first subset of actions from the plurality of actions, a first associated application that includes one or more functions for performing the first subset of actions; and identify, based on a second subset of actions from the plurality of actions, a second associated application that includes one or more functions for performing the second subset of actions, wherein a first additional graphical user interface from the one or more additional graphical user interfaces is associated with the first associated application, and wherein a second additional graphical user interface from the one or more additional graphical user interfaces is associated with the second associated application.

[0240] Example 37: The computer program product of example 36, wherein the first additional graphical user interface includes a first subset of graphical components from the second plurality of graphical components, wherein the first subset of graphical components is associated with the one or more functions for performing the first subset of actions, wherein the second additional graphical user interface includes a second subset of graphical components from the second plurality of graphical components, and wherein the second subset of graphical components is associated with the one or more functions for performing the second subset of actions.

[0241] Example 38: The computer program product of any of examples 34 through 37, wherein the one or more instructions further cause the at least one processor to: apply the machine learning model to the context information to determine suggested data, wherein the second plurality of graphical components associated with the one or more functions for performing the respective task includes the suggested data.

[0242] Example 39: The computer program product of any of examples 34 through 38, wherein the one or more instructions further cause the at least one processor to: receive an indication of a gesture detected at a location of the input device that corresponds to aDocket No.: 1333-882WO01 graphical component from the second plurality of graphical components associated with the one or more functions for performing the respective task; determine, based on one or more of the indication of the gesture and additional context information, whether the respective task was performed; and responsive to determining that the respective task was performed, update the instructions for generating the graphical user interface to indicate the respective task was performed.

[0243] Example 40: The computer program product of any of examples 34 through 39, wherein the context information includes one or more of application data, user data, historical user data, user feedback data, system data, device data, environmental data, time data, event data, notification data, device battery data, sensor data, and security data.

[0244] Example 41 : The computer program product of example 34, wherein the context information includes application data indicative of one or more user contacts, wherein the one or more instructions further cause the at least one processor to: determine, based on the historical user data, one or more user contacts associated with a messaging frequency greater than a threshold; and generate the instructions for generating the graphical user interface, wherein the graphical user interface includes at least one graphical component associated with the one or more user contacts associated with the messaging frequency greater than the threshold.

[0245] Example 42: The computer program product of any of examples 34 through 41, wherein the machine learning model is a language model.

[0246] Example 43: The computer program product of any of examples 34 through 42, wherein the one or more applications are one or more background applications.

[0247] Example 44: The computer program product of example 43, wherein the one or more applications are one or more background applications operating in a background of a computing device, wherein the one or more instructions further cause the at least one processor to: send, to the computing device, the instructions.

Claims

Docket No.: 1333-882WO01WHAT IS CLAIMED IS:

1. A method comprising: retrieving, by a computing system, context information from one or more applications; applying, by the computing system, a machine learning model to the context information to identify one or more tasks; determining, by the computing system, and for each of the one or more tasks, one or more associated applications, wherein each of the one or more associated applications includes one or more functions for performing a respective task; assigning, by the computing system, a respective priority score to each task from the one or more tasks; and generating, by the computing system, instructions for generating a graphical user interface including a plurality of graphical components, wherein each graphical component from the plurality of graphical components is associated with a respective task from the one or more tasks, and wherein the plurality of graphical components is arranged in the graphical user interface based on the respective priority score assigned to the respective task.

2. The method of claim 1, wherein the plurality of graphical components is a first plurality of graphical components, the method further comprising: receiving, by the computing system, an indication of a gesture detected at a location of an input device that corresponds to a graphical component from the first plurality of graphical components; and generating, by the computing system, and based on the indication of the gesture, instructions for generating one or more additional graphical user interfaces, wherein each of the one or more additional graphical user interfaces is associated with one of the one or more associated applications determined for a respective task, and wherein each of the one or more additional graphical user interfaces includes a subset of graphical components from a second plurality of graphical components associated with the one or more functions for performing the respective task.Docket No.: 1333-882WO013. The method of claim 2, wherein the respective task includes a plurality of actions, and wherein determining the one or more associated applications for the respective task further comprises: identifying, by the computing system, and based on a first subset of actions from the plurality of actions, a first associated application that includes one or more functions for performing the first subset of actions; and identifying, by the computing system, and based on a second subset of actions from the plurality of actions, a second associated application that includes one or more functions for performing the second subset of actions, wherein a first additional graphical user interface from the one or more additional graphical user interfaces is associated with the first associated application, and wherein a second additional graphical user interface from the one or more additional graphical user interfaces is associated with the second associated application.

4. The method of claim 3, wherein the first additional graphical user interface includes a first subset of graphical components from the second plurality of graphical components, wherein the first subset of graphical components is associated with the one or more functions for performing the first subset of actions, wherein the second additional graphical user interface includes a second subset of graphical components from the second plurality of graphical components, and wherein the second subset of graphical components is associated with the one or more functions for performing the second subset of actions.

5. The method of claim 2, the method further comprising: applying, by the computing system, the machine learning model to the context information to determine suggested data, wherein the second plurality of graphical components associated with the one or more functions for performing the respective task includes the suggested data.

6. The method of claim 2, further comprising: receiving, by the computing system, an indication of a gesture detected at a location of the input device that corresponds to a graphical component from the second plurality of graphical components associated with the one or more functions for performing theDocket No.: 1333-882WO01 respective task; determining, by the computing system, and based on one or more of the indication of the gesture and additional context information, whether the respective task was performed; and responsive to determining that the respective task was performed, updating, by the computing system, the instructions for generating the graphical user interface to indicate the respective task was performed.

7. The method of any of claims 1 through 6, wherein the context information includes one or more of application data, user data, historical user data, user feedback data, system data, device data, environmental data, time data, event data, notification data, device battery data, sensor data, and security data.

8. The method of claim 7, wherein the context information includes application data indicative of one or more user contacts, the method further comprising: determining, by the computing system, and based on the historical user data, one or more user contacts associated with a messaging frequency greater than a threshold; and generating, by the computing system, the instructions for generating the graphical user interface, wherein the graphical user interface includes at least one graphical component associated with the one or more user contacts associated with the messaging frequency greater than the threshold.

9. The method of claim 1, wherein the machine learning model is a language model.

10. The method of claim 1, wherein the one or more applications are one or more background applications.

11. The method of claim 10, wherein the one or more applications are one or more background applications operating in the background of a computing device, the method further comprising: sending, by the computing system and to the computing device, the instructions.

12. A computing system comprising means for performing any combination of the methods of claims 1 through 11.Docket No.: 1333-882WO0113. A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors, cause one or more processors to perform any combination of the methods of claims 1 through 11.

14. A computer program product encoded with instructions that, when executed by one or more processors, cause one or more processors to perform any combination of the methods of claims 1 through 11.

15. A computing device comprising means for performing any combination of the methods of claims 1 through 11.

Images