Appless glanceable widgets

Abstract

A computing device is configured to receive a request for information. The computing device is further configured to generate a prompt. The computing device further configured to generate, using the prompt as input to a large language model, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information. The computing device is further configured to obtain the information of the request for information. The computing device is further configured to generate, using a selected user interface template from the at least one user interface template, a user interface, wherein the user interface includes one or more visual elements that each replace a respective placeholder user interface element of the one or more placeholder user interface elements. The computing device is further configured to output the user interface.

Description

Docket No.: 1333-827WO01APPLESS GLANCEABLE WIDGETSBACKGROUND

[0001] Computing devices, such as mobile computing devices, may run a variety of software applications that display information to a user. In addition, the operating system of a device may generate a user interface that displays information to a user.SUMMARY

[0002] Techniques are described by which a computing device may use one or more large language models to generate widgets for a user interface of the computing device. In accordance with the techniques of this disclosure, the computing device may receive a request for information that may include natural language of a user interaction with the computing device (e.g., a user asking, “When are the low and high tides in Norfolk?”). The computing device may generate a prompt that includes configuration information of the computing device such as display resolution and shape. The computing device generates, using the prompt as input to a large language model, one or more user interface templates that include placeholder user interface elements corresponding to placeholder information. The computing receives user input of a selection of one of the user interface templates and obtains information to fulfill the request for information from one or more sources of information (e.g., obtaining information about tides from a government weather website, obtaining weather information from the memory of the computing device, etc.). The computing device uses the selected user interface template and the obtained information to generate a user interface that includes one or more visual elements that each replace the placeholder user interface elements. In this way, the computing device may enable a user to create a customized widget within the user interface that displays information sought by the user.

[0003] In some examples, a method includes receiving, by a computing device, a request for information; generating, by the computing device and based on the request for information, a prompt, wherein the prompt includes configuration information of the computing device; generating, by the computing device and using the prompt as input to a large language model, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information; obtaining, by the computing device and from a data source, the information of the request for information; generating, by the computing device and using the information obtained from the data source and a selected user interface template from the at least one userinterface template, a user interface, wherein the user interface includes one or more visual elements that each replace a respective placeholder user interface element of the one or more placeholder user interface elements; and outputting, by the computing device and via one or more display components, the user interface.

[0004] In some examples, a computing system includes one or more display components, a memory, and one or more programmable processors in communication with the memory and configured to receive a request for information; generate, based on the request for information, a prompt, wherein the prompt includes configuration information of the computing device; generate, using the prompt as input to a large language model, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information; obtain, from a data source, the information of the request for information; generate, using the information obtain from the data source and a selected user interface template from the at least one user interface template, a user interface, wherein the user interface includes one or more visual elements that each replace a respective placeholder user interface element of the one or more placeholder user interface elements; and output, via the one or more display components, the user interface.

[0005] In some examples, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of a computing device, causes at least one processor of a computing device to receive a request for information; generate, based on the request for information, a prompt, wherein the prompt includes configuration information of the computing device; generate, using the prompt as input to a large language model, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information; obtain, from a data source, the information of the request for information; generate, using the information obtain from the data source and a selected user interface template from the at least one user interface template, a user interface, wherein the user interface includes one or more visual elements that each replace a respective placeholder user interface element of the one or more placeholder user interface elements; and output, via one or more display components, the user interface.

[0006] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS

[0007] FIG. 1 is a conceptual diagram illustrating an example computing system for generating user interfaces based on requests for information, in accordance with one or more aspects of the present disclosure.

[0008] FIG. 2 is a block diagram illustrating details of a mobile computing device for generating user interfaces, in accordance with one or more techniques of this disclosure.

[0009] FIG. 3 is a block diagram illustrating details of a computing device for generating user interfaces, in accordance with one or more techniques of this disclosure.

[0010] FIG. 4 is a block diagram illustrating details of a computing system for generating user interfaces, in accordance with one or more techniques of this disclosure.

[0011] FIGS. 5A-5B are conceptual diagrams illustrating an example animation and user interface, in accordance with one or more techniques of this disclosure.

[0012] FIG. 6 is a flowchart illustrating an example operation for generating a user interface, in accordance with one or more techniques of this disclosure.DETAILED DESCRIPTION

[0013] FIG. 1 is a conceptual diagram illustrating an example computing system for generating user interfaces based on requests for information in accordance with one or more techniques of this disclosure. As shown in the example of FIG. 1, computing system 100 includes mobile computing device 102, computing device 130, and remote computing system 140. Examples of mobile computing devices 102 may include, but are not limited to, portable, mobile, or other devices, such as mobile phones (including smartphones), wearable computing devices (e.g., smart watches, smart glasses, etc.) virtual reality (VR) headsets, augmented reality (AR) headsets / glasses, and other types of mobile computing devices. Mobile computing device 102 may include a plurality of software and / or hardware components. Examples of computing devices 130 include portable, mobile, or other devices, such as mobile phones (including smartphones), laptop computers, desktop computers, tablet computers, smart television platforms, infotainment systems (e.g., vehicle head units), etc. Examples of computing systems 120 may include, but are not limited to server computers, mainframes, cloud compute nodes, distributed computing environments, and virtualized computing environments.

[0014] As shown in the example of FIG. 1, mobile computing device 102 includes one or more user interface devices such as user interface (“UI”) device 104. UI device 104 of mobile computing device 102 may be configured to function as an input device and / or anoutput device for mobile computing device 102. UI device 104 may be implemented using various technologies. For instance, UI device 104 may be configured to receive input from a user through tactile, audio, and / or video feedback. Examples of input devices include a presence-sensitive display, a presence-sensitive or touch-sensitive input device, a mouse, a keyboard, a voice responsive system, video camera, microphone or any other type of device for detecting a command from a user. In some examples, a presence-sensitive display includes a touch-sensitive or presence-sensitive input screen, such as a resistive touchscreen, a surface acoustic wave touchscreen, a capacitive touchscreen, a projective capacitive touchscreen, a pressure sensitive screen, an acoustic pulse recognition touchscreen, or another presence-sensitive technology. That is, UI device 104 of mobile computing device 102 may include a presence-sensitive device that may receive tactile input from a user of mobile computing device 102. UI device 104 may receive indications of the tactile input by detecting one or more gestures from the user (e.g., when the user touches or points to one or more locations of UI device 104 with a finger or a stylus pen).

[0015] UI device 104 may additionally or alternatively function as an output device by providing output to a user using tactile, audio, or video stimuli. Examples of output devices include a sound card, a video graphics adapter card, or any of one or more display devices, such as a liquid crystal display (LCD), dot matrix display, light emitting diode (LED) display, miniLED, microLED, organic light-emitting diode (OLED) display, e-ink, or similar monochrome or color display capable of outputting visible information to a user of mobile computing device 102. Additional examples of an output device include a speaker, a haptic device, or other device that can generate intelligible output to a user. For instance, UI device 104 may present output to a user of mobile computing device 102 as a graphical user interface that may be associated with functionality provided by mobile computing device 102. In this way, UI device 104 may present various user interfaces of applications executing at or accessible by mobile computing device 102 (e.g., an electronic message application, a health tracking application, a media player application, etc.). A user of mobile computing device 102 may interact with a respective user interface of an application to cause mobile computing device 102 to perform operations relating to a function.

[0016] In some examples, UI device 104 of mobile computing device 102 may detect two- dimensional and / or three-dimensional gestures as input from a user of mobile computing device 102. For instance, a sensor of UI device 104 may detect the user's movement (e.g., moving a hand, an arm, a pen, a stylus, etc.) within a threshold distance of the sensor of UI device 104. UI device 104 may determine a two- or three-dimensional vector representationof the movement and correlate the vector representation to a gesture input (e.g., a hand-wave, a pinch, a clap, a pen stroke, etc.) that has multiple dimensions. In other words, UI device 104 may, in some examples, detect a multidimensional gesture without requiring the user to gesture at or near a screen or surface at which UI device 104 outputs information for display. Instead, UI device 104 may detect a multi-dimensional gesture performed at or near a sensor which may or may not be located near the screen or surface at which UI device 104 outputs information for display.

[0017] In the example of FIG. 1, mobile computing device 102 includes user interface (UI) module 106. UI module 106 may perform operations described herein using hardware, software, firmware, or a mixture thereof residing in and / or executing at mobile computing device 102. Mobile computing device 102 may execute UI module 106 with one processor or with multiple processors. In some examples, mobile computing device 102 may execute UI module 106 as a virtual machine executing on underlying hardware. UI module 106 may execute as one or more services of an operating system or computing platform or may execute as one or more executable programs at an application layer of a computing platform.

[0018] UI module 106, as shown in the example of FIG. 1, may be operable by mobile computing device 102 to perform one or more functions, such as receiving input and sending indications of such input to other components associated with mobile computing device 102. For example, UI module 106 may receive data from a component such as UI device 104. UI module 106 may also receive data from components associated with mobile computing device 102 such as UI device 104. Using the data received, UI module 106 may cause other components associated with mobile computing device 102, such as UI device 104, to provide output based on the data. UI module 106 may generate an indication of the data received and provide an indication of the data received to one or more components of mobile computing device 102. For instance, UI module 106 may receive data from one or more application modules 108A-108N (hereinafter “application modules 108”) to display a graphical user interface (“GUI”) such as GUI 150. UI module 106 may provide an indication to UI device 104 to display GUI 150.

[0019] Application modules 108 may perform operations described herein using hardware, software, firmware, or a mixture thereof residing in and / or executing at mobile computing device 102. Mobile computing device 102 may execute application modules 108 with one or more processors. Application module 108 may include functionality to perform any variety of operations on mobile computing device 102. For instance, application modules 108 may include a word processor, a text application, a web browser, a multimedia player, a calendarapplication, a distributed computing application, a graphic design application, a video editing application, a web development application, or any other application. One of application modules 108 may be a text message, Short Message Service (SMS), and / or Rich Communication Services (RCS) application. Application modules 108 may interact with UI module 106 and enable and / or extend functionality for one or more features of an application executed by mobile computing device 102. For example, application modules 108 may include a multimedia player that enables mobile computing device 102 to play visual media acquired from another computing device such as computing device 130.

[0020] Mobile computing device 102 may communicate with other computing devices and / or systems such as computing device 130 and / or remote computing system 140 via one or more networks. Mobile computing device 102 may communicate with other computing devices via one or more networks and communication protocols such as cellular networks, WIFI networks, satellite communication networks, BLUETOOTH communications, ultra-wideband (UWB) communications, or other type of networks and communication protocols for transmitting data between computing devices. In some examples, the network may represent one or more packet switched networks, such as the Internet. Mobile computing device 102 may send and receive data across the network using any suitable communication techniques. For example, mobile computing device 102 may be operatively coupled to one or more networks using respective network links. The network may include network hubs, network switches, network routers, terrestrial and / or satellite cellular networks, etc., that are operatively inter-coupled thereby providing for the exchange of information between mobile computing device 102 and another computing device. In some examples, network links of the network may be Ethernet, ATM or other network connections. Such connections may include wireless and / or wired connections.

[0021] Mobile computing device 102 may execute user interface (UI) orchestrator 114 to generate GUIs such as GUI 150. UI orchestrator 114 may be a module, plugin, standalone program, process, or other type of software component configured to manage the generation of GUI 150 by mobile computing device 102. UI orchestrator 114 may orchestrate the generation of GUI 150 by one or more software components of mobile computing device 102. For example, UI orchestrator 114 may orchestrate the generation of GUI 150 as including one or more visual elements generated by an operating system of mobile computing device 102 (e.g., a battery status indicator, a system clock, etc.) and one or more visual elements generated by an application executed by mobile computing device 102 (e.g., a user interface of a storefront application. In another example, UI orchestrator 114 generates GUI150 as including visual elements of a glanceable widget (e.g., a widget that includes one or more visual elements that display information for a user).

[0022] In general, a user may use applications executed by mobile computing device 102 to obtain information via glanceable widgets displayed by UI device 104 as part of GUI 150. In many cases, a user may wish to quickly view information via GUI 150 generated by mobile computing device 102 but not have an application installed on mobile computing device 102 that includes functionality to generate a glanceable widget that displays information sought by a user. For example, a user may wish to view the weather forecast via GUI 150, but mobile computing device 102 may not have a weather application installed that can generate glanceable widgets displaying the forecast. Further, mobile computing device 102 may be unable to obtain an application that can generate a widget that displays the information requested by a user.

[0023] In accordance with the techniques of this disclosure, mobile computing device 102 may include one or more of large language models 112A-112N (hereinafter “LLMs 112”) used by UI orchestrator 114 to generate a user interface that includes visual elements displaying information sought by a user. UI orchestrator 114 may use LLMs 112 to dynamically generate a user interface that includes the information sought by the user. UI orchestrator 114 may generate a prompt for LLMs 112 and use the output of LLMs 112 to generate GUI 150 as including the information sought by a user. In addition, UI orchestrator 114 may use LLMs 112 to aid in obtaining the information of a request for information. In this way, UI orchestrator 114 may enable mobile computing device 102 to generate glanceable widgets that include information sought by a user without requiring an application capable of generating widgets. In turn, this may enable mobile computing device 102 to quickly identify and display information to a user without requiring the user to download a dedicated application to view the information. In addition, this may enable mobile computing device 102 to dynamically generate glanceable widgets that are customized to answer requests from a user.

[0024] Mobile computing device 102 may receive user input consistent with a user requesting the creation of a widget such as a glanceable widget. In some examples, mobile computing device 102 may prompt a user to provide input to specify information that the user would like to have displayed in a widget. For example, mobile computing device 102 may generate and output GUI 150 as including widget 152 with the text “WHAT WOULD YOU LIKE TO SEE ON THIS TILE?” visually displayed within widget 152. In another example, mobile computing device 102 may generate other types of output such as auditory output toprompt a user to provide input and specify what type of information that the user would like to have displayed in a widget.

[0025] Mobile computing device 102 may use UI module 106 to identify requests for information received via UI device 104. UI device 104 may receive user input, e.g., spoken input, from one or more devices such as a microphone or touchscreen in response to the prompting for input by mobile computing device 102. UI device 104 may generate input data and provide the input data to UI module 106 for processing. UI module 106 may process the input data into one or more forms of data such as text (e.g., process the input data into a transcript of what the user said to mobile computing device 102).

[0026] UI module 106 may determine whether the user input corresponds to a request for information. UI module 106 may identify a request for information based on one or more factors such as identifying particular words and / or phrases within spoken input (e.g., “Generate a UI that tells me the trains that leave from Penn Station”, “When does the next flight from London to Frankfurt depart?”, “When is my doctor’s appointment?”, etc.), typed input to mobile computing device 102, and / or whether mobile computing device 102 is displaying a visual indication indicating that the user can determine what information should be displayed (e.g., widget 152 of GUI 150 displaying “WHAT WOULD YOU LIKE TO SEE ON THIS TILE?”) among other factors. Based on identifying a request for information, UI module 106 provides the request for information to UI orchestrator 114. In some examples, UI module 106 may provide the request for information to an intake LLM of LLMs 112 (e.g., an LLM trained to analyze requests for information) for the intake LLM to determine whether clarifying questions are needed to answer the request for information before the request for information is provided to UI orchestrator 114.

[0027] UI orchestrator 114 may generate prompts for LLMs 112 based on the request for information. Mobile computing device 102 may include one or more of LLMs 112 that include one or more types of machine learning models such as recurrent neural networks, convolutional neural networks, and transformers among other types of machine learning models. UI orchestrator 114 may generate a prompt for LLMs 112 that is based on the request for information and that includes configuration information of mobile computing device 102. In addition, UI orchestrator 114 may generate the prompt for LLMs 112 as including one or more requirements for the output from LLMs 112. For example, UI orchestrator 114 may generate the prompt as telling LLMs 112 that they are an “expert UX designer” and requiring LLMs 112 to perform one or more steps to generate the glanceable widget. In an example, UI orchestrator 114 generates the prompt as including a first step thatrequires an LLM of LLMs 112 to provide an itemized list that explains how the LLM will answer the prompt and a second step that requires the LLM to explain what information would be useful to show a user.

[0028] UI orchestrator 114 may generate the prompt as including a requirement for LLMs 112 to generate UI template configuration information that includes information about the configuration of one or more UI templates. For example, UI orchestrator 114 may generate the prompt as including a requirement for LLMs 112 to generate a UI template using JavaScript Object Notation (JSON). In addition, UI orchestrator 114 may generate the prompt as including a requirement that LLMs 112 identify natural groupings of data points within the request for information. For example, UI orchestrator 114 may include a requirement such as “Briefly explain whether there should be any natural grouping of data points in your response,” and one or more examples (e.g., “a query about stock prices may have each element representing a company”, “a query about exchange rates may have elements for each currency in the conversion”, etc.) of groupings in the prompt for LLMs 112. UI orchestrator 114 may use the requirement to identify natural groupings to cause LLMs 112 to generate information regarding data elements that are groupings of data points within the request for information and associated data element properties that are properties of the data elements. For example, UI orchestrator 114 may receive output from LLMs 112 of a data element representing the name of a company and an associated stock price of the company and data element property that indicates that the data element should be bound and therefore periodically updated with new information.

[0029] In some examples, UI orchestrator 114 uses a second LLM of LLMs 112 to generate one or more UI templates based on the template configuration information. UI orchestrator 114 may generate and provide a prompt that includes the output of a first LLM to a second LLM of LLMs 112 and receive data regarding at least one UI template from LLMs 112. UI orchestrator 114 may provide the prompt that includes the UI template configuration information received from a first LLM (e.g., information about the data elements and associated data element properties among other information) to a second LLM. UI orchestrator 114 may receive one or more UI templates from the second LLM in response to providing the prompt to the second LLM of LLMs 112. For example, UI orchestrator 114 may receive a UI template that is in an Extensible Markup Language (XML) format. UI orchestrator 114 may receive multiple user interface templates with varying designs from LLMs 112. For example, UI orchestrator 114 may receive one or more user interface templates with varying user interface layouts.

[0030] In some examples, UI orchestrator 114 may use predetermined rules regarding user interface configuration to generate one or more UI templates. UI orchestrator 114 may maintain a record of one or more rules regarding the generation of user interface such as rules regarding the shape and color of visual elements, relative positioning of visual elements within a user interface, and other rules regarding the configuration of visual elements within a user interface. In an example, UI orchestrator 114 generates UI template configuration information using a first LLM of LLMs 112. UI orchestrator 114 generates a UI template using the UI template configuration information and the rules regarding user interface configuration. UI orchestrator 114 may generate one or more UI templates in XML format.

[0031] UI orchestrator 114 may receive and / or generate placeholder data for a UI template that includes placeholder user interface elements that correspond to placeholder information that represents the information of the request for information. For example, UI orchestrator 114 may receive a plurality of UI templates that each include placeholder information representing the information of the request for information from LLMs 112 such as a UI template that includes placeholder data regarding the departure time of the next flight from New York to Los Angeles. UI orchestrator 114 may generate placeholder data for the placeholder visual elements when generating the UI template. UI orchestrator 114 may generate placeholder data when generating UI templates using predetermined rules regarding the configuration of visual elements. For each UI template generated, UI orchestrator 114 may generate placeholder information that represents the information of the request for information. UI orchestrator 114 may generate placeholder data that is randomly generated based on the data elements and data element properties. For example, for a data element that represents the time of day for the next flight from Los Angeles to Miami, UI orchestrator 114 generates a randomized time of day as placeholder data. UI orchestrator 114 may use LLMs 112 to generate placeholder data that include arbitrary values that are based on the type of information requested by the request for information.

[0032] UI orchestrator 114 may generate instances of GUI 150 based on the one or more UI templates. For example, UI orchestrator 114 may generate an instance of GUI 150 based on one of the UI templates and cause UI device 104 to output the instance of GUI 150 for display. UI orchestrator 114 may generate one or more instances of GUI 150 based on the UI templates to enable a user to select a UI template from among the one or more UI templates. For example, UI orchestrator 114 may generate GUI 150 as including a UI template replacing widget 152 and as including navigation elements 154.

[0033] UI orchestrator 114 may generate GUI 150 as including navigation elements 154 (e.g., the visual elements that include left- and right-facing triangles) to enable a user to navigate between different UI templates and to enable a user to indicate which UI template they prefer (e.g., the visual element of navigation elements 154 that includes a “+” sign). In an example, UI orchestrator 114 generates GUI 150 as including widget 152 replaced with a first UI template and causes UI device 104 to output GUI 150 for display. UI device 104 receives user input consistent with the selection of a navigation element of navigation elements 154 that corresponds to navigating to the right and provides an indication of the user input to UI orchestrator 114. UI orchestrator 114 generates an updated instance of GUI 150 as replacing the first UI template with the second UI template and causes UI device 104 to output the updated instance of GUI 150. For example, UI device 104 may receive user input consistent with a selection of a particular UI template (e.g., receiving user input at a location consistent with the navigation element that includes the “+” sign). UI device 104 may provide an indication of the user input to UI orchestrator 114.

[0034] UI orchestrator 114 causes data acquisition module 116 to obtain the information of the request for information in response to user input consistent with a selection of a UI template. Data acquisition module 116 may be a plugin, module, program, process, or other type of software component configured to obtain information from one or more sources of information. Data acquisition module 116 may obtain information from one or more sources of information such as webpages, remote computing system 140, the memory of mobile computing device 102, and other sources of information external and internal to mobile computing device 102.

[0035] Data acquisition module 116 may use one or more of LLMs 112 to obtain information for UI orchestrator 114. Data acquisition module 116 may generate a prompt for LLMs 112 that includes an indication of the information of the request for information and a request to locate a source for the information. For example, data acquisition module 216 may generate a prompt that includes a request to generate a query for a search engine to obtain the information of the request for information. In another example, data acquisition module 216 generates a prompt for LLMs 112 that requests that LLMs 112 identify one or more potential sources of information of the request for information.

[0036] Data acquisition module 116 may obtain information of the request for information based on the output of LLMs 112. Data acquisition module 116 may use one or more techniques to obtain the information such as inputting a query generated by LLMs 112 into a search engine and scraping data from one or more websites identified by the search engine,searching local files of mobile computing device 102, and obtaining data from computing device 130 among other techniques to obtain the information. In an example, data acquisition module 116 inputs search terms generated by LLMs 112 into a search engine and causes the search engine to identify a list of one or more websites based on the search terms. Data acquisition module 116 processes the list of websites and uses one or more techniques to select one or more websites from the list of websites (e.g., data acquisition module 116 may compare keywords to websites descriptions to determine if a given website is likely to contain the needed information, data acquisition module 116 may compare the list of websites to a predetermined list of websites that are known to contain useful information, etc.).

[0037] Data acquisition module 116 may process the obtained information using LLMs 112. Data acquisition module 116 may provide the obtained information to one or more of LLMs 112 as part of a prompt that includes a request for the one or more of LLMs 112 to determine whether the request for information can be answered with the obtained information. Data acquisition module 116 may obtain information of the request for information using a second LLM of LLMs 112, where a first LLM is used to generate a user interface template. In an example, data acquisition module 116 generates a prompt that includes a request to determine whether the request for information can be answered with the obtained information. Data acquisition module 116 provides the prompt to one or more of LLMs 112 and receives an output from LLMs 112 of whether the request for information can be answered using the obtained information (e.g., an output of “Yes” or “No”, an output indicating one or more deficiencies in the obtained information, etc.). Based on receiving an indication that the obtained information is insufficient, data acquisition module 116 obtains information from a different or from additional sources of information. For example, based on receiving an indication that the obtained information is sufficient to answer the request for information data acquisition module 116 may provide the obtained information to UI orchestrator 114.

[0038] UI orchestrator 114 may generate GUI 150 as including the obtained information arranged according to the UI template selected by a user. UI orchestrator 114 may generate an updated instance of GUI 150 that includes a glanceable widget, where the glanceable widget includes one or more visual elements that each replace a respective placeholder user interface element of the UI template and that include the information of the request for information. In an example, UI orchestrator 114 generates a user interface that includes a glanceable widget. UI orchestrator 114 generates the glanceable widget based on a UI template selected by a user that includes a visual element visually centered within theglanceable widget and that displays a textual indication of the departure time of a particular flight from Chicago to Richmond. For instance, UI orchestrator 114 may generate an instance of GUI 150 based on a UI template that includes two placeholder visual elements that display placeholder (e.g., randomized) times for high and low tides in Norfolk, respectively. UI orchestrator 114 may generate GUI 150 as including two visual elements that each replace the two placeholder visual elements with the actual times for high and low tides in Norfolk. UI orchestrator 114 may cause UI device 104 to output GUI 150 for display.

[0039] In some examples, UI orchestrator 114 may use one or more software components of computing device 130 to perform one or more of the processes of generating glanceable widgets in response to receiving a request for information. UI orchestrator 114 may use device large language models 144 (hereinafter “device LLMs 144”) to generate one or more UI templates, obtain information, and generate data elements and data element properties among other tasks. In an example, UI orchestrator 114 receives a request for information and generates a prompt. UI orchestrator 114 may use communication manager 118 to transmit a prompt to computing device 130 for computing device 130 to generate UI templates using device LLMs 144. For instance, UI orchestrator 114 may provide UI templates to device UI module 134 for device UI module 134 to generate a user interface based on the UI templates. UI orchestrator 114 may cause computing device 130 to generate a GUI and display the GUI via one or more display components of computing device 130. In an example, mobile computing device 102 receives user input consistent with a user requesting a GUI that includes a glanceable widget. Mobile computing device 102 generates the GUI and provides information regarding the GUI to computing device 130. Computing device 130 outputs the GUI for display via one or more display components. UI orchestrator 114 may additionally cause computing device 130 to perform one or more steps of generating a GUI in response to mobile computing device 102 receiving a request to generate a GUI and display it via mobile computing device 102.

[0040] In some examples, computing device 130 may receive user input consistent with a request to generate a GUI that includes a glanceable widget. Computing device 130 may generate a GUI that includes the glanceable widget. Responsive to the generation of the GUI, computing device 130 may display the GUI via one or more display components of computing device 130. Computing device 130 may additionally or alternatively provide data regarding the GUI to mobile computing device 102. Mobile computing device 102 may display the GUI generated by computing device 130 via UI device 104.

[0041] In some examples, UI orchestrator 114 may use remote UI generator 142 executed by remote computing system 140 to generate glanceable widgets. Remote UI generator 142 may be a program, plugin, module, process, or other type of software component configured to generate user interfaces. Remote UI generator 142 may provide the same functionality as UI orchestrator 114 and perform any one or more of the same steps as UI orchestrator 114. For example, remote UI generator 142 may use one or more LLMs to generate UI templates, obtain information, and generate GUIs that include a glanceable widget. In an example, UI orchestrator 114 causes communication manager 118 to transmit information regarding a request for information to remote computing system 140. Remote computing system 140 causes remote UI generator 142 to generate a UI template based on the request for information. Remote computing system 140 transmits information regarding the UI template to mobile computing device 102 for use by UI orchestrator 114. Additionally, remote UI generator 142 may receive information regarding a UI template selected by a user from mobile computing device 102. Remote UI generator 142 may cause one or more other software components of remote computing system 140 to obtain information and generate a GUI that includes a glanceable widget that incorporates information obtained by the one or more other software components. Remote UI generator 142 causes remote computing system 140 to transmit information regarding the GUI to mobile computing device 102 for UI device 104 to output for display to a user.

[0042] The techniques of this disclosure include one or more advantages. For example, the use of one or more of LLMs 112 may enable mobile computing device 102 to dynamically generate widgets without requiring the installation of an application that generates widgets. In another example, the use of LLMs 112 may enable mobile computing device 102 to quickly and seamlessly generate widgets that are customized to answer requests by a user rather than being limited by what widgets are provided by applications installed on mobile computing device 102. For example, mobile computing device 102 may enable a user to save a particular widget that displays changing information (e.g., information that is not necessarily fixed such as a forecast or current stock price) and easily review the changing information in the future by navigating to the particular widget within GUI 150.

[0043] FIG. 2 is a block diagram illustrating details of mobile computing device 202 for generating user interfaces, in accordance with one or more techniques of this disclosure. Mobile computing device 202 may be an example of mobile computing device 102 as illustrated in FIG. 1 and provide similar functionality. Additionally, FIG. 2 illustrates only a particular example of mobile computing device 202, and many other examples of mobilecomputing device 202 may be used in other instances and may include a subset of the components included in mobile computing device 202 or may include additional components not shown in FIG. 2.

[0044] As shown in FIG. 2, mobile computing device 202 includes one or more user interface devices 204 (“UI devices 204”), one or more processors 266 (“processors 266”), one or more communication units 268 (“communication units 268”), one or more communication channels 270 (“communication channels 270”, illustrated as “COMM. CHANNEL(S) 270” in FIG. 2), and one or more storage devices 272 (“storage devices 272”). Also, as shown in FIG. 2 UI devices 204 include one or more of input devices 262 (“input devices 262”) and one or more output devices 264 (“output devices 264”).

[0045] UI devices 204 may include one or more types of user interface devices and may be similar to UI device 104 as illustrated in FIG. 1. UI devices 204 may include devices and components capable of receiving user input such as input devices 262. Input devices 262 may include one or more devices capable of receiving user input such as tactile, audio, and video input. Input devices 262, in one example, include a presence-sensitive display, a fingerprint sensor, touch-sensitive screen, mouse, keyboard, voice responsive system, video camera, microphone or any other type of device for detecting input from a human or machine. Input devices 262 may include one or more sensors. Numerous examples of sensors exist and include any input component configured to obtain environmental information about the circumstances surrounding mobile computing device 202 and / or physiological information that defines the activity state and / or physical well-being of a user of mobile computing device 202. In some examples, a sensor may be an input component that obtains physical position, movement, and / or location information of mobile computing device 202. For instance, sensors may include one or more location sensors (e.g., GNSS components, Wi-Fi components, cellular components), one or more temperature sensors, one or more motion sensors (e.g., multi-axial accelerometers, gyros), one or more pressure sensors (e.g., barometer), one or more ambient light sensors, and one or more other sensors (e.g., microphone, camera, infrared proximity sensor, hygrometer, and the like).

[0046] UI devices 204 may include devices capable of generating output such as output devices 264. Output devices 264 of mobile computing device 202 may generate one or more outputs. Examples of outputs are tactile, audio, and video output. Output devices 264, in one example, include a presence-sensitive display, sound card, video graphics adapter card, speaker, liquid crystal display (LCD), or any other type of device for generating output to a human or machine.

[0047] Processors 266 may implement functionality and / or execute instructions within mobile computing device 202. For example, processors 266 may receive and execute instructions that provide the functionality of one or more software components of mobile computing device 202. These instructions executed by processors 266 may cause mobile computing device 202 to store and / or modify information within storage devices 272 during program execution. Processors 266 may execute instructions of software components of mobile computing device 202 to perform one or more operations. That is, the one or more software components such as operating system 260 may be operable by processors 266 to perform various functions described herein.

[0048] Communication units 268 of mobile computing device 202 may communicate with one or more external devices via one or more wired and / or wireless networks by transmitting and / or receiving network signals on the one or more networks. Examples of communication units 268 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, a satellite communication radio, or any other type of device that can send and / or receive information via one or more terrestrial and / or satellite networks. Other examples of communication units 268 may include short wave radios, cellular data radios, satellite data radios, wireless network radios, as well as universal serial bus (USB) controllers.

[0049] Communication channels 270 may interconnect one or more components of mobile computing device 202 such as components 204, 266, 268, and 272. Communication channels 270 may enable inter-components communications physically, communicatively, and / or operatively. In some examples, communication channels 270 may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data. For instance, communication channels 270 may communicate program instructions between processors 266 and storage devices 272 via a system bus.

[0050] Storage devices 272 of mobile computing device 202 may store information for processing during operation of mobile computing device 202 (e.g., mobile computing device 202 may store information and instructions of one or more software components of mobile computing device 202). In some examples, storage devices 272 may be a temporary memory, meaning that a primary purpose of storage devices 272 is not long-term storage. Storage devices 272 of mobile computing device 202 may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic randomaccess memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.

[0051] Storage devices 272 may include one or more computer-readable storage media. Storage devices 272 may be configured to store larger amounts of information than volatile memory. Storage devices 272 may include storage devices with comparatively greater storage capacity than the volatile memory of mobile computing device 202. Storage device 272 may further be configured for long-term storage of information as non-volatile memory space and retain information after power on / off cycles. Examples of non-volatile memories include magnetic hard disks, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. For example, storage devices 272 may store instructions of operating system 260 (hereinafter “OS 260”) for execution by processors 266. Storage devices 272 may be a non- transitory computer-readable storage medium encoded with instructions that cause mobile computing device 202 to perform the instructions of one or more software components such as OS 260.

[0052] OS 260 may be an operating system of mobile computing device 202. OS 260 may be one or more types of operating system such as a mobile operating system, integrated operating system, or other type of operating system. OS 260 may provide an execution environment for one or more software components of mobile computing device 202 such as user interface module 206.

[0053] Processors 266 may execute user interface module 206 (hereinafter “UI module 206”). UI module 206 may be a component of mobile computing device 202 implemented using hardware, software, firmware, or a combination therefore by mobile computing device 202. UI module 206 may be similar to UI module 106 as illustrated in FIG. 1 and provide similar functionality. For example, UI module 206 may execute as one or more services of OS 260 or computing platform or may execute as one or more executable programs at an application layer of a computing platform.

[0054] UI module 206, as shown in the example of FIG. 2, may be operable by mobile computing device 202 to perform one or more functions, such as receiving input and sending indications of such input to other components associated with mobile computing device 202. UI module 206 may also receive data from components of mobile computing device 202 such as input devices 262. UI module 206 may generate indications of the received data (e.g., input data from input devices 262) and provide indications of the input to one or more components such as output devices 264. Using the data received, UI module 206 may causeother components associated with mobile computing device 202, such as output devices 264, to provide output based on the data. For instance, UI module 206 may receive data from one or more application modules 208A-208N (hereinafter “application modules 208”) to generate a GUI, generate data of the GUI, and cause output devices 264 to output a GUI based on the generated data for display.

[0055] Processors 266 may execute one or more of application modules 208. Application modules 208 may perform operations described herein using hardware, software, firmware, or a mixture thereof residing in and / or executing at mobile computing device 202. Application modules 208 may include functionality to perform any variety of operations on mobile computing device 202. For instance, application modules 208 may include instructions that, when executed by one or more of processors 266, cause processors 266 to perform one or more operations. Application modules 208 may be similar to application modules 108 as illustrated in FIG. 1 and provide similar functionality. For example, application modules 208 may include a multimedia player that enables mobile computing device 202 to play visual media.

[0056] Mobile computing device 202 may communicate with other computing devices and / or systems. Mobile computing device 202 may communicate with other computing devices via one or more networks and communication protocols such as cellular networks, WIFI networks, satellite communication networks, BLUETOOTH communications, ultra-wideband (UWB) communications, or other type of networks and communication protocols for transmitting data between computing devices. In some examples, the network may represent one or more packet switched networks, such as the Internet. Mobile computing device 202 may send and receive data across the network using any suitable communication techniques. For example, mobile computing device 202 may be operatively coupled to one or more networks using respective network links. The network may include network hubs, network switches, network routers, terrestrial and / or satellite cellular networks, etc., that are operatively inter-coupled thereby providing for the exchange of information between mobile computing device 202 and another computing device. In some examples, network links of the network may be Ethernet, ATM or other network connections. Such connections may include wireless and / or wired connections.

[0057] Mobile computing device 202 may execute user interface orchestrator 214 (hereinafter “UI orchestrator 214”) to generate GUIs that include glanceable widgets. UI orchestrator 214 may be a module, plugin, standalone program, process, or other type of software component configured to manage the generation of GUIs that includes glanceablewidgets by mobile computing device 202. UI orchestrator 214 may orchestrate the generation of GUIs by one or more software components of mobile computing device 202. For example, UI orchestrator 214 may orchestrate the generation of a GUI as including one or more visual elements generated by OS 260 and one or more visual elements generated by an application executed by mobile computing device 202.

[0058] UI module 206 may receive user input consistent with a request for information via input devices 262. For example, UI module 206 may receive input data generated by a microphone of input devices 262 consistent with a user requesting information. UI module 206 may provide an indication of the request for information to UI orchestrator 214. For example, UI module 206 may process a user input to determine whether the user input corresponds to a request for information. UI module 206 may generate an indication based on the input data that includes text of the user input (e.g., a transcript of spoken input) to be provided to UI orchestrator 214. In some examples, UI module 206 may use one or more techniques such as natural language processing (NLP) to analyze the input data and determine whether a user has requested the generation of a GUI that includes a glanceable widget.

[0059] Mobile computing device 202 may execute one or more large language models 212A- 212N (hereinafter “LLMs 212”). LLMs 212 may include one or more large language models that are one or more types of machine learning models. LLMs 212 may include one or more large language models that are each trained for a particular purpose. For example, LLMS 212 may include a first LLM that is trained to analyze natural language requests for information and a second LLM that is trained to generate user interface layouts (e.g., user interface layouts in JSON format).

[0060] UI module 206 may provide input data to one or more of LLMs 212 for initial analysis. UI module 206 may provide the input data to an LLM trained to determine whether a request for information can be answered or whether further clarifying information is required. UI module 206 may generate a prompt for the LLM that includes input data consistent with a request for information and a request to determine whether the request for information can be answered by mobile computing device 202. UI module 206 may receive input data consistent with a user requesting information. UI module 206 may generate a prompt for an LLM of LLMs 212 that includes the input data and a request to determine whether the request for information can be answered by mobile computing device 202. UI module 206 may receive a response from the LLM indicating that further information is required before the request for information may be answered. UI module 206 may causeoutput devices 264 to output an indication (e.g., a visual indication, auditory indication, etc.) that further information is required before the request for information may be answered. LLMs 212 may output an indication of what type of clarifying information is required to UI module 206. For example, LLMs 212 may indicate that the user request for information is unclear and therefore cannot be answered by mobile computing device 202. In an example, UI module 206 provides the request for information to an intake LLM of LLMs 212. UI module 216 receives, from the intake LLM, an indication that clarifying information is needed to answer the request for information. UI module 206 generates one or more clarifying questions (e.g., questions generated based on the indication from the intake LLM). UI module 206 outputs a visual indication of the one or more clarifying questions via one or more display components such as output devices 264.

[0061] UI orchestrator 214 may generate prompts for LLMs 212 to generate UI configuration data. UI orchestrator 214 may generate prompts based on the request for information and configuration information of mobile computing device 202. For example, UI orchestrator 214 may generate a prompt that includes a summary of the request for information and configuration information such as the resolution and shape of a display of output devices 264, information regarding one or more speakers of output devices 264, and other information.

[0062] UI orchestrator 214 may generate the prompts for UI configuration data as including one or more requirements for generating the UI configuration data. UI orchestrator 214 may generate the prompt as including a statement that an LLM of LLMs 212 should treat itself as being an expert UI designer and that the LLM should generate the UI configuration data for itself. UI orchestrator 214 may additionally include one or more steps that the LLM should perform in generating the UI configuration data. For example, UI orchestrator 214 may generate the prompt as including an indication of a first step that includes a list of actions that the LLM will take to answer the request for information (e.g., “Provide an itemized list that explains how you plan to answer the query”), a second step that requires the LLM to explain what should be shown to the user (e.g., “Briefly explain what pieces of information would be useful to show to the user”), a third step that requires the LLM to organize one or more data points of the request for information into groupings of data points (e.g., “Briefly explain whether there should be any natural grouping of data points in your response”), and a fourth step that requires the LLM to provide an output in a particular format (e.g., JSON).

[0063] UI orchestrator 214 may receive an output from LLMs 212 that includes UI configuration data. LLMs 212 may process the prompt and generate text output of the UI configuration data (e.g., JSON text of the configuration data). LLMs 212 may generate UIconfiguration data that includes information regarding one or more data elements, where the data elements are particular data points of the request for information. For example, LLMs 212 may generate UI configuration data as including a first data element regarding a stock price of a first company and a second data element of a stock price of a second company. LLMs 212 may output configuration data that includes data elements as including one or more data element properties that are indicators of one or more properties of a given data element. In an example, LLMs 212 output an indication of a data element of the stock price of a particular company and a first data property that indicates the name of the company, a data element property that indicates the current price of the stock of the company in United States dollars (USD), and a data element property that indicates a percent daily change in the price of the stock. LLMs 212 may output configuration data that includes data elements where one or more of the data elements are images. For example, LLMs 212 may output a data element that includes a candlestick graph illustrating the performance of a stock over predetermined period of time.

[0064] UI orchestrator 214 may receive an output from LLMs 212 of UI configuration data that includes indications of whether a data element property should be bound or fixed. UI orchestrator 214 may receive an indication that a given data element and / or data element property should be bound and therefore periodically updated. UI orchestrator 214 may use LLMs 212 to determine whether to bind a data element of one or more data elements. Responsive to determining to bind the data element, UI orchestrator may periodically obtain updated information for the data element. In an example, UI orchestrator 214 uses LLMs 212 to determine that a data element corresponds to the stock price of a company should be bound as the stock price is likely to change and therefore require updating. UI orchestrator 214 may receive UI configuration data as including an indication to bind the particular data element property from LLMs 212 (e.g., a variable in JSON data indicating that the variable is bound). UI orchestrator 214 may use LLMs 212 to determine that a given data element property should be fixed. For example, UI orchestrator 214 may receive an indication from LLMs 212 that a given data element property should be fixed and therefore the value of the data element property will not be updated (e.g., the name of a company is unlikely to change within the timeframe of using a glanceable widget). LLMs 212 may provide an indication that a data element that corresponds to an image should be bound. For example, UI orchestrator 214 may receive an indication from LLMs 212 that a data element corresponding to the top photo of the day on a social media site should be bound and therefore updated once a day.

[0065] UI orchestrator 214 may generate one or more one or more UI templates for selection by a user. UI orchestrator 214 may generate the UI templates based on the UI configuration data and using one or more of LLMs 212. UI orchestrator 214 may generate a prompt for LLMs 212 that includes the UI configuration information and one or more requests to generate UI templates. For example, responsive to the generation of UI configuration data by a first LLM UI orchestrator 214 may generate a prompt for a second LLM of LLMs 212 that includes the configuration data, an indication to generate the UI templates (e.g., a textual indication such as “Generate three user interfaces templates using the below information”), and other information such as UI schema documentation (e.g, basic rules regarding generating user interfaces, user interface formatting information, and other information regarding generating user interfaces). UI orchestrator 214 may receive an output from the second LLM of LLMs 212 that includes the UI templates. UI orchestrator 214 may receive UI templates from LLMs 212 as free-form designs generated by LLMs 212. UI orchestrator 214 may receive the UI templates in one or more forms such as in XML format or other types of formats for UI configuration.

[0066] UI orchestrator 214 may generate one or more UI templates using one or more of user interface rules 274 (hereinafter “UI rules 274”). UI rules 274 may be a store of predetermined rules regarding UI generation and configuration maintained by mobile computing device 202. For example, UI rules 274 may include rules regarding the shape and color of visual elements, relative placement of visual elements within a user interface, and rules regarding how data elements should be displayed within a user interface among other rules. UI orchestrator 214 may process the UI configuration data using UI rules 274 to generate the UI templates. For example, UI orchestrator 214 may apply one or more of UI rules 274 to the UI configuration data and generate one or more UI templates. UI orchestrator 214 may apply one or more subsets of UI rules 274 to the UI configuration data to create different UI templates, where each of the different UI templates is generated using a different subset of UI rules 274.

[0067] UI orchestrator 214 may generate placeholder data for inclusion in the UI templates. UI orchestrator 214 may generate placeholder data to take the place of the information of the request for information when displaying the UI templates. For example, UI orchestrator 214 may generate a UI template that includes a first visual element and a second visual element with the first visual element displaying information regarding a user’s maximum speed while skiing and the second visual element displaying a user’s distance traveled while skiing. UI orchestrator 214 may generate placeholder data that includes a randomized maximum speed for the first visual element and a randomized distance for the second visual element. UIorchestrator 214 may use the UI configuration data to generate the placeholder data. UI orchestrator 214 may use information regarding data elements and data element properties to generate the placeholder data. For example, UI orchestrator 214 may use the configuration data to determine that placeholder data for a given UI template should be a randomized time of day and a randomized temperature within a predetermined range of temperature. UI orchestrator 214 may generate a prompt that includes a request for placeholder data that includes one or more requirements for the placeholder data (e.g., generate a realistic temperature value or stock price) and provide the prompt to LLMs 212. LLMs 212 may generate placeholder data based on the prompt. In some examples, UI orchestrator 214 may generate placeholder data according to one or more predetermined rules regarding placeholder data.

[0068] UI orchestrator 214 may generate user interfaces based on the UI templates that include the placeholder data for display. UI orchestrator 214 may generate a user interface based on a UI template, where the user interface includes one or more placeholder visual elements that display placeholder information. In an example, UI orchestrator 214 generates a user interface based on a UI template, where the UI template includes a first visual element that displays the stock ticker of a company, a second visual element that displays a current price of a share of the company in USD, and a third visual element that displays a weekly percent change in the share price. UI orchestrator 214 generates the user interface as including a first placeholder visual element that takes the place of the first visual element and includes a randomized company name, a second placeholder visual element that takes the place of the second visual element and includes a randomized share price in USD, and a third placeholder visual element that takes the place of the third visual element and that displays a randomized weekly percent change in share price. UI orchestrator 214 may generate UI templates using an LLM trained to generate user interface templates that is different from the LLM used to generate the UI configuration data and the LLM used to obtain the information of the request for information. In an example, UI orchestrator 214 generates UI template configuration information using a first LLM of LLMs 212, obtains information of the request for information using a second LLM of LLMs 212, and generates, based on the UI template configuration information, a user interface template using a third LLM of LLMs 212.

[0069] UI orchestrator 214 may generate one or more instances of a GUI based on the one or more UI templates. UI orchestrator 214 may generate one or more instances of a GUI, where each instance of the GUI is based on a respective UI template. For example, UI orchestrator 214 may generate three instances of a GUI, where each instance is based on a respective UItemplate with a different design from the other two UI templates. UI orchestrator 214 may generate instances of a GUI based on UI templates to enable a user of mobile computing device 202 to view and select a UI template from among the one or more UI templates that the user wishes to use. For example, UI orchestrator 214 may generate one or more instances of a GUI based on UI templates to enable a user to select a design of a glanceable widget within the GUI that the user prefers. UI orchestrator 214 may use one or more software components of mobile computing device 202, such as UI module 206, to cause output devices 264 to output the instances of the GUI for display. UI orchestrator 214 may cause output devices 264 to display a GUI that includes one or more navigation elements that enable a user to switch between the one or more instances of the GUI and to select an instance of the GUI.

[0070] UI orchestrator 214 may receive input data consistent with a user of mobile computing device 202 selecting an instance of the GUI and, by extension, a UI template. One or more of input devices 262 may receive user input consistent with a selection of an instance of the GUI. For example, input devices 262 may receive user input consistent with a selection a UI template from one or more UI templates as a selected UI template via a user interacting with a touchscreen of input devices 262. Input devices 262 may generate data based on the user input and provide the data to UI orchestrator 214. Based on the user input data, UI orchestrator 214 may determine that a user has selected an instance of the GUI and, by extension, an associated UI template.

[0071] UI orchestrator 214 may select a UI template without requiring user interact! on / selecti on. UI orchestrator 214 may generate a single UI template and use the UI template without requiring user selection of the UI template. In an example, UI orchestrator 214 generates a single UI template and uses the single UI template as the basis for a GUI without requiring user input (e.g., as there is only a single UI template) to select from. UI orchestrator 214 may generate multiple UI templates and select a UI template from the multiple UI templates without user selection of a UI template. UI orchestrator 214 may use one or more of LLMs 212 to select a UI template and / or use one or more predetermined rules to select a UI template from the one or more UI templates as a selected UI template.

[0072] Mobile computing device 202 may execute data acquisition module 216. Data acquisition module 216 may be a module, plugin, process, program, or other type of software component executed by processors 266. UI orchestrator 214 may cause data acquisition module 216 to obtain the information of the request for information. Data acquisition module 216 may obtain the information of the request for information from one or more sources of information. Data acquisition module 216 may search one or more of storage devices 272 forinformation related to the request for information. For example, data acquisition module 216 may search a calendar of a user that is stored in storage devices 272 for information regarding upcoming appointments of the user.

[0073] Data acquisition module 216 may obtain the information of the request for information from one or more sources external to mobile computing device 202. Data acquisition module 216 may obtain information from external sources such as webpages, databases, and other sources of information. Data acquisition module 216 may use one or more of LLMs 212 to obtain information for UI orchestrator 214. Data acquisition module 216 may generate a prompt for one or more of LLMs 212 that includes the request for information (e.g., the request for information from the user, a summarization of the information sought by the request for information, etc.) and one or more requirements for answering the prompt (e.g., “Determine which search engine should be used to answer this question or to find this information”, “What is the best search engine query for this input?”, “Where can I find the answer to this question on the Internet?”, etc.). Data acquisition module 216 may provide the prompt to LLMs 212. Data acquisition module 216 may receive output from LLMs 212 indicating one or more sources of information that may be used to obtain the information of the request for information and / or a textual query for a search engine that may be used to obtain the information using a search engine among other information. For example, data acquisition module 216 may receive output of a textual query for a search engine from LLMs 212 in addition to an indication of particular types of webpages within the search results from the search engine that may be used to obtain the information of the request for information.

[0074] Data acquisition module 216 may use one or more search engines to obtain the information of the request for information. Data acquisition module 216 may cause communication manager 218 to interact with a search engine and provide a textual query and obtain a list of websites from the search engine. Data acquisition module 216 may interact with a search engine and obtain a list of websites from the search engine. Data acquisition module 216 may determine which websites of the obtained websites should be used to obtain the information of the request for information. Data acquisition module 216 may use one or more techniques to determine which websites should be used such as obtain information from a predetermined number of the top search results identified by the search engine, obtaining data from predetermined types of websites (e.g., only government and major corporate websites), websites with predetermined types of domains (e.g., only “.gov” and “.org” domains) and other techniques for determining which websites should be used. Dataacquisition module 216 may process the obtained list of websites to parse information from one or more of the selected websites from the list of websites.

[0075] Data acquisition module 216 may use LLMs 212 to determine whether the obtained information can be used to answer the request for information. Data acquisition module 216 may generate a verification prompt for LLMs 212. Data acquisition module 216 may generate a verification prompt that includes the information obtained from the one or more websites and a request to determine whether the information can be used to answer the request for information. For example, data acquisition module 216 may generate a prompt that includes the selected one or more websites and a request to verify whether the information can be used to answer the request for information (e.g., “Using the given page, can you answer the user’s question?”). Data acquisition module 216 may receive the output from LLMs 212 that indicates whether the request for information can be answered using the selected one or more websites (e.g., “Yes / No” indicator). Based on the indication from LLMs 212, data acquisition module 216 may determine whether to obtain information from a different source of information. In addition, data acquisition module 216 may use LLMs 212 to determine whether there are any issues with the sources of information (e.g., whether the source is considered to be an unreliable source of information). Data acquisition module 216 may determine that an LLM of LLMs 212 has encountered one or more issues with obtaining the information of the request for information (e.g, one or more issues with the selected websites. Data acquisition module 216 may cause the LLM of LLMs 212 to obtain the information of the request for information from a different source of information. For example, data acquisition module 216 may receive a revised textual query for a search engine from LLMs 212. Data acquisition module 216 may use the revised textual query to obtain a second list of websites.

[0076] UI orchestrator 214 may use LLMs 212 to process the obtained information that answers the request for information. UI orchestrator 214 may generate a prompt for LLMs 212 that includes information obtained from the one or more selected websites, UI configuration data, and a request for LLMs 212 to answer the request for information in the form of updated UI configuration data similar to the UI configuration data of the UI templates. UI orchestrator 214 may include UI configuration data of the UI template selected by the user in the prompt to enable LLMs 212 to generate updated UI configuration data that includes the data elements and data element properties of the UI template. UI orchestrator 214 may receive updated UI configuration data from LLMs 212 in one or more formats such as JSON. In an example, UI orchestrator 214 generates a prompt for LLMs 212 that includesUI configuration data of a user-selected UI template, parsed contents of one or more webpages selected to answer the request for information, and a request to answer the request for information in JSON format according to the UI configuration data of the UI template. UI orchestrator 214 receives updated UI configuration data in JSON format that includes the information of the request for information in JSON format as output from LLMs 212.

[0077] UI orchestrator 214 generates a GUI that includes a glanceable widget based on the updated UI configuration data. Similar to the generation of the UI templates, UI orchestrator 214 may generate the GUI using one or more of LLMs 212 and / or UI rules 274. UI orchestrator 214 may generate a prompt that includes the updated UI configuration data and a request to generate a GUI that displays the data elements included in the updated UI configuration data in one or more formats (e.g., JSON). UI orchestrator 214 may receive output from LLMs 212 of the GUI in one or more formats (e.g., XML). In some examples, UI orchestrator 214 may use UI rules 274 to generate a GUI that includes the data elements.

[0078] UI orchestrator 214 may output the GUI that includes the glanceable widget for display. Based on the generation of the GUI, UI orchestrator 214 may cause output devices 264 to output the GUI for display via a display of output devices 264. In some examples, UI orchestrator 214 may provide the GUI to UI module 206 for processing and for UI module 206 to cause output devices 264 to output the GUI for display. UI module 206 may integrate one or more additional visual elements (e.g., system status indicators such as battery charge indicator and system time indicator) into the GUI alongside the glanceable widget within the GUI.

[0079] In some examples, UI orchestrator 214 may generate the GUI as including one or more visual elements that enable a user to select the source of information that they would prefer to use. UI orchestrator 214 may determine one or more sources of information (e.g., websites) that may be used to generate the GUI. UI orchestrator 214 may generate the GUI as including visual elements that indicate the one or more sources of information and that enable a user to select from the one or more sources of information. In an example, UI orchestrator 214 determines that, for a data element regarding a stock price, multiple websites may each be used as a source of information regarding the stock price. UI orchestrator 214 generates a GUI as including a visual element that displays an indication of the websites. Input devices 262 receive user input consistent with the selection of a particular website of the multiple websites. Based on the user input, UI orchestrator 214 generates an updated instance of the GUI and causes output devices 264 to output the updated instance of the GUI for display.

[0080] UI orchestrator 214 may generate updated instances of the GUI that include updates to the information of the request for information. UI orchestrator 214 may generate updated instances of the GUI based on a predetermined interval of time and / or based on one or more triggers (e.g., a data element property indicating that the GUI should be updated every thirty minutes, receipt of user input consistent with a user requesting an update, receipt of user input consistent with the user interacting with a glanceable widget within the GUI after a period of inactivity, etc.). UI orchestrator 214 may use a predetermined period of time determined by LLMs 212 (e.g., when generating the UI template LLMs 212 may assign a period of time as a data element property and / or assign an indicator of when data displayed within the GUI should be updated). For example, UI orchestrator 214 may use a predetermined period of time that is based on the type of data (e.g., tide forecasts may change only every few hours while train departures may require more frequent updates based on departure timetables). UI orchestrator 214 may cause data acquisition module 216 to re-fetch the source of information used to generate the GUI to obtain updated information for the GUI. UI orchestrator 214 may generate an updated instance of the GUI using a similar process to the one used to generate the initial instance of the GUI. For example, UI orchestrator 214 may use LLMs 212 to generate an XML layout of the GUI based on the UI configuration data and the updated information. UI orchestrator 214 may re-use the sources of information identified by data LLMs 212 in generating updated instances of the GUI to avoid the need to re-identify sources of information. For example, UI orchestrator 214 may store the uniform resource locator (URL) of a website from which information was obtained and fetch updated information from the website using the stored URL. UI orchestrator 214 may cause output devices 264 to output the updated instance of the GUI for display.

[0081] UI orchestrator 214 may use LLMs 212 to generate a script and / or regular expression (“regex”) for updating the GUI during the initial generation of the GUI. UI orchestrator 214 may generate a prompt for LLMs 212 that requests that LLMs 212 generate a script and / or regex for updating one or more data elements and / or data element properties. For example, UI orchestrator 214 may generate a prompt for LLMs 212 that includes a request for LLMs 212 to generate a script for updating one or more data element properties. UI orchestrator 214 may receive output from LLMs 212 that includes the script for updating the one or more data element properties. UI orchestrator 214 may periodically execute the script to update the information of the request for information displayed by the GUI. UI orchestrator 214 may use scripts / regexes to avoid calling one or more of LLMs 212.

[0082] UI orchestrator 214 may use communication manager 218 to facilitate the generation of GUIs that include glanceable widgets by one or more other computing devices / sy stems such as computing device 130 and remote computing systems 140 as illustrated in FIG. 1. UI orchestrator 214 may generate requests for the generation of a GUI that includes a glanceable widget. UI orchestrator 214 may cause communication manager 218 to generate and provide the request for the generation of the GUI to one or more recipient devices such as computing device 130 and remote computing system 140. For example, UI orchestrator 214 may generate a request for the generation of a GUI and cause communication manager 218 to provide the request along with the request for information to computing device 130. UI orchestrator 214 may receive information regarding a GUI that includes a glanceable widget displaying the information of the request for information from computing device 130. UI orchestrator 214 may cause output devices 264 to display the GUI that includes the glanceable widget. In some examples, UI orchestrator 214 may perform a subset of the techniques for generating a GUI that includes a glanceable widget and may cause another computing device / system to perform a different subset of the techniques for generating a GUI that includes the glanceable widget. For instance, UI orchestrator 214 may cause computing device 130 to generate UI configuration data for UI templates while UI orchestrator generates the UI templates themselves using the UI configuration data received from computing device 130.

[0083] FIG. 3 is a block diagram illustrating details of computing device 330 for generating user interfaces, in accordance with one or more techniques of this disclosure. Computing device 330 may be an example of computing device 130 as illustrated in FIG. 1 and provide similar functionality. For example, computing device 330 may be a smartphone, laptop computer, desktop computer, automotive head unit, tablet computer, wearable computing device, AR headset, VR headset, virtual machine, or virtualized computing environment among other types of computing device.

[0084] As shown in FIG. 3, computing device 330 includes one or more user interface devices 304 (“UI devices 304”), one or more processors 366 (“processors 366”), one or more communication units 368 (“communication units 368”), one or more communication channels 370 ("communication channels 370”, illustrated in FIG. 2 as “COMM. CHANNEL(S) 370”), and one or more storage devices 372 (“storage devices 372”).

[0085] UI devices 304 may include one or more types of user interface devices and may be similar to UI devices 204 as illustrated in FIG. 2. UI devices 304 may include devices and components capable of receiving user input such as input devices 362. Input devices 362 mayinclude one or more devices capable of receiving user input such as tactile, audio, and video input. Input devices 362, in one example, include a presence-sensitive display, a fingerprint sensor, touch-sensitive screen, mouse, keyboard, voice responsive system, video camera, microphone or any other type of device for detecting input from a human or machine. Input devices 362 may include one or more sensors. Numerous examples of sensors exist and include any input component configured to obtain environmental information about the circumstances surrounding computing device 330 and / or physiological information that defines the activity state and / or physical well-being of a user of computing device 330. In some examples, a sensor may be an input component that obtains physical position, movement, and / or location information of computing device 330. For instance, sensors may include one or more location sensors (e.g., GNSS components, Wi-Fi components, cellular components), one or more temperature sensors, one or more motion sensors (e.g., multi-axial accelerometers, gyros), one or more pressure sensors (e.g., barometer), one or more ambient light sensors, and one or more other sensors (e.g., microphone, camera, infrared proximity sensor, hygrometer, and the like).

[0086] UI devices 304 may include devices that generate output such as output devices 364. Output devices 364 of computing device 330 may generate one or more outputs. Examples of outputs are tactile, audio, and video output. Output devices 364, in one example, include a presence-sensitive display, sound card, video graphics adapter card, speaker, liquid crystal display (LCD), or any other type of device for generating output to a human or machine.

[0087] Processors 366 may implement functionality and / or execute instructions within computing device 330. Processors 366 may include one or more physical and / or logical processors. Processors 366 may include processors with one or more logical cores configured to execute the instructions of computing device 330. For example, processors 366 may receive and execute instructions that provide the functionality of one or more software components of computing device 330. These instructions executed by processors 366 may cause computing device 330 to store and / or modify information within storage devices 372 during program execution. Processors 366 may execute instructions of software components of computing device 330 to perform one or more operations. That is, the one or more software components such as operating system 360 may be operable by processors 366 to perform various functions described herein.

[0088] Communication units 368 of computing device 330 may communicate with one or more external devices via one or more wired and / or wireless networks by transmitting and / or receiving network signals on the one or more networks. Examples of communication units368 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, a satellite communication radio, or any other type of device that can send and / or receive information via one or more terrestrial and / or satellite networks. Other examples of communication units 368 may include short wave radios, cellular data radios, satellite data radios, wireless network radios, as well as universal serial bus (USB) controllers. Communication units 368 may enable computing device 330 to communicate with other computing devices such as mobile computing device 202 as illustrated in FIG. 2.

[0089] Communication channels 370 may interconnect one or more components of computing device 330 such as components 304, 366, 368, and 372. Communication channels 370 may enable inter-components communications physically, communicatively, and / or operatively. In some examples, communication channels 370 may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data. For instance, communication channels 370 may communicate program instructions between processors 366 and storage devices 372 via a system bus.

[0090] Storage devices 372 of computing device 330 may store information for processing during operation of computing device 330 (e.g., computing device 330 may store information and instructions of one or more software components of computing device 330). In some examples, storage devices 372 may be a temporary memory, meaning that a primary purpose of storage devices 372 is not long-term storage. Storage devices 372 of computing device 330 may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.

[0091] Storage devices 372 may include one or more computer-readable storage media. Storage devices 372 may be configured to store larger amounts of information than volatile memory. Storage devices 372 may include one or more storage devices with a comparatively higher storage capacity than the volatile memory of computing device 330. Storage device 372 may further be configured for long-term storage of information as non-volatile memory space and retain information after power on / off cycles. Examples of non-volatile memories include magnetic hard disks, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. For example, storage devices 372 may store instructions of operating system 360 (hereinafter “OS 360”) for execution by processors 366.

[0092] OS 360 may be an operating system of computing device 330. OS 360 may be one or more types of operating system such as a mobile operating system, integrated operating system, or other type of operating system. OS 360 may provide an execution environment for one or more software components of computing device 330 such as device UI module 334.

[0093] Processors 366 may execute device UI module 334. Device UI module 334 may be a component of computing device 330 implemented using hardware, software, firmware, or a combination therefore by computing device 330. Device UI module 334 may be similar to UI module 206 of mobile computing device 202 as illustrated in FIG. 2 and provide similar functionality. For example, device UI module 334 may execute as one or more services of OS 360 or computing platform or may execute as one or more executable programs at an application layer of a computing platform.

[0094] Device UI module 334, as shown in the example of FIG. 3, may perform one or more functions, such as receiving input and sending indications of such input to other components associated with computing device 330. Device UI module 334 may also receive data from components associated with computing device 330 such as input devices 362. Using the data received, device UI module 334 may cause other components associated with computing device 330, such as output devices 364, to provide output based on the data. Device UI module 334 may process user input received at another computing device and provided to computing device 330. For example, device UI module 334 may process data regarding user input received by mobile computing device 202 and provided to computing device 330.

[0095] Computing device 330 may communicate with other computing devices and / or systems. Computing device 330 may communicate with other computing devices via one or more networks and communication protocols such as cellular networks, WIFI networks, satellite communication networks, BLUETOOTH communications, ultra-wideband (UWB) communications, or other type of networks and communication protocols for transmitting data between computing devices. In some examples, the network may represent one or more packet switched networks, such as the Internet. Computing device 330 may send and receive data across the network using any suitable communication techniques. For example, computing device 330 may be operatively coupled to one or more networks using respective network links. The network may include network hubs, network switches, network routers, terrestrial and / or satellite cellular networks, etc., that are operatively inter-coupled thereby providing for the exchange of information between computing device 330 and another computing device. In some examples, network links of the network may be Ethernet, ATM or other network connections. Such connections may include wireless and / or wired connections.

[0096] Computing device 330 may execute remote UI generator 342 to generate GUIs for computing device 330 and other computing devices such as mobile computing device 202. Remote UI generator 342 may be a module, plugin, standalone program, process, or other type of software component configured to manage the generation of GUIs that includes glanceable widgets by computing device 330. Remote UI generator 342 may be similar to UI orchestrator 214 as illustrated in FIG. 2 and provide similar functionality. Remote UI generator 342 may orchestrate the generation of GUIs by one or more software components of computing device 330. For example, remote UI generator 342 may orchestrate the generation of a GUI as including one or more visual elements generated by OS 360 and one or more visual elements generated by an application executed by computing device 330 and / or an application executed by mobile computing device 202.

[0097] Remote UI generator 342 may receive a request to generate a GUI that includes a glanceable widget. Remote UI generator 342 may receive the request to generate the GUI from a user of computing device 330 via UI devices 304 and device UI module 334. Remote UI generator 342 may additionally / altematively receive requests to generate a GUI from another computing device such as mobile computing device 202. For example, remote UI generator 342 may receive a request that includes a request for information from mobile computing device 202 via communication units 368. Computing device 330 may receive user input consistent with a request to generate a GUI that includes a glanceable widget via one or more components such as input devices 362. For example, computing device 330 may receive user input data consistent with a user speaking into a microphone and requesting information. Computing device 330 may receive data regarding user input to mobile computing device 202 requesting the generation of a GUI. For example, computing device 330 may receive an indication of a request to generate a GUI from mobile computing device 202.

[0098] Computing device 330 may execute one or more device large language models 344 (hereinafter “LLMs 344”). LLMs 344 may include one or more large language models that are one or more types of machine learning models. LLMs 344 may be similar to LLMs 212 and provide similar functionality. In some examples, LLMs 344 may include comparatively more sophisticated large language models than LLMs 212.

[0099] Remote UI generator 342 may provide user input data to one or more of LLMs 344 for initial analysis. Remote UI generator 342 may generate a prompt that includes the user input data and a request to determine whether LLMs 344 can answer a request for information within the user input data. In an example, remote UI generator 342 receives dataregarding user input that includes a request for information from mobile computing device 202. Remote UI generator 342 generates a prompt based on the user input data and provides the prompt to LLMs 344. Remote UI generator 342 receives an output from LLMs 344 indicating that LLMs 344 are capable of answering the request for information.

[0100] Remote UI generator 342 may generate prompts for LLMs 344 to generate UI configuration data. Remote UI generator 342 may generate similar prompts as UI orchestrator 214. For example, remote UI generator 342 may generate a prompt that includes the request for information and one or more steps that LLMs 344 should take in generating UI configuration data. In some examples, remote UI generator 342 may receive a prompt generated by UI orchestrator 214 and provide the prompt to LLMs 344.

[0101] Remote UI generator 342 may receive an output from LLMs 344 of UI configuration data. LLMs 344 may generate UI configuration data that includes data regarding one or more data elements and one or more data element properties that are associated with respective data elements. For example, LLMs 344 may generate UI configuration data as including a data element of high tide in Boston and including a data element property of time of the high tide and a data element property of the location of Boston. LLMs 344 may generate the data element properties as indicating whether each data element property should be bound (e.g., periodically updated) or fixed (set to a fixed value and not updated). LLMs 344 may generate UI configuration data in JSON format and provide the UI configuration data to remote UI generator 342. Remote UI generator 342 may provide the UI configuration data to mobile computing device 202. In some examples, remote UI generator 342 may receive UI configuration data from a remote computing system, such as remote computing system 140 as illustrated in FIG. 1, that is generated based on a prompt provided to remote computing system 140.

[0102] Remote UI generator 342 may generate one or more UI templates. Remote UI generator 342 may generate the UI templates using LLMs 344 and / or rules regarding the generation of UI templates. Remote UI generator 342 may generate a prompt for LLMs 344 that includes the UI configuration data, an indication to generate the UI templates, and UI schema documentation. Remote UI generator 342 may provide the prompt to LLMs 344. Remote UI generator 342 may receive one or more UI templates from LLMs 344 in one or more formats such as XML. Remote UI generator 342 may generate one or more UI templates using rules such as UI rules 274 as illustrated in FIG. 2. Remote UI generator 342 may provide the one or more UI templates to a computing device such as mobile computingdevice 202 for user selection of a UI template. In some examples, remote UI generator 342 may receive one or more UI templates from remote computing system 140.

[0103] Remote UI generator 342 may generate placeholder data for inclusion in the UI templates. Remote UI generator 342 may generate placeholder data to take the place of the information of the request for information when the UI templates are displayed to a user. For example, remote UI generator 342 may generate randomized data for one or more data elements and data element properties of a UI template. Remote UI generator 342 may use LLMs 344 to generate the data for the UI templates. For example, remote UI generator 342 may generate a prompt for LLMs 344 that includes one or more requirements for generating the placeholder data (e.g., that the placeholder data must include realistic values such as actual locations, temperatures values within a particular range, etc.). Remote UI generator 342 may provide the prompt to LLMs 344 and receive placeholder data from LLMs 344. Remote UI generator 342 may provide placeholder data for the one or more UI templates to mobile computing device 202.

[0104] Remote UI generator 342 may generate one or more user interfaces based on the UI templates that include the placeholder data for display. Remote UI generator 342 may generate a user interface based on a UI template, where the user interface includes one or more placeholder visual elements that display placeholder information. For example, remote UI generator 342 may generate a user interface that includes a first visual element displaying the name of a city (e.g., London, New York, Singapore, Tokyo, etc.) and a second visual element displaying the current time in that city. Remote UI generator 342 may generate the user interface as including a first placeholder visual element that takes the place of the first visual element and replaces the name of the city with a randomized or preset city name (e.g., replacing city names with “Mountain View” or “Cupertino”) and as including a second placeholder visual element replacing the second visual element and replacing the time with a randomized time. Remote UI generator 342 may generate a respective user interface for each UI template of the one or more UI templates.

[0105] Remote UI generator 342 may cause output devices 364 to display the one or more user interfaces based on the one or more UI templates. Remote UI generator 342 may cause output devices 364 to display the user interfaces to enable a user to select a user interface, and by extension, a user interface template that they prefer. Remote UI generator 342 may simultaneously display multiple instances of the user interfaces where computing device 330 has a comparatively larger display compared to mobile computing device 202 (for instance, where computing device 330 is a tablet computer and mobile computing device 202 is asmartwatch). In an example, remote UI generator 342 generates three user interfaces that are respectively based on three different UI templates. Input devices 362 receive user input consistent with a selection of one of the three user interfaces. In some examples, remote UI generator 342 may receive an indication of a user selection of a user interface based on a UI template from another computing device such as mobile computing device 202.

[0106] Computing device 330 may execute data acquisition module 316. Data acquisition module 316 may be a module, plugin, process, program, or other type of software component executed by processors 366. Data acquisition module 316 may be similar to data acquisition module 216 as illustrated in FIG. 2 and provide similar functionality. Remote UI generator 342 may cause data acquisition module 316 to obtain the information of the request for information. Data acquisition module 316 may obtain the information of the request for information from one or more sources of information. Data acquisition module 316 may search one or more of storage devices 372 for information related to the request for information. For example, data acquisition module 316 may search a calendar of a user that is stored in storage devices 372 for information regarding upcoming appointments of the user. Data acquisition module 316 may query one or more computing systems associated with applications executed by computing device 330 for information. For example, data acquisition module 316 may query a computing system that remotely stores copies of documents used by a productivity application executed by computing device 330 for information.

[0107] Data acquisition module 316 may obtain the information of the request for information from one or more sources external to computing device 330. Data acquisition module 316 may obtain information from external sources such as webpages, databases, and other sources of information. Data acquisition module 316 may use one or more of LLMs 344 to obtain information for remote UI generator 342. Data acquisition module 316 may generate a prompt for one or more of LLMs 344 that includes the request for information (e.g., the request for information from the user, a summarization of the information sought by the request for information, etc.) and one or more requirements for answering the prompt (e.g., “Determine which search engine should be used to answer this question or to find this information”, “What is the best search engine query for this input?”, “Where can I find the answer to this question on the Internet?”, etc.). Data acquisition module 316 may provide the prompt to LLMs 344. Data acquisition module 316 may receive output from LLMs 344 indicating one or more sources of information that may be used to obtain the information of the request for information and / or a textual query for a search engine that may be used toobtain the information using a search engine among other information. For example, data acquisition module 316 may receive output of a textual query for a search engine from LLMs 344 in addition to an indication of particular types of webpages within the search results from the search engine that may be used to obtain the information of the request for information.

[0108] Data acquisition module 316 may use one or more search engines to obtain the information of the request for information. Data acquisition module 316 may cause communication manager 218 to interact with a search engine and provide a textual query and obtain a list of websites from the search engine. Data acquisition module 316 may interact with a search engine and obtain a list of websites from the search engine. Data acquisition module 316 may determine which websites of the obtained websites should be used to obtain the information of the request for information. Data acquisition module 316 may use one or more techniques to determine which websites should be used such as those used by data acquisition module 216.

[0109] Data acquisition module 316 may use LLMs 344 to determine whether the obtained information can be used to answer the request for information. Data acquisition module 316 may generate a verification prompt for LLMs 344. Data acquisition module 316 may generate a verification prompt that includes the information obtained from the one or more websites and a request to determine whether the information can be used to answer the request for information. In some examples, data acquisition module 316 may receive an indication from LLMs 344 of one or more issues encountered with the selected websites and a revised textual query for a search engine. Data acquisition module 316 may use the revised textual query to obtain a second list of websites.

[0110] Remote UI generator 342 may use LLMs 344 to process the obtained information that answers the request for information. Remote UI generator 342 may generate a prompt for LLMs 344 that includes information obtained from the one or more selected websites, UI configuration data, and a request for LLMs 344 to answer the request for information in the format of updated UI configuration data similar to the UI configuration data of the UI templates. Remote UI generator 342 may provide the prompt to LLMs 344 as input. Remote UI generator 342 may receive updated UI configuration data in JSON format as output from LLMs 344. Remote UI generator 342 may provide the updated UI configuration data to mobile computing device 202 for display by one or more components of mobile computing device 202. In some examples, remote UI generator 342 may provide the prompt to remote computing system 140 and receive the updated UI configuration data from remote computing system 140.[OHl] Remote UI generator 342 may enable a user to select one or more sources of information that may be used to answer the request for information. Remote UI generator 342 may output a GUI that includes one or more visual elements that correspond to one or more sources of information to enable a user to select from the one or more sources of information. In an example, remote UI generator 342 generates and outputs a GUI that includes visual indicators representing several sources of weather forecasting. Input devices 362 receive user input consistent with the selection of one of the several sources of information. Based on the user input, remote UI generator 342 generates updated UI configuration data using the selected source of information.

[0112] Remote UI generator 342 may generate a GUI that includes a glanceable widget using the updated UI configuration data. Remote UI generator 342 may use LLMs 344 and / or one or more predetermined rules to generate the GUI. Remote UI generator 342 may generate a prompt for LLMs 344 that includes the updated UI configuration data and a requirement for LLMs 344 to generate a GUI in accordance with UI schema documentation. Remote UI generator 342 may receive output from LLMs 344 of the GUI in one or more formats such as XML data. Remote UI generator 342 may provide data regarding the GUI to mobile computing device 202 for mobile computing device 202 to display. Mobile computing device 202 may output the received GUI for display.

[0113] Computing device 330 may output the GUI for display via one or more components. Computing device 330 may output the GUI for display via a display component of output devices 364. In an example, computing device 330 generates the GUI in response to a request received from mobile computing device 202. Computing device 330 outputs the GUI for display by one or more of display devices 364. Computing device 330 may generate the GUI in response to a request received by one or more components of computing device 330. For example, computing device 330 may generate the GUI in response to user input received by one or more of input devices 362. Computing device 330 may output the GUI via one or more of output devices 364 and / or may provide data regarding the GUI to mobile computing device 202.

[0114] FIG. 4 is a block diagram illustrating details of computing system 440 for generating user interfaces, in accordance with one or more techniques of this disclosure. Computing system 440 may be an example of remote computing system 140 as illustrated in FIG. 1 and provide similar functionality.

[0115] As shown in FIG. 4, computing system 440 includes one or more user interface devices 404 (“UI devices 404”), one or more processors 466 (“processors 466”), one or morecommunication units 468 (“communication units 468”), one or more communication channels 470 (“communication channels 470”, illustrated as “COMM. CHANNEL(S) 470” in FIG. 4), and one or more storage devices 472 (“storage devices 272”). Also, as shown in FIG. 4 UI devices 404 include one or more of input devices 462 (“input devices 462”) and one or more output devices 464 (“output devices 464”).

[0116] UI devices 404 may include one or more types of user interface devices. UI devices 404 may include devices and components capable of receiving user input such as input devices 462. Input devices 462 may include one or more devices capable of receiving user input such as tactile, audio, and video input. Input devices 462, in one example, include a presence-sensitive display, a fingerprint sensor, touch-sensitive screen, mouse, keyboard, voice responsive system, video camera, microphone or any other type of device for detecting input from a human or machine. Input devices 462 may include one or more sensors. For instance, sensors may include one or more location sensors (e.g., GNSS components, Wi-Fi components, cellular components), one or more temperature sensors, one or more motion sensors (e.g., multi-axial accelerometers, gyros), one or more pressure sensors (e.g., barometer), one or more ambient light sensors, and one or more other sensors (e.g., microphone, camera, infrared proximity sensor, hygrometer, and the like).

[0117] UI devices 404 may include devices capable of generating output such as output devices 464. Output devices 464 of computing system 440 may generate one or more outputs. Examples of outputs are tactile, audio, and video output. Output devices 464, in one example, include a presence-sensitive display, sound card, video graphics adapter card, speaker, liquid crystal display (LCD), or any other type of device for generating output to a human or machine.

[0118] Processors 466 may implement functionality and / or execute instructions within computing system 440. For example, processors 466 may receive and execute instructions that provide the functionality of one or more software components of computing system 440. These instructions executed by processors 466 may cause computing system 440 to store and / or modify information within storage devices 472 during program execution. Processors 466 may execute instructions of software components computing system 440 to perform one or more operations. That is, the one or more software components such as operating system 460 may be operable by processors 466 to perform various functions described herein.

[0119] Communication units 468 of computing system 440 may communicate with one or more external devices via one or more wired and / or wireless networks by transmitting and / or receiving network signals on the one or more networks. Examples of communication units468 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, a satellite communication radio, or any other type of device that can send and / or receive information via one or more terrestrial and / or satellite networks. Other examples of communication units 468 may include short wave radios, cellular data radios, satellite data radios, wireless network radios, as well as universal serial bus (USB) controllers.

[0120] Communication channels 470 may interconnect one or more components of computing system 440 such as components 404, 466, 468, and 472. Communication channels 470 may enable inter-components communications physically, communicatively, and / or operatively. In some examples, communication channels 470 may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data. For instance, communication channels 470 may communicate program instructions between processors 466 and storage devices 472 via a system bus.Communication channels 470 may enable communication among one or more nodes of computing system 440 in the case of computing system 440 including one or more distributed compute nodes.

[0121] Storage devices 472 of computing system 440 may store information for processing during operation of computing system 440 (e.g., computing system 440 may store information and instructions of one or more software components of computing system 440). In some examples, storage devices 472 may be a temporary memory, meaning that a primary purpose of storage devices 472 is not long-term storage. Storage devices 472 of computing system 440 may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.

[0122] Storage devices 472 may include one or more computer-readable storage media. Storage devices 472 may be configured to store larger amounts of information than volatile memory. For example, storage devices 472 may include one or more storage devices with comparatively larger storage capacity than the volatile memory of computing system 440. Storage device 472 may further be configured for long-term storage of information as nonvolatile memory space and retain information after power on / off cycles. Examples of nonvolatile memories include magnetic hard disks, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable andprogrammable (EEPROM) memories. For example, storage devices 472 may store instructions of operating system 460 (hereinafter “OS 460”) for execution by processors 466.

[0123] OS 460 may be an operating system of computing system 440. OS 460 may be one or more types of operating system such as a distributed operating system, integrated operating system, server operating system, mainframe operating system, or other type of operating system. OS 460 may provide an execution environment for one or more software components of computing system 440 such as user interface module 406.

[0124] Processors 466 may execute user interface module 406 (hereinafter “UI module 406”). UI module 406 may be a component of mobile computing device 202 implemented using hardware, software, firmware, or a combination therefore by computing system 440. UI module 406 may be similar to UI module 106 as illustrated in FIG. 1 and provide similar functionality. For example, UI module 406 may execute as one or more services of OS 460 or computing platform or may execute as one or more executable programs at an application layer of computing system 440.

[0125] UI module 406 may perform one or more functions, such as receiving input and sending indications of such input to other components associated with computing system 440. UI module 406 may also receive data from components associated with computing system 440 such as input devices 462. Using the data received, UI module 406 may cause other components associated with computing system 440, such as output devices 464, to provide output based on the data.

[0126] Computing system 440 may communicate with other computing devices and / or systems such as computing device 130 and / or mobile computing device 102 as illustrated in FIG. 1. Computing system 440 may communicate with other computing devices via one or more networks and communication protocols such as cellular networks, WIFI networks, satellite communication networks, BLUETOOTH communications, ultra-wideband (UWB) communications, or other type of networks and communication protocols for transmitting data between computing devices. In some examples, the network may represent one or more packet switched networks, such as the Internet. Computing system 440 may send and receive data across the network using any suitable communication techniques. For example, computing system 440 may be operatively coupled to one or more networks using respective network links. The network may include network hubs, network switches, network routers, terrestrial and / or satellite cellular networks, etc., that are operatively inter-coupled thereby providing for the exchange of information between computing system 440 and another computing device. In some examples, network links of the network may be Ethernet, ATMor other network connections. Such connections may include wireless and / or wired connections. In addition, computing system 440 may be communicatively coupled to one or more other computing systems. For example, computing system 440 may be located within a datacenter and communicatively coupled to other computing systems located within the datacenter via one or more networks internal to the datacenter.

[0127] Computing system 440 may execute remote UI generator 442 to generate GUIs for computing device 330 and other computing devices such as mobile computing device 102. Remote UI generator 442 may be similar to remote UI generator 342 of computing system 330 as illustrated in FIG. 3 and provide similar functionality. Remote UI generator 442 may be a module, plugin, standalone program, process, distributed process, microservice, or other type of software component configured to manage the generation of GUIs that includes glanceable widgets by computing system 440. Remote UI generator 442 may orchestrate the generation of GUIs by one or more software components of computing system 440. For example, remote UI generator 442 may orchestrate the generation of a GUI as including one or more visual elements generated by OS 460 and one or more visual elements generated by an application executed by computing device 330 and / or an application executed by mobile computing device 102.

[0128] Remote UI generator 442 may receive a request that includes a request for information from one or more computing devices such as computing device 130 and mobile computing device 102. Remote UI generator 442 may receive the request via communication units 468. Remote UI generator 442 may determine whether the request for information may be answered by computing system 440 by generating a GUI that includes a glanceable widget that displays the information of the request for information. For example, remote UI generator 442 may generate a prompt for a large language model such as one of remote large language models 444 that includes a request to determine whether the request for information can be answered by computing system 440. Remote computing system 440 may receive the output of remote large language models 444 that includes an indication of whether the request for information can be answered by remote computing system 440. Based on the indication from remote large language models 444, remote computing system 440 may either proceed with generating a GUI that includes a glanceable widget or provide an indication to computing device 130 and / or mobile computing device 102 that the request cannot be answered by remote computing system 440.

[0129] Remote UI generator 442 may receive a request to generate a GUI that includes a glanceable widget. Remote UI generator 442 may be similar to remote UI generator 342 andprovide similar functionality. Remote UI generator 442 may receive a request to generate the GUI from computing device 130 and / or mobile computing device 102. For example, computing system 440 may receive, via communication units 468, data regarding a request for information from mobile computing device 102. Computing system 440 may provide the data regarding the request for information to remote UI generator 442 for remote UI generator 442 to generate a GUI in response to the request.

[0130] Computing system 440 may execute one or more remote large language models 444 (hereinafter “LLMs 444”). LLMs 444 may include one or more large language models that are one or more types of machine learning models. LLMs 444 may include one or more large language models that are comparatively more sophisticated than LLMs 144 executed by computing device 130 and / or LLMs 112 executed by mobile computing device 102. For example, computing system 440 may be capable of executing comparatively sophisticated and expansive (e.g., in terms of a number of parameters) by nature of the storage capacity and processing power available to a computing system such as computing system 440.

[0131] Remote UI generator 442 may orchestrate the generation of GUIs that include glanceable widgets in response to requests for information from other computing devices. Remote UI generator 442, similar to remote UI generator 342, may generate prompts for LLMs 444. Remote UI generator 442 may generate a prompt that includes the request for information, configuration data of mobile computing device 102, and one or more steps that LLMs 444 should take in generating UI configuration data. Remote UI generator 442 may receive UI configuration data as output from LLMs 444. Remote UI generator 442 may receive UI configuration data that includes one or more data elements and one or more data element properties that are each associated with a respective data element. For example, remote UI generator 442 may receive UI configuration data that includes a data element and two data element properties with one of the data element properties being fixed and the other data element property being bound. Remote UI generator 442 may cause communication units 468 to provide data regarding the UI configuration data to one or more recipient devices such as computing device 130 and mobile computing device 102.

[0132] Remote UI generator 442 may generate one or more UI templates. Remote UI generator 442 may generate the one or more UI templates using LLMs 444 and / or one or more rules regarding the generation of UI templates. Remote UI generator 442 may generate a prompt for LLMs 444 that includes the UI configuration data, an indication to generate the UI templates, and UI schema documentation (e.g., data regarding configuration of the UI templates). Remote UI generator 442 may provide the prompt to LLMs 444. Remote UIgenerator 442 may receive one or more UI templates from LLMs 444 in one or more formats such as XML. Remote UI generator 442 may generate one or more UI templates using one or more rules defining the generation of UI templates. For example, remote UI generator 442 may apply one or more hardcoded rules to the UI configuration data to generate one or more UI templates. Remote UI generator 442 may provide the generated UI templates to one or more recipient devices such as computing device 130 and mobile computing device 102. Remote UI generator 442 may provide the UI templates to one or more recipient devices to enable a user of the recipient devices to select a UI template that they prefer. Computing system 440 may receive an indication from a recipient device of a UI template selected by a user. Computing system 440 may provide the indication to remote UI generator 442.

[0133] Remote UI generator 442 may generate placeholder data for inclusion with the one or more UI templates. Remote UI generator 442 may generate placeholder data that includes randomized data for data elements and data element properties of the one or more UI templates. For example, remote UI generator 442 may generate randomized data for each data element and data element property of each UI template based on the type of data element and data element property. Remote UI generator 442 may include the placeholder data with the UI templates provided to computing device 130 and mobile computing device 102.

[0134] Computing system 440 may execute data acquisition module 416. Data acquisition module 416 may be a module, plugin, process, program, or other type of software component executed by processors 466. Data acquisition module 416 may be similar to data acquisition module 116 as illustrated in FIG. 1 and provide similar functionality. Remote UI generator 442 may cause data acquisition module 416 to obtain the information of the request for information. Data acquisition module 416 may obtain the information of the request for information from one or more sources of information. Data acquisition module 416 may search one or more of storage devices 472 for information related to the request for information. Data acquisition module 416 may search system data 476 for data related to the request for information. System data 476 may include data associated with one or more applications executed by computing device 130 and mobile computing device 102. For example, data acquisition module 416 may search for data regarding a user’s calendar events maintained in system data 476.

[0135] Data acquisition module 416 may obtain information from one or more sources external to computing system 440. Data acquisition module 416 may obtain data from one or more external sources such as webpages, databases, and other sources of information. Data acquisition module 416 may use one or more of LLMs 444 to obtain information for remoteUI generator 442. Data acquisition module 416 may generate a prompt for one or more of LLMs 444 that includes the request for information (e.g., the request for information from the user, a summarization of the information sought by the request for information, etc.) and one or more requirements for answering the prompt (e.g., “Determine which search engine should be used to answer this question or to find this information”, “What is the best search engine query for this input?”, “Where can I find the answer to this question on the Internet?”, etc.). Data acquisition module 316 may provide the prompt to LLMs 444. Data acquisition module 416 may receive output from LLMs 444 indicating one or more sources of information that may be used to obtain the information of the request for information and / or a textual query for a search engine that may be used to obtain the information using a search engine among other information. For example, data acquisition module 416 may receive output of a textual query for a search engine from LLMs 444 in addition to an indication of particular types of webpages within the search results from the search engine that may be used to obtain the information of the request for information. Data acquisition module 416 may provide the textual query to a search engine. Data acquisition module 416 may obtain one or more sources of information using the search engine. Data acquisition module 416 may parse the one or more sources of information obtained using the search engine. In some examples, data acquisition module 416 may use LLMs 444 to determine whether the obtained one or more sources of information should be used or whether data acquisition module 416 should obtain information from other sources of information. Data acquisition module 416 may provide the parsed information to computing device 130 and mobile computing device 102.

[0136] Computing system 440 may receive a selection of one or more sources of information from one or more recipient devices. Computing system 440 may receive a selection of a source of information such as a particular website from computing device 130 and / or mobile computing device 102. Remote UI generator 442 may receive an indication of a selection of one or more sources of information from the sources of information identified by data acquisition module 416.

[0137] Remote UI generator 442 may generate updated UI configuration data that incorporates information from the selected one or more sources of information. Remote UI generator 442 may generate a prompt, similar to the prompt for generating the initial UI configuration data, to request that LLMs 444 generate updated UI configuration data. Remote UI generator 442 may receive, as output from LLMs 444, updated configuration data. In some examples, remote UI generator 442 may cause communication units 468 to provide the updated UI configuration data to one or more recipient devices such as computing device 130and / or mobile computing device 102 for the recipient devices to generate a GUI based on the updated UI configuration data.

[0138] Remote UI generator 442 may generate a GUI that includes a glanceable widget based on the updated UI configuration data. Similar to the generation of the UI templates, remote UI generator 442 may generate the GUI using one or more of LLMs 444 and / or rules regarding the generation of GUIs. Remote UI generator 442 may generate a prompt that includes the updated UI configuration data and a request to generate a GUI that displays the data elements included in the updated UI configuration data in one or more formats (e.g., JSON). Remote UI generator 442 may provide the prompt to LLMs 444. Remote UI generator 442 may receive output from LLMs 444 of the GUI in one or more formats (e.g., XML). Remote UI generator 442 may additionally and / or alternatively apply the rules regarding the generation of GUIs to generate the GUI that includes the glanceable widget. Remote UI generator 442 may cause communication units 468 to provide data regarding the GUI that includes the glanceable widget to one or more recipient devices such as computing device 130 and / or mobile computing device 102. The one or more recipient devices may output the received GUI for display via one or more components.

[0139] Remote UI generator 442 may generate updated instances of the GUI that include updates to the information of the request for information. Remote UI generator 442 may generate updated instances of the GUI based on a predetermined interval of time and / or based on one or more triggers (e.g., a data element property indicating that the GUI should be updated every thirty minutes, receipt of a request to update the data from one or more computing devices, etc.). Remote UI generator 442 may cause data acquisition module 416 to re-fetch the source of information used to generate the GUI to obtain updated information for the GUI. Remote UI generator 442 may generate an updated instance of the GUI using a similar process to the one used to generate the initial instance of the GUI. For example, remote UI generator 442 may use LLMs 444 to generate an XML layout of the GUI based on the UI configuration data and the updated information. Remote UI generator 442 may cause communication units 468 to provide data regarding the updated GUI to one or more recipient devices.

[0140] Remote UI generator 442 may use LLMs 444 to generate a script and / or regular expression (“regex”) for updating the GUI during the initial generation of the GUI. Remote UI generator 442 may generate a prompt for LLMs 444 that requests that LLMs 444 generate a script and / or regex for updating one or more data elements and / or data element properties. For example, remote UI generator 442 may generate a prompt for LLMs 444 that includes arequest for LLMs 444 to generate a script for updating one or more data element properties. Remote UI generator 442 may receive output from LLMs 444 that includes the script for updating the one or more data element properties. Remote UI generator 442 may periodically execute the script / regexes to update the information of the request for information displayed by the GUI. In some examples, remote UI generator 442 may cause communication units 468 to provide data regarding the script / regex to one or more recipient devices to enable the one or more recipient devices to execute the script and update the information displayed by the GUI themselves.

[0141] FIGS. 5A-5E are conceptual diagrams illustrating an example animation and user interface, in accordance with one or more techniques of this disclosure. The examples of FIGS. 5A-5E illustrate a progression of generating a GUI that includes a glanceable widget. For the purposes of clarity, FIGS. 5A-5E are discussed in the context of FIG. 1 and output by one or more elements of mobile computing device 502. Mobile computing device 502 may be similar to mobile computing device 102, provide similar functionality, and include one or more of the same components as mobile computing device 102.

[0142] Mobile computing device 502 may receive one or more indications of user input consistent with a user requesting the creation of a glanceable widget that includes information requested by a user. Mobile computing device 502 may receive the indications of user input via one or more components such as UI device 504. UI device 504 may be similar to UI device 104 and provide similar functionality. For example, mobile computing device 502 may receive spoken user input via a microphone of UI device 504 consistent with the user requesting the creation of a glanceable widget.

[0143] Mobile computing device 502, responsive to receiving input, may cause one or more components to generate a GUI that includes a glanceable widget. For example, mobile computing device 502 may cause UI orchestrator 114 to generate a GUI using LLMs 112. Mobile computing device 102 may output the GUI via one or more components such as a display of UI device 504. In some examples, mobile computing device 502 may provide a request to generate a GUI to one or more other computing devices / sy stems for the other computing devices and systems to generate the GUI.

[0144] In the example of FIG. 5A, mobile computing device 502 displays GUI 550A as including widget 552A and navigation elements 554A. FIG. 5A illustrates an initial stage of a progression of an animation regarding the generation of a GUI that includes a glanceable widget. FIG. 5 A illustrates widget 552 A as including a textual prompt (e.g., “WHAT WOULD YOU LIKE TO SEE ON THIS TILE?”) for a user to indicate information that theuser would like to have displayed via widget 552A. In some examples, mobile computing device 502 may generate an auditory indication to indicate to a user that they should provide input to mobile computing device 502.

[0145] Mobile computing device 502 may generate one or more updates to GUI 550A responsive to receiving user input. For example, mobile computing device 502 may generate an updated instance of GUI 550A that includes one or more color changes or animations to indicate that mobile computing device 502 has detected the receipt of user input. In an example, mobile computing device 502 generates GUI 550A as including a red ring around the text “WHAT WOULD YOU LIKE TO SEE ON THIS TILE?”. Mobile computing device 502 generates an updated instance of GUI 550A that changes the color of the ring to green in response to determining that a user is providing auditory input to mobile computing device 502.

[0146] Mobile computing device 502 may generate GUI 550A as including one or more navigation elements 554A. Mobile computing device 502 may generate GUI 550A as including navigation elements 554A in GUI 550A to enable a user to navigate within GUI 550A. In the example of FIG. 5A, mobile computing device 502 generates GUI 550A as including a navigation element 55 A with a left arrow and a navigation element 554B with a right arrow to enable a user to navigate between different widgets of GUI 550A such as widget 552 A.

[0147] In the example of FIG. 5B, GUI 550B illustrates text field 556B and navigation elements 554B. FIG. 5B illustrates a request for information received from a user by mobile computing device 502 within text field 556B. Mobile computing device 502 may display a textual representation of the request for information as text field 556B.

[0148] Mobile computing device 502 may generate GUI 550B as including navigation elements 554B to facilitate user interaction with GUI 550B. Mobile computing device 502 may generate navigation elements 554B as including a visual element that corresponds to returning to a previous instance of GUI 550B (e.g., the visual element in the shape of a return arrow illustrated in FIG. 5B), a visual element that corresponds to requesting that mobile computing device 502 process auditory input (e.g., the visual element shaped like a microphone within a circle illustrated in FIG. 5B), and a visual element that corresponds to requesting that a keyboard be displayed within GUI 550B (e.g., the visual element shaped like a keyboard illustrated in FIG. 5B).

[0149] Mobile computing device 502 may display text field 556B to enable a user to confirm whether the request for information within text field 556B is correct. Mobile computingdevice 502 may display text field 556B to enable a user to make one or more changes to the request for information and to confirm that the request for information displayed by text field 556B is correct. In the example of FIG. 5B, mobile computing device 502 causes UI device 504 display request for information 506B as including the text “NEXT TRAIN DEPARTING FROM GREENWICH STATION”. Mobile computing device 502 may enable a user to make one or more changes to text field 556B. In an example, mobile computing device 502 enables a user to change “Greenwich Station” to “Westminster Underground Station” in text field 556B. Responsive to the change by the user, mobile computing device 502 generates an updated instance of GUI 550B that includes text field 556B updated to display the text of “NEXT TRAIN DEPARTING FROM WESTMINSTER UNDERGROUND STATION”. Mobile computing device 502 may enable a user to change text field 556B through spoken and / or textual input. For example, in response to user interaction consistent with the selection of a visual element corresponding to providing auditory input (e.g., the navigation element shaped like a microphone of navigation element 554B), mobile computing device 502 may receive auditory user input. Mobile computing device 502 may determine whether text field 556B should be updated based on the auditory user input.

[0150] In the example of FIG. 5C, mobile computing device 502 displays GUI 550C as including text field 556C. In some examples, mobile computing device 502 may display one or more visual indicators in GUI 550C that indicate that mobile computing device 502 is performing one or more processes in generating a GUI that includes a glanceable widget.

[0151] Mobile computing device 502 may cause UI orchestrator 114 to verify a request for information. UI orchestrator 114 may verify whether a request for information may be answered by mobile computing device 502 or whether further clarifying information is required. For example, UI orchestrator 114 may determine that a request for information is nonsensical and therefore cannot be answered by mobile computing device 502. UI orchestrator 114 may provide a prompt to LLMs 112 and receive an indication from LLMs 112 as to whether the request for information may be answered by mobile computing device 502.

[0152] Mobile computing device 502 may generate GUI 550C as including text field 556C to indicate to a user that the generation of a GUI that includes a glanceable widget is currently in progress. In the example of FIG. 5C, mobile computing device 502 generates GUI as including text field 556C, with text field 556C including the text “CHECKING YOUR QUERY. . .” to indicate to a user that mobile computing device 502 is verifying whether the request for information may be answered by mobile computing device 502. In someexamples, mobile computing device 502 may generate text field 556C as including other textual indications as to the progress of generating a GUI that includes a glanceable widget and as indicating what step of the process that mobile computing device 502 is currently conducting (e.g., obtaining information for the request for information, generating the GUI, verifying a request for information, etc.).

[0153] In the example of FIG. 5D, mobile computing device 502 displays GUI 550D as including text field 556D. In some examples, mobile computing device 502 may display GUI 550D as including one or more status updates on the generation of the GUI that includes a glanceable widget.

[0154] Mobile computing device 502 may generate GUI 550D as including text field 556D, where text field 556D includes text indicating the progress of mobile computing device 502 in generating the GUI that includes a glanceable widget. In the example of FIG. 5D, text field 556D includes the text “BUILDING UI. . Mobile computing device 502 may generate text such as the text in text field 556D to indicate to a user of mobile computing device 502 that the GUI that includes the glanceable widget is being generated.

[0155] Mobile computing device 502 may use UI orchestrator 114 to generate a GUI that includes the glanceable widget. UI orchestrator 114 may use one or more LLMs of LLMs 112 to generate the GUI. For example, UI orchestrator 114 may generate UI configuration data using a first LLM and generate one or more UI templates using a second LLM. UI orchestrator 114 may provide an indication to UI device 504 to generate a GUI that includes text field 556D to indicate to a user that UI orchestrator 114 is generating the GUI.

[0156] In some examples, mobile computing device 502 may generate instances of GUI 550D based on UI templates generated by UI orchestrator 114. Mobile computing device 502 may generate one or more instances of GUI 550D based on the UI templates and output the instances for display to enable a user to select a design (e.g., UI template) that they prefer. Based on user selection of a UI template, mobile computing device 502 may cause one or more components such as data acquisition module 116 to obtain information to be displayed via a glanceable widget.

[0157] In some examples, mobile computing device 502 may output instances of GUI 550D that include placeholder information displayed by one or more instances of GUI 550D. Mobile computing device 502 may cause one or more components such as UI orchestrator 114 to generate placeholder data. UI orchestrator 114 may generate the one or more instances of GUI 550D that are based on the UI templates as including the placeholder information. UI orchestrator 114 may cause mobile computing device 502 to output the instance of GUI 550Dthat includes the placeholder information to aid a user in visualizing how the GUI that includes a glanceable widget may look.

[0158] In the example of FIG. 5E, mobile computing device 502 displays GUI 550E as including text fields 556E. As illustrated with respect to FIG. 5D, mobile computing device 502 may display GUI 550E as including one or more indicators as to the status of generating a GUI that includes a glanceable widget.

[0159] Mobile computing device 502 may generate GUI 550E as including text field 556E, where text field 556E includes a textual indicator that mobile computing device 502 is obtaining information to answer the request for information. In the example of FIG. 5E, text field 556E includes the text “FINDING A SOURCE. . .” to indicate to a user that mobile computing device 502 is in the processing of obtaining information of the request for information.

[0160] Mobile computing device 502 may obtain information from one or more sources of information for the request for information. Mobile computing device 502 may use one or more components such as data acquisition module 116 to obtain information from sources of information that are external and / or internal to mobile computing device 502. For example, data acquisition module 116 may use LLMs 112 to generate a textual query for a search engine. Data acquisition module 116 may use the results from the search engine to obtain information for the request for information.

[0161] In some examples, mobile computing device 502 may generate an instance of GUI 550E that includes visual indicators of one or more sources of information used by data acquisition module 116. Mobile computing device 502 may generate an instance of GUI 550E that includes a visual indicator of each source of information and one or more interactable visual elements that enable a user to indicate which source of information, if any, that they prefer. In an example, mobile computing device 502 generates an instance of GUI 550E that includes three visual indicators that each indicate a respective source of information and one or more visual indicators that enable a user to select one of the three sources of information. Mobile computing device 502 receives user input consistent with a selection of one of the sources of information and provides an indication to data acquisition module 116. Mobile computing device 502 uses the selected source of information to generate the GUI that includes the glanceable widget.

[0162] In the example of FIG. 5F, mobile computing device 502 displays GUI 550F as including text fields 556F and navigation elements 554F. The example of FIG. 5F illustrates an example of a GUI that includes a glanceable widget (e.g., text fields 556F).

[0163] Mobile computing device 502 may generate GUI 550F as including a glanceable widget displaying the information of the request for information. In the example of FIG. 5F, mobile computing device 502 generates text fields 556F as including the text “GREENWICH”, “19:27”, “RAINHAM (KENT)”, and “TRAINWEBSITE.COM” to respectively illustrate the origination train station (e.g., Greenwich station), the time of the next departing train (e.g., 19:27), the destination of the next departing train (e.g., Rainham (Kent) station), and the source of information used (e.g., a website).

[0164] Mobile computing device 502 may save the configuration of GUI 550F to enable a user to view the information of the request for information without requiring generating a new GUI that includes the glanceable widget displaying the information of the request for information. For example, mobile computing device 502 may store data regarding the configuration of GUI 550F and enable a user to request that mobile computing device 502 display GUI 550F and update the information displayed by the glanceable widget of GUI 550F (e.g., text fields 556F).

[0165] Mobile computing device 502 may store information regarding a source of information for the request for information. Mobile computing device 502 may store information (e.g., a URL) to enable mobile computing device 502 to obtain updates to the information displayed by GUI 550F without requiring the generation of new prompt and the use of LLMs 112 to identify a source of information. For example, mobile computing device 502 may store the URL of “trainwebsite.com” to enable mobile computing device 502 to obtain updated information using the URL. Mobile computing device 502 may use the URL and one or more of LLMs 112 to retrieve updated information from the website. Mobile computing device 502 may update GUI 550F based on the updated information.

[0166] Mobile computing device 502 may enable a user to perform one or more actions regarding GUI 550F by interacting with navigation elements 554F. In the example of FIG. 5F, GUI 550F includes a first navigation element of navigation elements 554F with a gear and the text “CHANGE” displayed with the first navigation element and a second navigation element of navigation elements 554F with an arrow and the text “PHONE” displayed within the second navigation element. Mobile computing device 502 may enable a user to change one or more settings of GUI 550F via interacting with the first navigation element such as changing the visual orientation of one or more of text fields 556F, changing which text fields of text fields 556F are displayed, changing the source of information used to populate text fields 556F, and other settings of GUI 550F. Mobile computing device 502 may enable a user to send the glanceable widget of GUI 550F to one or more other computing devices such ascomputing device 130 using the second navigation element of navigation elements 554F. For example, in response to user interaction with the second navigation element mobile computing device 502 may provide data regarding GUI 550F to computing device 130 for computing device 130 to display text fields 556F (e.g., the glanceable widget) via one or more output devices of computing device 130.

[0167] FIG. 6 is a flowchart illustrating an example operation for generating a user interface, in accordance with one or more techniques of this disclosure. For the purposes of clarity, FIG. 6 is discussed in the context of FIG. 1.

[0168] A computing device, such as mobile computing device 102, receives a request for information (602). Mobile computing device 102 may receive the request for information as user input such as voice and / or typed user input. Mobile computing device 102 may receive a request for information such as a comparison of two stock prices over time, the predicted high and low tides for a particular harbor, and other types of information.

[0169] Mobile computing device 102 generates, based on the request for information, a prompt, where the prompt includes configuration information of mobile computing device 102 (604). Mobile computing device 102 may generate the prompt as including other information and requirements such as a requirement to follow a series of steps to generate a user interface, a requirement to output the response to the prompt in a particular format such as JSON, a requirement to identify natural groupings of elements in the response to the request for information, among information and requirements. For example, mobile computing device 102 may generate the prompt as including a requirement to identify one or more data elements of the request for information and one or more data element properties that are each associated with a respective data element.

[0170] Mobile computing device 102 generates, using the prompt as input to a large language model such as a large language model of LLMs 112, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information (606). Mobile computing device 102 may provide the prompt to a first LLM of LLMs 112 to generate UI configuration data. Mobile computing device 102 may provide the generated UI configuration data to a second LLM of LLMs 112 to generate the one or more user interface templates in one or more formats such as XML. Mobile computing device 102 may generate a second prompt that includes the UI configuration data, UI schema documentation, and a requirement to design a UI to the second LLM for the second LLM to generate one or more user interface templates. Mobile computing device 102 may generate placeholder data (e.g., randomizeddata). Mobile computing device 102 may combine the placeholder data with the user interface templates to fill in placeholder visual elements with placeholder data for use in displaying how a user interface may look with actual data.

[0171] Mobile computing device 102 receives user input selecting a user interface template from among the one or more user interface templates as a selected user interface template (608). Mobile computing device 102 may output the selected user interface template for display via one or more components of mobile computing device 102 such as user interface device 104. Mobile computing device 102 may generate a GUI that includes a user interface template and one or more visual elements that enable a user to navigate between one or more user interface templates and to indicate which user interface template that they prefer.

[0172] Mobile computing device 102 obtains, from a data source, the information of the request for information (610). Mobile computing device 102 may use LLMs 112 to generate a textual query for a search engine to obtain the information of the request for information. In an example, mobile computing device 102 generates a prompt and provides the prompt to LLMs 112. Mobile computing device 102 receives an output from LLMs 112 of a textual query for a search engine. Mobile computing device 102 uses the textual query to search for websites as sources of information. Mobile computing device 102 parses the websites to obtain the information of the request for information.

[0173] Mobile computing device 102 generates, using the information obtained from the data source and the selected user interface template, a user interface, where the user interface includes one or more visual elements that each replace a respective user interface element of the one or more placeholder user interface elements (612). Mobile computing device 102 may use one or more of LLMs 112 to generate the user interface. For example, mobile computing device 102 may generate a prompt for LLMs 112 that includes the information of the request for information, configuration information of the selected user interface template, and other information. Mobile computing device 102 may receive information regarding the user interface from LLMs 112 in one or more formats such as XML.

[0174] Mobile computing device 102 outputs, via one or more display components such as user interface device 104, the user interface (614). Mobile computing device 102 may output the user interface as a glanceable widget that includes the information of request for information. In some examples, mobile computing device 102 may periodically update the information displayed within the user interface.

[0175] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionsmay be stored on or transmitted over, as one or more instructions or code, a computer- readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media, which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that may be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0176] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0177] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may beprovided within dedicated hardware and / or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0178] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0179] Example 1 : A method includes receiving, by a computing device, a request for information; generating, by the computing device and based on the request for information, a prompt, wherein the prompt includes configuration information of the computing device; generating, by the computing device and using the prompt as input to a large language model, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information; obtaining, by the computing device and from a data source, the information of the request for information; generating, by the computing device and using the information obtained from the data source and a selected user interface template from the at least one user interface template, a user interface, wherein the user interface includes one or more visual elements that each replace a respective placeholder user interface element of the one or more placeholder user interface elements; and outputting, by the computing device and via one or more display components, the user interface.

[0180] Example 2: The method of example 1, wherein generating the at least one user interface template further comprises: receiving, from the large language model, data regarding the at least one user interface template, wherein the data includes one or more data elements.

[0181] Example 3: The method of example 2, wherein each of the one or more data elements includes one or more data element properties that are indicators of one or more properties of a respective data element of the one or more data elements.

[0182] Example 4: The method of example 3, further includes determining, by the computing device and using the large language model and based on the one or more data element properties, whether to bind a data element of the one or more data elements; and responsiveto determining to bind the data element, periodically obtaining, by the computing device, updated information for the data element.

[0183] Example 5: The method of any of examples 1-4, wherein the large language model is first large language model, wherein generating the at least one user interface template includes generating the at least one user interface template using the first large language model, and wherein obtaining the information of the request for information includes obtaining the information of the request for information using a second large language model.

[0184] Example 6: The method of example 5, wherein generating the at least one user interface template further comprises: generating user interface template configuration information using the first large language model; and generating, based on the user interface template configuration information, the at least one user interface template using a third large language model.

[0185] Example 7: The method of any of examples 1-6, wherein obtaining the information of the request for information includes obtaining the information of the request for information from one or more sources of information external to the computing device.

[0186] Example 8: The method of any of examples 1-7, further includes providing, by the computing device, the request for information to an intake large language model; receiving, by the computing device and from the intake large language model, an indication that clarifying information is needed to answer the request for information; generating, by the computing device, one or more clarifying questions; and outputting, by the computing device and via the one or more display components, a visual indication of the one or more clarifying questions.

[0187] Example 9: The method of any of examples 1-8, further includes determining, by the computing device, that the large language model has encountered one or more issues with obtaining the information of the request for information; and causing, by the computing device, the large language model to obtain the information of the request for information from a different source of information.

[0188] Example 10: The method of any of examples 1-9, further comprising receiving, by the computing device, user input selecting a user interface template from the at least one user interface template as the selected user interface template.

[0189] Example 11 : A computing device includes one or more display components, a memory, and one or more programmable processors in communication with the memory and configured to: receive a request for information; generate, based on the request for information, a prompt, wherein the prompt includes configuration information of thecomputing device; generate, using the prompt as input to a large language model, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information; obtain, from a data source, the information of the request for information; generate, using the information obtain from the data source and a selected user interface template from the at least one user interface template, a user interface, wherein the user interface includes one or more visual elements that each replace a respective placeholder user interface element of the one or more placeholder user interface elements; and output, via the one or more display components, the user interface.

[0190] Example 12: The computing device of example 11, wherein to generate the at least one user interface template, the one or more programmable processors are further configured to: receive, from the large language model, data regarding the at least one user interface template, wherein the data includes one or more data elements.

[0191] Example 13: The computing device of example 12, wherein each of the one or more data elements includes one or more data element properties that are indicators of one or more properties of a respective data element of the one or more data elements.

[0192] Example 14: The computing device of example 13, wherein the one or more programmable processors are further configured to: determine, using the large language model and based on the one or more data element properties, whether to bind a data element of the one or more data elements; and responsive to determining to bind the data element, periodically obtain updated information for the data element.

[0193] Example 15: The computing device of any of examples 11-14, wherein the large language model is first large language model, wherein generate the at least one user interface includes generate the at least one user interface template using the first large language model, and wherein obtain the information of the request includes obtain the information of the request for information using a second large language model.

[0194] Example 16: The computing device of example 15, wherein to generate the at least one user interface template the one or more programmable processors are further configured to: generate user interface template configuration information using the first large language model; and generate, based on the user interface template configuration information, the at least one user interface template using a third large language model.

[0195] Example 17: The computing device of any of examples 11-16, wherein obtain the information of the request for information includes obtain the information of the request for information from one or more sources of information external to the computing device.

[0196] Example 18: The computing device of any of examples 11-17, wherein the one or more programmable processors are further configured to: provide the request for information to an intake large language model; receive, from the intake large language model, an indication that clarifying information is needed to answer the request for information; generate one or more clarifying questions; and output, via the one or more display components, a visual indication of the one or more clarifying questions.

[0197] Example 19: The computing device of any of examples 11-18, wherein the one or more programmable processors are further configured to: determine that the large language model has encountered one or more issues with obtaining the information of the request for information, and cause the large language model to obtain the information of the request for information from a different source of information.

[0198] Example 20: The computing device of any of examples 11-19, wherein the one or more programmable processors are further configured to receive user input selecting a user interface template from the at least one user interface template as the selected user interface template.

[0199] Example 21 : A non-transitory computer-readable storage medium encoded with instructions that, when executed, causes at least one processor of a computing device to perform any combination of the methods of examples 1-10.

[0200] Example 22: A computing system comprising means for perform any combination of the methods of examples 1-10.

[0201] Various examples of the disclosure have been described. Any combination of the described systems, operations, or functions is contemplated. These and other examples are within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method, comprising: receiving, by a computing device, a request for information; generating, by the computing device and based on the request for information, a prompt, wherein the prompt includes configuration information of the computing device; generating, by the computing device and using the prompt as input to a large language model, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information; obtaining, by the computing device and from a data source, the information of the request for information; generating, by the computing device and using the information obtained from the data source and a selected user interface template from the at least one user interface template, a user interface, wherein the user interface includes one or more visual elements that each replace a respective placeholder user interface element of the one or more placeholder user interface elements; and outputting, by the computing device and via one or more display components, the user interface.

2. The method of claim 1, wherein generating the at least one user interface template further comprises: receiving, from the large language model, data regarding the at least one user interface template, wherein the data includes one or more data elements.

3. The method of claim 2, wherein each of the one or more data elements includes one or more data element properties that are indicators of one or more properties of a respective data element of the one or more data elements.

4. The method of claim 3, further comprising: determining, by the computing device and using the large language model and based on the one or more data element properties, whether to bind a data element of the one or more data elements is bound; and responsive to determining to bind the data element, periodically obtaining, by the computing device, updated information for the data element.

5. The method of any of claims 1-4, wherein the large language model is first large language model, wherein generating the at least one user interface template includes generating the at least one user interface template using the first large language model, and wherein obtaining the information of the request for information includes obtaining the information of the request for information using a second large language model.

6. The method of claim 5, wherein generating the at least one user interface template further comprises: generating user interface template configuration information using the first large language model; and generating, based on the user interface template configuration information, the user interface template using a third large language model.

7. The method of any of claims 1-6, wherein obtaining the information of the request for information includes obtaining the information of the request for information from one or more sources of information external to the computing device.

8. The method of any of claims 1-7, further comprising: providing, by the computing device, the request for information to an intake large language model; receiving, by the computing device and from the intake large language model, an indication that clarifying information is needed to answer the request for information; generating, by the computing device, one or more clarifying questions; outputting by the computing device and via the one or more display components, a visual indication of the one or more clarifying questions.

9. The method of any of claims 1-8, further comprising: determining, by the computing device, that the large language model has encountered one or more issues with obtaining the information of the request for information; and causing, by the computing device, the large language model to obtain the information of the request for information from a different source of information.

10. The method of any of claims 1-9, further comprising: receiving, by the computing device, user input selecting a user interface template from the at least one user interface template as the selected user interface template.

11. A computing device comprising: one or more display components, a memory, and one or more programmable processors in communication with the memory and configured to: receive a request for information; generate, based on the request for information, a prompt, wherein the prompt includes configuration information of the computing device; generate, using the prompt as input to a large language model, at least one user interface template that includes one or more placeholder user interface elements that correspond to placeholder information representing information of the request for information; obtain, from a data source, the information of the request for information; generate, using the information obtain from the data source and a selected user interface template from the at least one user interface template, a user interface, wherein the user interface includes one or more visual elements that each replace a respective placeholder user interface element of the one or more placeholder user interface elements; and output, via the one or more display components, the user interface.

12. The computing device of claim 11, wherein to generate the at least one user interface template, the one or more programmable processors are further configured to: receive, from the large language model, data regarding the at least one user interface template, wherein the data includes one or more data elements.

13. The computing device of claim 12, wherein each of the one or more data elements includes one or more data element properties that are indicators of one or more properties of a respective data element of the one or more data elements.

14. The computing device of claim 13, wherein the one or more programmable processors are further configured to: determine, using the large language model and based on the one or more data element properties, whether to bind a data element of the one or more data elements; and responsive to determining to bind the data element, periodically obtain updated information for the data element.

15. A non-transitory computer-readable storage medium encoded with instructions that, when executed, causes at least one processor of a computing device to perform any combination of the method of claims 1-10.

Images