Systems and methods for a personalized intelligent assistant for autofill based on proactive and reactive data retrieval and for proactive and reactive personalized suggestive multimodal prompts

Abstract

An example method for proactive information retrieval includes detecting a user interaction with an editable interface at a computing device. The method includes determining, based on the user interaction, a data item expected to be entered in the editable interface. The method includes generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine a suggested entry associated with the data item from content accessible by the computing device. The method includes providing the generated prompt to the LLM to determine the suggested entry associated with the data item. The method additionally includes receiving, from the LLM, the suggested entry. The method further includes providing the suggested entry at the editable interface.

Description

SYSTEMS AND METHODS FOR A PERSONALIZED INTELLIGENT ASSISTANT FOR AUTOFILL BASED ON PROACTIVE AND REACTIVE DATA RETRIEVAL AND FOR PROACTIVE AND REACTIVE PERSONALIZED SUGGESTIVE MULTIMODAL PROMPTSCROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATION BY REFERENCE

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 643,639, filed on May 7, 2024, and to U.S. Provisional Patent Application No. 63 / 739,752, filed on December 30, 2024, the contents of which are hereby incorporated by reference in their entirety.BACKGROUND

[0002] Large language models take input prompts to generate outputs. Retrieval tasks, such as question answering, are based on natural language processing and can involve retrieving knowledge from a large collection of documents.SUMMARY

[0003] This application generally relates to generating suggested prompts that may be input to a large language model (LLM) and / or a Large Multimodal Model (LMM). Many users who trigger conversational artificial intelligence (Al) agents find it challenging to generate a prompt that will enable the Al agent to output a response tailored to the user’s needs. This can cause the user to spend time reviewing content that may not be relevant to what they need. It can also cause the user to spend time and computational resources in generating new, additional prompts to produce additional, better responses which consume even more time and computational resources. Such actions can cause unnecessary utilization of limited compute resources, including memory, power, and / or network bandwidth, which is problematic generally but even more so when performed on a mobile device. Also, for example, performing prompt engineering and prompt suggestion activities on the device enhances security while protecting user information by restricting it to the device.

[0004] This application also generally relates to generating suggestions for possible data items to be entered in an editable interface. For example, many users find it challenging to quickly and efficiently find relevant information needed as input to fillable forms. More generally, a user may be interacting with any interface, such as a chatbot interface, a short messaging service (SMS) interface, etc. where certain information may need to be provided as part of a user interaction.

[0005] Although existing prompt interfaces allow a user to enter a prompt or to select a suggested prompt, there is a lack of a multimodal editing interface for the user to edit a suggested prompt (e.g., in a multimodal manner), select from suggested options to complete a partial prompt (e.g., in a multimodal manner), and / or to enter a prompt and review suggested prompts (e.g., in a multimodal manner). In the realm of web browsers, some approaches suggest prompts based on a user’s web history. However, such prompts are not based on context information from application programs on a device, context information based on the device itself, and / or content or context from across various tabs of the browser. The techniques described herein also enable a prompt suggestion system to learn from user interactions with the prompt interface to present suggested prompts personalized (e.g., progressively over time) for an individual.

[0006] Also, existing approaches to autofill features are limited to web-based forms and information such as credit card numbers, address, telephone number, name, etc. This functionality does not extend to the extracting data from emails, chats, and other OS apps. Also, the autofill feature itself does not extend to other app interfaces. The approach described herein expands the autofill feature to extract specific user data (e.g, TSA number, driver’s license, travel related reservations, invoice numbers, order numbers, mileage number, etc.) from within various apps (e.g, GMAIL®, SMS). Thus, the auto-fill feature for existing web-based forms can be improved with the additional extracted data. The approach also expands the autofill feature from web-based forms to other forms (e.g., within apps), or in chat interfaces (e.g., a customer service chat box). The functionality is to be performed by an Al agent (on the platform, at a web browser) and can therefore work across apps.

[0007] The Al agent can be trained on user interactions and behavior based on interactions across the web and apps, instead of web-forms alone. The predictive aspects of LLM can be used to identify and provide customized data relevant to a particular context. The technology described herein is directed to extracting specific user data (e.g., complicated alphanumeric data) and enabling efficient, customized, and accurate autofill across web and apps.

[0008] Accordingly, there is a need for a machine learning model that can generate prompts that can capture a user’s intent, thereby causing an LLM to provide outputs that are relevant to the user. The techniques described herein can analyze user data and context from an operating system (OS) and automatically formulate a set of prompts that meets the user's needs. Also, for example, there is a need for a machine learning model that can analyze an editable interface and predict what information a user may be expected to enter in the editable interface. Such a machine learning model may generate appropriate prompts that can cause an LLM to determinethe information from content accessible by the computing device. The techniques described herein can analyze user interaction and context from an operating system (OS) and automatically formulate a set of prompts that can enable the LLM to determine the information and provide the information for use at the editable interface.

[0009] In one aspect, a computer-implemented method for proactive information retrieval is provided. The method includes detecting a user interaction with an editable interface at a computing device. The method also includes determining, based on the user interaction, a data item expected to be entered in the editable interface. The method further includes generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggested entry associated with the data item from content accessible by the computing device. The method also includes providing the generated prompt to the LLM to determine the at least one suggested entry associated with the data item. The method additionally includes receiving, from the LLM, the at least one suggested entry. The method further includes providing the at least one suggested entry at the editable interface.

[0010] In a second aspect, a device for proactive information retrieval is provided. The device includes one or more processors and data storage. The data storage has stored thereon computer-executable instructions that, when executed by one or more processors, cause the device to carry out functions. The functions include: detecting a user interaction with an editable interface at a computing device; determining, based on the user interaction, a data item expected to be entered in the editable interface; generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggested entry associated with the data item from content accessible by the computing device; providing the generated prompt to the LLM to determine the at least one suggested entry associated with the data item; receiving, from the LLM, the at least one suggested entry; and providing the at least one suggested entry at the editable interface.

[0011] In a third aspect, a computer program for proactive information retrieval is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. The functions include: detecting a user interaction with an editable interface at a computing device; determining, based on the user interaction, a data item expected to be entered in the editable interface; generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggestedentry associated with the data item from content accessible by the computing device; providing the generated prompt to the LLM to determine the at least one suggested entry associated with the data item; receiving, from the LLM, the at least one suggested entry; and providing the at least one suggested entry at the editable interface.

[0012] In a fourth aspect, an article of manufacture for proactive information retrieval is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a device, cause the device to carry out functions. The functions include: detecting a user interaction with an editable interface at a computing device; determining, based on the user interaction, a data item expected to be entered in the editable interface; generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggested entry associated with the data item from content accessible by the computing device; providing the generated prompt to the LLM to determine the at least one suggested entry associated with the data item; receiving, from the LLM, the at least one suggested entry; and providing the at least one suggested entry at the editable interface.

[0013] In a fifth aspect, a system for proactive information retrieval is provided. The system includes means for detecting a user interaction with an editable interface at a computing device; means for determining, based on the user interaction, a data item expected to be entered in the editable interface; means for generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggested entry associated with the data item from content accessible by the computing device; means for providing the generated prompt to the LLM to determine the at least one suggested entry associated with the data item; means for receiving, from the LLM, the at least one suggested entry; and means for providing the at least one suggested entry at the editable interface.

[0014] In a sixth aspect, a computer-implemented method that exhibits improved computational efficiency is provided. The method includes receiving, by an operating system of a device, application data from a plurality of application programs running on the device. The method also includes applying, by the operating system and based on the application data, a trained machine learning model to generate one or more suggested prompts for a large language model (LLM), wherein the one or more suggested prompts, upon being provided as input to the LLM, cause the LLM to generate an output relevant to an expected user activity.The method further includes providing, by the operating system, at least one of the one or more suggested prompts to be presented by a user interface associated with the operating system.

[0015] In a seventh aspect, a device is provided. The device includes one or more processors and data storage. The data storage has stored thereon computer-executable instructions that, when executed by one or more processors, cause the device to carry out functions. The functions include: receiving, by an operating system of a device, application data from a plurality of application programs running on the device; applying, by the operating system and based on the application data, a trained machine learning model to generate one or more suggested prompts for a large language model (LLM), wherein the one or more suggested prompts, upon being provided as input to the LLM, cause the LLM to generate an output relevant to an expected user activity; and providing, by the operating system, at least one of the one or more suggested prompts to be presented by a user interface associated with the operating system.

[0016] In an eighth aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. The functions include: receiving, by an operating system of a device, application data from a plurality of application programs running on the device; applying, by the operating system and based on the application data, a trained machine learning model to generate one or more suggested prompts for a large language model (LLM), wherein the one or more suggested prompts, upon being provided as input to the LLM, cause the LLM to generate an output relevant to an expected user activity; and providing, by the operating system, at least one of the one or more suggested prompts to be presented by a user interface associated with the operating system.

[0017] In a ninth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a device, cause the device to carry out functions. The functions include: receiving, by an operating system of a device, application data from a plurality of application programs running on the device; applying, by the operating system and based on the application data, a trained machine learning model to generate one or more suggested prompts for a large language model (LLM), wherein the one or more suggested prompts, upon being provided as input to the LLM, cause the LLM to generate an output relevant to an expected user activity; and providing, by the operating system, at least one of the one or more suggested prompts to be presented by a user interface associated with the operating system.

[0018] In a tenth aspect, a system is provided. The system includes means for receiving, by an operating system of a device, application data from a plurality of application programs running on the device; means for applying, by the operating system and based on the application data, a trained machine learning model to generate one or more suggested prompts for a large language model (LLM), wherein the one or more suggested prompts, upon being provided as input to the LLM, cause the LLM to generate an output relevant to an expected user activity; and means for providing, by the operating system, at least one of the one or more suggested prompts to be presented by a user interface associated with the operating system.

[0019] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF THE FIGURES

[0020] FIG. 1 is a diagram illustrating an example data item suggestion system, in accordance with example embodiments.

[0021] FIG. 2 illustrates an overview of an example data item suggestion system, in accordance with example embodiments.

[0022] FIG. 3 is a diagram illustrating an example data item suggestion system, in accordance with example embodiments.

[0023] FIG. 4 illustrates an example editable interface in the context of a shopping app, in accordance with example embodiments.

[0024] FIG. 5 illustrates an example editable interface in the context of an airline reservation app, in accordance with example embodiments.

[0025] FIG. 6 illustrates an example data item in the context of an airline reservation app, in accordance with example embodiments.

[0026] FIG. 7 illustrates example suggested data items in the context of an airline reservation app, in accordance with example embodiments.

[0027] FIG. 8 illustrates example suggested data items in the context of an airline reservation app, in accordance with example embodiments.

[0028] FIG. 9 illustrates example fillable data items in the context of an airline reservation app, in accordance with example embodiments.

[0029] FIG. 10 illustrates an example editable interface in a web browser in the context of shopping, in accordance with example embodiments.

[0030] FIG. 11 illustrates an example editable interface in a web browser in the context of tracking a shipment, in accordance with example embodiments.

[0031] FIG. 12 illustrates an example editable interface in a web browser in the context of renting an apartment, in accordance with example embodiments.

[0032] FIG. 13 illustrates an example editable interface in a web browser in the context of searching for a dentist, in accordance with example embodiments.

[0033] FIG. 14A is a diagram illustrating an example prompt suggestion system, in accordance with example embodiments.

[0034] FIG. 14B illustrates an overview of an example prompt suggestion system, in accordance with example embodiments.

[0035] FIG. 15 is a diagram illustrating an example prompt suggestion system, in accordance with example embodiments.

[0036] FIG. 16A illustrates example suggested prompts in the context of a shopping app, in accordance with example embodiments.

[0037] FIG. 16B illustrates example suggested prompts in the context of a transportation app, in accordance with example embodiments.

[0038] FIG. 16C illustrates example suggested prompts in the context of a dating app, in accordance with example embodiments.

[0039] FIG. 17A illustrates an example user interface for presenting suggested prompts, in accordance with example embodiments.

[0040] FIG. 17B illustrates another example user interface for presenting suggested prompts, in accordance with example embodiments.

[0041] FIG. 18A illustrates an example user interface for presenting suggested prompts in a foldable device, in accordance with example embodiments.

[0042] FIG. 18B illustrates another example user interface for presenting suggested prompts in a foldable device, in accordance with example embodiments.

[0043] FIG. 18C illustrates another example user interface for presenting suggested prompts in a foldable device, in accordance with example embodiments.

[0044] FIG. 19A illustrates an example user interface for presenting suggested prompts in a web browser window, in accordance with example embodiments.

[0045] FIG. 19B illustrates another example user interface for presenting suggested prompts in a web browser window, in accordance with example embodiments.

[0046] FIG. 19C illustrates another example user interface for presenting suggested prompts in a web browser window, in accordance with example embodiments.

[0047] FIG. 20A illustrates an example prompt suggestion system for an application program, in accordance with example embodiments.

[0048] FIG. 20B illustrates another example prompt suggestion system for an application program, in accordance with example embodiments.

[0049] FIG. 21 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments.

[0050] FIG. 22 depicts a distributed computing architecture, in accordance with example embodiments.

[0051] FIG. 23 is a block diagram of a computing device, in accordance with example embodiments.

[0052] FIG. 24 depicts a network of computing clusters arranged as a cloud-based server system, in accordance with example embodiments.

[0053] FIG. 25 is a flowchart of a method for proactive information retrieval, in accordance with example embodiments.

[0054] FIG. 26 is a flowchart of a method that exhibits improved computational efficiency, in accordance with example embodiments.DETAILED DESCRIPTIONOverview

[0055] This application generally relates to generating suggestions for possible data items to be entered in an editable interface. This involves generating prompts that may be input to a large language model (LLM) to cause the LLM to determine and provide the suggested data items. The term “prompt” as used herein generally refers to an input that may be provided to an LLM to generate an output. The prompt may be multimodal. Also, for example, the prompt may be a single phrase, audio, image, video, or a combination of one or more of these. A prompt may be an initial prompt, or a prompt that continues a previously initiated conversation with the LLM. A conversation may be a sequence of inputs to, and outputs from, an LLM.

[0056] Some approaches to suggestions for an autofill feature and / or an autocomplete feature are related technologies that provide input textual suggestions based on integrated user data and combining real-time text analysis with contextual actionable items. Generally, inputs and outputs to an artificial intelligence (Al) model can be multimodal (e.g., text, audio, video, image, etc.). Autofill and autocomplete features are generally not configured to utilize operating system awareness, and do not have multimodal capabilities. Although personalizedsuggestions can be generated based on user activities and interactions with application programs, such suggestions are in the form of tasks, hyperlinks, text, etc., and are not based on determining specific user data. Also, these suggestions are not multimodal.

[0057] For example, the Al agent may determine the context to be travel related and may identify a fillable reservation form (for an airline reservation, a hotel reservation, etc.). Accordingly, an LLM may be prompted to search across apps (and / or tabs of a web browser) for information related to the form, and retrieve information such as frequent flyer numbers, known traveler number (KTN), passport details, flight details, and so forth. The Al agent may provide such information at the relevant portions of the form, or auto-populate the information. Instead of, or in addition to, accessing saved lists for stored information, data retrieval can be performed “on the fly” by the LLM. For example, current systems save data for specific fields in the browser. Such an operation may be initiated by a user (e.g., saving a password to a site). As a system is trained over time, user inputs and / or other data may be continually assessed based on semantic data understanding. This may then be stored in the browser for later retrieval. For example, instead of the user looking through their emails, chats, messages, photos, etc. to find the information, the LLM is able to retrieve the relevant data based on the context. Although the user may directly use an LLM to attempt to retrieve such data, this may be challenging as the user may not be able to generate appropriate prompts to retrieve accurate information.

[0058] Generally speaking, the Al agent identifies the interface and the context, generates prompts in the background for an LLM to retrieve data relevant to the interface. There is prompt engineering that occurs on the fly based on the interface and the context. The interface has a structured format, and the context can be substantially accurately determined. Accordingly, the prompt can be designed to be definite and may also include a list of apps or locations to look for the information. For example, the prompt may include instructions on where to look for the information (e.g., GMAIL®, PHOTOS®, DRIVE®, CHROME®, etc ). As such, the LLM is not prone to “hallucinate” (e.g., generate incorrect, false, and / or non-existent data items) as it is searching within a known corpus of websites, apps, and / or documents. Once the LLM retrieves the data, the information may be ranked and provided to the user as suggestions. In some embodiments, the Al agent may auto-populate portions of the form. This can be a learnable aspect that improves over time based on user behavior. For example, the Al agent may be trained based on reinforcement learning techniques, or as a recurrent neural network (RNN) with ground truth data. An LLM may be configured to provide different typesof outputs such as learn from user activities, summarize content, compare content, search for information, and so forth.

[0059] In some embodiments, a data item suggestion system can be trained to guide the user to enter relevant information in an editable interface by highlighting all or portions of editable content presented by the editable interface. The user may additionally interact with suggested data items in several ways. For example, the user may select a suggested data item, complete a partial data item, ignore suggested data items, and / or enter their own data item (e.g. , without selecting any of the suggested data items). The data item suggestion system can be trained to learn from a history of such user interactions with previously suggested data items to deliver more customized and / or personalized data items in the future. The data item suggestion system may be configured to be trained on an ongoing basis to generate proactive suggested data items in the longer term. The proactive suggested data items may, in some aspects, be based on a history of user interactions with suggested reactive data items, such as types of data items selected by the user.

[0060] Another aspect of the data item suggestion system may be to integrate it with a web browser (e.g, CHROME® on a desktop device, or a mobile web browsing application). For example, the data item suggestion system could use signals from user interactions with application programs, an operating system context, and / or a web context to determine suggested data items. In some embodiments, the data item suggestion system may use an address in the web browser to crawl the HTML code for the page and determine web context and / or content being viewed by the user. Also, for example, user inputs into a search box (e.g, Omnibox in CHROME®) and / or web browsing history may be used to determine context. In some embodiments, data from across different tabs of a web browser may be used to determine context.

[0061] The data item suggestion system may be configured to create and / or complete actions on an OS platform, such as, for example, entering information into fillable forms, providing information when a user is interacting with a chatbot in a chat interface, selecting and / or entering relevant data items when the user is interacting with a web browser, etc. The data item suggestion system may also be integrated with existing apps (e.g., user media content libraries), Autofill and / or Autocomplete features to generate content across platforms and devices.

[0062] The suggested data items may be ranked based on various factors such as a context of the user interaction, a type of user interaction, a type of editable interface, and so forth.

[0063] In some embodiments, the data item suggestion system and / or the LLM (e.g., using GEMINI NANO®) can run locally on a mobile device. Such a smaller LLM can run faster and more efficiently. Once the data item suggestion system generates a prompt, the generated prompt may be provided to the on-device version of the LLM that retrieves the relevant and / or suggested data items. Also, for example, the generated prompt may be provided to a full version of the LLM residing in the cloud. In some embodiments, the LLM may be a large multimodal model (LMM).Example Data item suggestion systems

[0064] FIG. 1 is a diagram illustrating an example data item suggestion system 100, in accordance with example embodiments. An operating system of device 110 may support a plurality of software application programs (apps) 110a. Device 110 may include computing devices such as a laptop, a desktop computer, a smart television, an electronic reading device, a streaming content device, a gaming console, a tablet device, a wearable device, or other related computing devices that are configured to execute software instructions and application programs.

[0065] Some embodiments involve detecting a user interaction with an editable interface at a computing device. For example, device 110 may include a user interface 110b associated with the operating system. User interface 110b may include an editable interface 110c that enables a user to interact with device 110. The editable interface 110c can be configured for multimodal input, including speech, text, image, video, gestures, and so forth. For example, interface HOd indicates speech inputs via a microphone and / or image or video inputs via a camera.

[0066] In some embodiments, the editable interface 110c may include one of a web form, a chat interface, a short messaging service (SMS) interface, or an email interface. For example, the editable interface 110c may be a search box (e.g., Omnibox). Also, for example, the editable interface 110c may be an instant messaging interface. As another example, the editable interface 110c may be an email application. Also, for example, the editable interface 110c may be a chat interface of a company (e.g., a provider of services) or an application program. As another example, the editable interface 110c may be a fillable form (e.g. , on a web browser, in an app interface, etc.). In some embodiments, the editable interface may be configured for multimodal inputs, multimodal outputs, or both.

[0067] The term “interaction” can broadly refer to any activity, active and / or passive, performed by a user with device 110, or an application program of the plurality of software application programs 110a on device 110. For example, an interaction can involve viewingcontent, listening to content, inputting, editing, and / or modifying content (e.g., via a keyboard, a mouse, a tap, and so forth), a sensory interaction (e.g., haptic, visual, auditory, tactile, and so forth), a scrolling interaction, a voice interaction, a user selection, and so forth. In some embodiments, the interaction may not be a direct interaction of the user with the content. For example, the user may listen to a particular genre of songs or watch a particular genre of movies. The computing device may determine user interaction with a particular song from a particular genre as an interaction with songs of the same genre in the library. Likewise, the computing device may determine user interaction with a particular movie from a particular genre as an interaction with movies of the same genre in the library. As another example, a user interaction with an electronic mail can be determined to be an interaction with an entire chain of electronic mails, and / or a plurality of mail exchanges with a particular sender of the electronic mail.

[0068] In some embodiments, the user interaction may involve a search query input into a search field. Also, for example, the user interaction may involve an instant messaging conversation with one or more contacts. As another example, the user interaction may involve one or more electronic messages exchanged via an email application. Also, for example, the user interaction may involve an interaction with a chat interface of a company (e.g., a provider of services). As another example, the user interaction may involve an interaction with fillable forms (e.g., on a web browser, in an app interface, etc.). Additional and / or alternative forms of user interactions are considered within the scope of this application

[0069] The term “application program” (e.g., plurality of software application programs 110a, 1410a) or “app” as used herein, can be any computer program that is configured to interact with a user of device 110, 1410. Example application programs can include a search application, an email application, a text message application, an instant messaging application, a web browsing application, a mapping application, a media playback application, a weather application, a phone application, a video communication application, a camera application, an application associated with a service provider (e.g., financial, insurance, etc.), an application associated with a digital assistant (e.g., a home assistant), an application associated with a home improvement service, application associated with a crafts service, application associated with a shopping service, application associated with a rideshare service, application associated with a food delivery service, and / or any other application program configured to receive user input such as speech audio input, digital text input, alpha-numeric input, character input, video input, and / or digital image input.

[0070] In some embodiments, the ML model may reside on an operating system of the device 110. For example, data item suggestion system 105 may be configured as an on-device system. For example, data item suggestion system 105 may be configured to run on an operating system of device 110. In some embodiments, the data item suggestion system 105 may be, and / or one or more of its components may be, an intelligent assistant such as a machine learning (ML) model that can be trained to guide the user to enter relevant information in an editable interface. For example, the ML model may highlight all or portions of editable content presented by the editable interface. The data item suggestion system 105 may be configured to be trained on an ongoing basis to generate proactive suggested data items in the longer term. The proactive suggested data items may, in some aspects, be based on a history of user interactions with suggested reactive data items, such as types of data items selected by the user. Accordingly, rather than being exclusively restricted to fields and forms, data item suggestion system 105 may be configured to predict and interact with users based on multiple interactions across web and apps that across the operating system. Also, for example, data item suggestion system 105 may be configured to exist across multiple platforms and form factors, not just on mobile devices, but also on tablets, Android Auto, Google TV, and so forth.

[0071] Another aspect of the data item suggestion system 105 may be to integrate it with a web browser (e.g., CHROME® on a desktop device, or a mobile web browsing application). For example, the data item suggestion system 105 could be configured to use signals from user interactions with application programs, an operating system context, and / or a web context to determine suggested data items. Also, for example, the data item suggestion system 105 may be configured to create and / or complete actions on an OS platform, such as, for example, entering information into fillable forms, providing information when a user is interacting with a chatbot in a chat interface, selecting and / or entering relevant data items when the user is interacting with a web browser, etc. The data item suggestion system 105 may also be integrated with existing apps (e.g., user media content libraries), Autofill and / or Autocomplete features to generate content across platforms and devices.

[0072] In some embodiments, data item suggestion system 105 can include one or more interfaces 115a (e.g., by an application programming interface (API)) to communicate with the plurality of software application programs 110a on device 110. APIs are being used for illustrative purposes only. Generally, the application programs may interact with the operating system via additional, and / or alternative features such as software development kits, function calls, procedures, and so forth. Also, for example, the operating system may interact with one or more aspects of the plurality of software application programs 110a via cloud interfaces. Insome embodiments, the trained machine learning model may be configured as a chat agent for an application program of the plurality of application programs. For example, data item suggestion system 105 may be configured to operate as an intelligent assistant tailored to a particular app.

[0073] Some embodiments involve determining, based on the user interaction, a data item expected to be entered in the editable interface (e.g., editable interface 110c). For example, data item suggestion system 105 may include an interface analysis model 120 configured to analyze the editable interface.

[0074] In some embodiments, the editable interface may include a form with one or more fillable fields, and wherein the determining of the data item involves determining respective data values for the one or more fillable fields, wherein the determining is based on one or more of extracting forms, fields, or field labels. For example, the interface analysis model 120 may be configured to analyze forms (e.g., extract forms, fields and field labels).

[0075] In some embodiments, the interface analysis model 120 may include a data item determination model 125. The data item determination model 125 may be configured to identify a data item expected to be entered in the editable interface. For example, data item determination model 125 may be configured to use heuristics and / or crowdsourcing to predict form and field patterns, and determine what data item a user may need to search for and enter. In some embodiments, data item suggestion system 105 may combine data from multiple apps (e.g., email, cloud server, photos, contacts, etc.) to generate relevant data for data fields that would otherwise require investigation and / or advanced memory.

[0076] For example, the interface analysis model 120 may identify that the editable interface 110c is a fillable airline reservation form, and the data item determination model 125 may determine one or more data items expected to be entered into the airline reservation form. Such data items may include names of one or more passengers, dates of travel, origin and destination, passport numbers, known traveler numbers, frequent flyer numbers, coupons and / or discounts, and so forth.

[0077] As another example, the interface analysis model 120 may identify that the editable interface 110c is a fillable shopping form, and the data item determination model 125 may determine one or more data items expected to be entered into the shopping form. Such data items may include a shipping address, payment information, shoe size, waist length, inner seam length, a dress size, a color preference, hair color, measurements for prescription glasses, and so forth.

[0078] Also, for example, the interface analysis model 120 may identify that the editable interface 110c is a search interface for a home rental company, and the data item determination model 125 may determine one or more data items expected to be entered into the search interface. Such data items may include a location (e.g., a work address, a school location), a number of bedrooms, a number of bathrooms, one or more amenities, and so forth.

[0079] As another example, the interface analysis model 120 may identify that the editable interface 110c is a chat interface (e.g., for a telephone service provider), and the data item determination model 125 may determine one or more data items expected to be entered into the chat interface. Such data items may include a telephone number, an account number, a confirmation number for a service, and so forth.

[0080] Also, for example, the interface analysis model 120 may identify that the editable interface 110c is a search interface for a shipping service, and the data item determination model 125 may determine one or more data items expected to be entered into the search interface. Such data items may include a shipping confirmation number, an invoice number, a date of purchase, a shipping address, and so forth.

[0081] In some embodiments, the interface analysis model 120 may use the user interaction to determine the one or more data items expected to be entered. The term “expected to be entered in the editable interface” as used herein, may generally refer to an expected user action to be performed while interacting with editable interface 110c. For example, an expected user action may be to find dates for travel from an instant messaging application, a shipping confirmation number from an email application, a passport number from an image stored in an image library, a frequent flyer number, a known traveler number, a coupon, and so forth. The data item determination model 125 may use app data, user data, and / or device data to identify a data item expected to be entered in the editable interface. In some embodiments, the app data, user data, and / or device data may be represented as weighted signals that are aggregated to determine candidate data items expected to be entered in the editable interface. In some embodiments, the candidate data items may be ranked (e.g., based on user activities, preferences, etc.) and data item expected to be entered in the editable interface may be determined based on the ranking.

[0082] Some embodiments involve determining a context associated with the operating system. The candidate data items may be based on the determined context. For example, data item suggestion system 105 may include a context capture model (not shown) configured to capture application data. The application data may include data from the plurality of software application programs 110a via the one or more interfaces 115a. For example, the applicationdata may include an indication of an app category (e.g., transportation, navigation, media content, news, game, dating, food, crafts, messaging, calendar, email, and so forth). Also, for example, the context capture model may be configured to capture application data that includes user data, and / or device data 115b from the operating system, and / or a web browser. The user and / or device data 115b may include information related to the user, device location, time, history, and so forth.

[0083] Some embodiments involve generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggested entry associated with the data item from content accessible by the computing device 110. Upon determining the data item expected to be entered in the editable interface, prompt engineering model 130 may generate one or more prompts tailored to determine the at least one suggested entry. In some embodiments, the generated prompt may be a multimodal prompt.

[0084] In some embodiments, the generating of the prompt involves predicting, by the ML model, content relevant to the determination of the at least one suggested entry by the trained LLM. For example, the content relevant to the determining of the at least one suggested entry may include an instruction to search for the at least one suggested entry in an email application, a chat conversation, an image library, and so forth. In some embodiments, the ML model may identify content, such as a document, an image, an email, and so forth. Such embodiments also involve providing, by the ML model, the predicted content to the trained LLM. For example, the predicted content may be provided as part of the generated prompt that is provided to the LLM. Also, for example, the identified content such as the document, the image, the email, and so forth may be provided to the LLM.

[0085] In some embodiments, the LLM may be trained to determine the at least one suggested entry based on crowdsourcing to predict form and field patterns. For example, a plurality of user interactions of a plurality of users with editable interfaces may be analyzed to train the LLM. For example, the LLM may classify editable interfaces based on one or more fillable characteristics and learn to identify relevant data items based on the one or more fillable characteristics.

[0086] Some embodiments involve providing the generated prompt to the LLM to determine the at least one suggested entry associated with the data item. For example, when the one or more suggested prompts are provided as input to the LLM, the LLM may search for data items relevant to the data item expected to be entered. For example, the LLM may search one or more applications for a shipping confirmation number, details of a purchase transaction,a known traveler number, details of a driver’s license, details of a passport, contact information, an account number in an invoice, and so forth.

[0087] Some embodiments involve receiving, from the LLM, the at least one suggested entry. For example, the results of the search performed by the LLM may be received. In some embodiments, the LLM may determine the optimal results and provide such results. In some embodiments, the receiving of the at least one suggested entry involves receiving a ranked list of a plurality of suggested entries, and wherein the providing of the at least one suggested entry involves providing the ranked list of the plurality of suggested entries. For example, the results may be provided as a ranked list (e.g., ranked based on relevance to the user interaction, relevant to the context, relevant to the editable interface, and so forth). In some embodiments, the LLM may provide candidate data items and the data item suggestion system 105 may refine and / or rank the results based on one or more factors such as relevance to the user interaction, relevant to the context, relevant to the editable interface, relevance to the user, and so forth.

[0088] Some embodiments involve providing the at least one suggested entry at the editable interface 110c. In some embodiments, the editable interface 110c may be configured to provide functionality that enables a user to edit the at least one suggested entry. For example, the user may be able to modify an entry, complete a suggested partial entry, and so forth. Some embodiments involve receiving user selection from the ranked list of the plurality of suggested entries as the data item expected to be entered. Such embodiments involve entering the user selection as the data item expected to be entered. For example, the user may be able to select an entry from the ranked list of the plurality of suggested entries. Some embodiments involve automatically entering the at least one suggested entry as the data item expected to be entered.

[0089] In some embodiments, the functionality that enables the user to edit the at least one suggested entry may include functionality that enables multimodal edits. For example, an editable interface 110c may comprise an editing interface that facilitates editing of text, audio, video, images, etc. For example, when the at least one suggested entry is in textual format, the user may edit the text. Also, for example, when the at least one suggested entry is in image format, the user may be able to edit the image (e.g., using photo editing features such as cropping, changing a resolution, changing one or more image characteristics such as color, brightness, sharpness, contrast, etc.).

[0090] Data item suggestion system 105 can interface with one or more hardware components 135 e.g., a central processing unit (CPU), a graphics processing unit (GPU), memory, input and / or output devices of device 110). For example, data item suggestion system 105 can interface with one or more input devices, such as a keyboard, microphone(s) and / orcamera(s) 140. In some embodiments, the speech input by the user can be captured by microphone(s) 140. Microphone(s) may be a part of device 110, or may be an audio input device (e.g., a wired or wireless microphone) separate from device 110, and communicatively linked to device 110. Also, for example, data item suggestion system 105 can interface with an input method editor for text input. The text input may be by a digital or physical keyboard. For example, a touch screen of device 110 can display a digital keyboard. In some embodiments, different digital keyboards corresponding to different languages may be displayed. Also, for example, different digital keyboards corresponding to different layouts, designs, and so forth, may be displayed. As illustrated, device 110 may include user interface 110b associated with the operating system, and keyboard, microphone(s) and / or camera(s) 140 may be associated with user interface 110b, editable interface 110c, interface 1 lOd and so forth.

[0091] In some embodiments, the ML model may be located at a web browsing application, and wherein the editable interface is a fillable web form. For example, data item suggestion system 105 may interface with the web browsing application and enable automatic identification and entry of data items that may be expected to be entered in a fillable form displayed by the web browsing application.

[0092] In some embodiments, the ML model may be located at an interactive user interface for another LLM. For example, another LLM (e.g., residing on the device) may provide an interface for user interaction, and data item suggestion system 105 may interface with the interface for user interaction.

[0093] FIG. 2 illustrates an overview of an example data item suggestion system 200, in accordance with example embodiments. In some embodiments, one or more of a plurality of application programming interfaces (APIs) 205 running on the device may provide editable interfaces, and the user may interact with such editable interfaces. In some embodiments, interface analysis system 210 may detect such user interactions. In some embodiments, interface analysis system 210 may determine a data item expected to be entered in an editable interface.

[0094] In some embodiments, a prompt engineering system 215 may generate a prompt 220 to be provided to a large language model (LLM) 225, wherein the prompt 220 comprises one or more instructions that cause the LLM 225 to determine at least one suggested entry 230 associated with the data item from content accessible by the computing device (e.g., APIs 205). One or more suggested entries 235 may be presented at the editable interface. In some embodiments, the content accessible by the computing device may include content from one or more application programs on the computing device. For example, the content may be oneor more emails from an email application, images from an image library, contact information, chat conversations, and so forth. In some embodiments, the content accessible by the computing device may include content from an operating system of the computing device. For example, the content may include a state of the operating system, location data, timing data, sensor data, and so forth.

[0095] FIG. 3 is a diagram illustrating an example data item suggestion system 300, in accordance with example embodiments. A plurality of application programs (apps) are shown, however, this list is not exhaustive and is for purposes of illustration only. Generally speaking, the data item suggestion system 300 may be able to analyze an editable interface provided by any app that interfaces with the computing device (e.g., an operating system of the device). The plurality of apps may include an electronic mail (email) app 305, a short messaging system (SMS) app 310, a weather app 315, a web browser app 320, a calendar app 325, a phone app 330, a cloud storage app 335, a camera app 340, a rideshare app 345, a travel app 350, and so forth.

[0096] Generally speaking, data item suggestion system 300 may be configured with appropriate privacy controls that can be managed by a user. For example, a user may be able to opt-in to the data item suggestion service, and select which, if any, apps may be used for purposes of the data item suggestion service. For example, the user may have opted not to share information from apps shown with dashed boundaries (e.g., SMS app 310, phone app 330, camera app 340, etc.). Accordingly, although information about an app category may be available to data item suggestion system 300, the user may elect not to share more detailed information about the app.

[0097] For example, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user’s social network, social contacts, or activities, a user’s preferences, or a user’s current location, and so forth), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personal data is removed, secured, encrypted, and so forth. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user. In addition to user controls, where user information is used to generatedata item suggestions, such user information may be restricted to the user’s device, and not shared with a server, and / or with other devices. Also, for example, the user information may be deleted after use. For example, in the instance where the user consents to the use of such application data, the application data may be used to determine the data item suggestions, and the application data and / or the user inputs may then be securely stored on the device, and not shared with other devices, servers, and so forth.

[0098] Interface analysis model 355 may share one or more aspects in common with interface analysis model 120 of FIG. 1. Interface analysis model 355 may be configured to analyze an editable interface associated with the plurality of apps. In some embodiments, the interface analysis model 355 may use a user interaction with an editable interface to determine the one or more data items expected to be entered.

[0099] Data item determination model 360 may share one or more aspects in common with data item determination model 125 of FIG. 1. Data item determination model 360 may be configured to identify a data item expected to be entered in the editable interface. In some embodiments, the interface analysis model 355 may include a context analysis model (not shown) that may determine an app category. An app category may be, for example, a social media app, a news app, a business app, a gaming app, an education app, a lifestyle app, a finance app, a productivity app, a travel app, a weather app, a navigation / mapping app, and so forth. So apps may belong to more than one category. The app category may be indicative of a general user intent to search for a particular data item when interacting with an editable interface.[000100] Prompt engineering model 365 may share one or more aspects in common with prompt engineering model 130 of FIG. 1. Prompt engineering model 365 may be configured to apply a trained machine learning model to generate one or more prompts for large language model (LLM) 370. The one or more prompts may be tailored to determine at least one suggested entry, and the one or more prompts may be input to LLM 370 to generate at least one suggested entry associated with the data item.[000101] In some embodiments, information indicating an app category may enable a prompt engineering system to include such information as part of a prompt. For example, a travel app (e.g., a hotel app and an airline app) may indicate that the user is searching for data items related to a travel reservation. As another example, a calendar app and a food delivery app (e.g., a grocery store app) may indicate that the user is shopping for a meal and may search for data items associated with a previous purchase (e.g., items ordered, etc.). Also, for example, a crafts app, a media content app, and a photo library app may indicate that the user is lookingto create an invitation to an event, purchase a gift for an event, etc. and that the user may search for names and addresses of contacts, event dates, venue reservations, and so forth.[000102] The term “large language model” as used herein, may generally refer to any machine learning model that is capable of performing natural language processing (NLP) tasks. In some embodiments, the underlying architecture for the LLM may be transformer based, although other types of architectures may be used as well. In some embodiments, LLM 370 may include one or more LLMs, and a particular LLM may be selected based on a determination of one or more data items expected to be entered. LLM 370 may include, for example, language representation models that generate natural language outputs, multimodal models that can process and generate content in multiple modalities, such as text, audio, image, video, etc. Also, for example, LLM 370 may include zero-shot models that are not trained based on task-specific training data and can perform multiple tasks. As another example, LLM 370 may include fine-tuned models that are trained to perform specific tasks. In some embodiments, LLM 370 may be a retrieval augmented generation (RAG) system, a semantic embedding retrieval system, and so forth.[000103] LLM 370 may be configured to provide different types of output such as Al generated content, learn from user activities, summarize content, compare content, search for information, provide explanations, and / or inspire a user to perform a new action. Accordingly, one or more signals can be used to generate prompts (e.g., a type of modality, a type of query, etc.) that can be input to the LLM 370 for a desired type of output.[000104] In some embodiments, a version of the LLM may reside on the device. For example, LLM 370 can run locally on a mobile device. Such a smaller LLM 370 can run faster and more efficiently. Once a prompt is generated, the prompt may be provided to the on-device version of LLM 370. In some embodiments, the prompt may be provided to a full version of LLM 370 residing in the cloud.[000105] Applying LLM 370 at the device has several advantages. For example, the application data does not leave the device (e.g., is not shared with a server, with other devices, and so forth), and is used for on-device processing. This has several advantages. For example, by restricting content to the device, any user information that may be used for generating suggested prompts is maintained with appropriate privacy and / or security controls. For example, the content may be encrypted, stored in a dedicated memory location on the device, and so forth. Such features also enhance the capabilities of the data item suggestion system, as the suggested prompts can be suitably personalized to a particular user, user preferences, and so forth, based on user control of an extent of the personalization. Also, for example, the usercan retain control over what information is collected about the user, how that information is used, and what information is provided to the user. On-device processing is advantageous as it enables faster processing, lower latency, and less data transmission over a network. Also, for example, in situations where a server may be unavailable, a network may be unavailable, a secured network may be unavailable, and / or network bandwidth may be limited, on-device processing can continue to generate customized prompt suggestions.[000106] Graphical user interface (GUI) 375 may be associated with the operating system and may share one or more aspects in common with user interface 110b. In some embodiments, GUI 375 may include an editable interface 380. For example, in an input mode, GUI 375 may include editable interface 110c, and interface HOd of FIG. 1. Also, for example, in an output mode, editable interface 380 may be configured to provide the output generated by LLM 370. [000107] In some embodiments, the one or more generated prompts may be configured as multimodal inputs to the LLM 370. A generated prompt may be in textual format, audio format (e.g., speech, music), image format, video format, and so forth. Such embodiments involve generating a feature embedding associated with the particular prompt. The feature embedding generally transforms the particular prompt into an input format compatible with an expected input for LLM 370. Such embodiments also involve providing, to LLM 370, the feature embedding. Such embodiments further involve receiving, from LLM 370, and based on the feature embedding, a particular output associated with the particular prompt. Such embodiments also involve providing, by GUI 375, the particular output.Example Suggested Data Items[000108] FIG. 4 illustrates an example editable interface 400 in the context of a shopping app, in accordance with example embodiments. For example, a user may be interacting with a shopping app and a payment page of the shopping app may include an editable interface. The editable interface may be configured to enable the user to enter a promotion code (promo code) 405 in a dialog box 410. Accordingly, the interface analysis model may determine a category for the app, the data item determination model may determine that the promo code is a data item expected to be entered. A prompt may be generated to search for the promo code. In some embodiments, the prompt may instruct the LLM to search in an email application, the shopping app, an SMS app, or a combination thereof. The LLM may search and find the promo code. In some embodiments, the promo code may be automatically entered in the dialog box 410. In some embodiments, the user may be provided a selection of promo codes that may be applied, and the user may select a from among these promo codes, which may then be entered in thedialog box 410. In some embodiments, the user may choose to manually enter a promo code, possibly different from the promo codes provided by the LLM.[000109] FIG. 5 illustrates an example editable interface 500 in the context of an airline reservation app, in accordance with example embodiments. For example, a user may be interacting with an airline reservation app and a check in page of the airline reservation app may include an editable interface. The editable interface may be configured to enable the user to enter a confirmation number 505. Accordingly, the interface analysis model may determine a category for the app, the data item determination model may determine that the confirmation number is a data item expected to be entered. A prompt may be generated to search for the confirmation number. In some embodiments, the prompt may instruct the LLM to search in one or more applications such as an email application, the airline reservation app, an SMS app, or a combination thereof. The LLM may search and find the confirmation number. In some embodiments, the confirmation number may be automatically entered. In some embodiments, the user may be provided a selection of confirmation numbers that may be applied, and the user may select form among these confirmation numbers, which may then be entered. In some embodiments, the user may choose to manually enter a confirmation number, possibly different from the confirmation numbers provided by the LLM.[000110] FIG. 6 illustrates an example data item in the context of an airline reservation activity 600, in accordance with example embodiments. For example, a user may be interacting with a web browser 605 displaying a reservation form. The reservation form may include a dialog box 610 where the user may enter a known traveler number (KTN). Accordingly, the interface analysis model may determine that the web browser relates to an airline reservation, and the data item determination model may determine that the KTN is a data item expected to be entered. A prompt may be generated to search for the KTN. In some embodiments, the user may be provided the determined KTN 615, and the user may select the KTN, which may then be entered into dialog box 610.[000111] Also, for example, the user may be interacting with an airline app 620 displaying a reservation form. The reservation form may include a dialog box 625 where the user may enter a known traveler number (KTN). Accordingly, the interface analysis model may determine that the app relates to an airline reservation, and the data item determination model may determine that the KTN is a data item expected to be entered. A prompt may be generated to search for the KTN. In some embodiments, the user may be provided the determined KTN 615, and the user may select the KTN, which may then be entered into dialog box 625.[000112] FIG. 7 illustrates example suggested data items in the context of a shipment tracking app 700, in accordance with example embodiments. For example, a user may be interacting with a shipment tracking app 700 displaying a dialog box 705 where the user may enter a tracking number. Accordingly, the interface analysis model may determine that the app relates to shipment tracking, and the data item determination model may determine that the tracking number is a data item expected to be entered. A prompt may be generated to search for the tracking number. In some embodiments, the user may be provided a list of tracking numbers 710. For example, the user may be expecting multiple shipments from different companies such as Company A, Company B, Company C. Each shipment may be associated with a respective tracking number. Accordingly, the user may be provided with these tracking numbers (e.g., as a ranked list), and the user may select one of the tracking numbers from the list of tracking numbers 710, which may then be entered into dialog box 705.[000113] FIG. 8 illustrates example suggested data items in the context of a chat interface for an airline reservation app 800, in accordance with example embodiments. For example, a user may be interacting with an airline app 800 displaying a chat interface. For example, the user may be communicating with an agent (e.g., a live agent that may be real or virtual). The user may enter a first query 805 that states “Need help with my flight.” The chat agent may provide a response 810 that states “What’s your confirmation number for your flight?” The chat interface may include a query box 815 where the user may enter the confirmation number. Accordingly, the interface analysis model may determine that the app relates to an airline reservation, and the data item determination model may determine that the confirmation number is a data item expected to be entered. A prompt may be generated to search for the confirmation number. In some embodiments, the user may be provided the determined confirmation number 820, and the user may select the confirmation number 820, which may then be entered into query box 815.[000114] FIG. 9 illustrates example fillable data items in the context of a travel reservation app 900, in accordance with example embodiments. For example, the user may be interacting with the travel reservation app 900. The app interface may display multiple tabs, such as, for example, a “Book” tab 905A to make a reservation, a “Flight status” tab 905B to check a status of a flight, a :Check-in” tab 905C to enable a user to check-in for an upcoming flight, check-in to a cruise, check-in to a hotel, etc., a “My trips” tab 905D that provides information about a user’s past and / or upcoming trips, and so forth.[000115] In the illustrative example, the user may be interacting with the “Book” tab 905 A. In some embodiments, the “Book” tab 905 A may include further tabs for different typesof reservations such as for flights, packages, hotels, cars, cruises, etc. As an illustration, the flight tab may include an editable interface with a fillable form 910. The fillable form may include multiple data fields, such as for an originating airport 915, a destination airport 920, travel dates 925, a number of travelers 930, a class of travel 935 (e.g., basic economy, economy, premium economy, business, first, etc.).[000116] Accordingly, the interface analysis model may determine that the editable interface relates to an airline reservation, and the data item determination model may determine that the there are multiple data items corresponding to the multiple data fields that are expected to be filled. A prompt may be generated to search for the multiple data items. In some embodiments, the user may be provided the determined multiple data items, which may then be entered into the respective data fields. For example, the originating airport 915 may be determined to be “Denver,” the destination airport 920 may be determined to be “New York City,” the travel dates 925 may be determined to be “Jun 30 - Jul 06,” the number of travelers 930 may be determined to be “4,” and a class of travel 935 may be determined to be “Economy.” In some embodiments, the LLM may determine the multiple data items from one or more apps and / or conversations. For example, a family chat group or an email conversation thread may have included a discussion of vacation plans. Accordingly, the multiple data items corresponding to the multiple data fields may be determined from such user interactions. As previously noted, a user would need to permit access to such apps and data and be provided with opt-out options.[000117] Another aspect of the data item suggestion system may be to integrate it with a web browser (e.g., CHROME® on a desktop device, or a mobile web browsing application). For example, the data item suggestion system could use the signals from user activities and OS context and add web context to identify a data item expected to be entered in an editable interface and to generate personalized data items relevant for input to the editable interface. In some embodiments, the data item suggestion system may use an address in the web browser to crawl the HTML code for the page and determine web context and / or content being viewed by the user. Also, for example, user inputs into a search box (e.g., Omnibox in CHROME®) and / or web browsing history may be used to determine context and / or to determine at least one suggested entry associated with the data item. In some embodiments, data from across different tabs of a web browser may be used to determine context and / or to determine at least one suggested entry associated with the data item. Generally speaking, the data item suggestion system may use a combination of application data from the apps, device data from the OS, and / or browsing data from a web browser to generate a prompt to be provided to a LLM,wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggested entry associated with the data item from content accessible by the computing device. Also, for example, the LLM may use a combination of application data from the apps, device data from the OS, and / or browsing data from a web browser to determine at least one suggested entry associated with the data item.[000118] FIG. 10 illustrates an example editable interface 1000 in a web browser in the context of shopping, in accordance with example embodiments. For example, a user interaction may include entering a prompt in a search box 1005 (e.g., Omnibox in CHROME®. The search query or prompt 1010 may be “Get me the best deal on Co. A brand X 8 shoes.” Accordingly, the interface analysis model may determine that the web browser and / or browsing activity relates to shopping for shoes. In some embodiments, search techniques and / or an LLM may be applied to find different shoe merchants, and one or more browser tabs may be automatically opened, such as separate tabs for Shoe Co. A, Shoe Co. B, Shoe Co. C, and so forth. In some embodiments, search techniques and / or an LLM may be applied to identify the shoe merchant offering the best deal. For example, the data item suggestion system may display a web page for Shoe Co. A with a “Co. A brand X 8 shoe.” Subsequently, the data item determination model may determine that the shoe size for the user is a data item expected to be entered. A prompt may be generated to search for the shoe size, and the data item determination model may perform an autofill 1015 of the determined shoe size (e.g., “7”).[000119] Also, for example, once the selected shoe is added to the shopping cart, the interface analysis model may analyze a displayed payment page to complete a purchase based on a best deal via autofill 1020. For example, the interface analysis model may determine that the payment page relates to shopping for a shoe on a website for Shoe Co. A. Accordingly, the data item determination model may determine that payment information 1025 may be the one or more data items expected to be entered. In some embodiments, a stored payment method may be used to autofill the payment information 1025. Also, for example, the data item determination model may determine that information related to a “best deal” may be one or more additional data items expected to be entered. A prompt may be generated to search for various deals 1030, such as, for example, credit card deals, merchant coupons, and so forth. For example, deals 1030 may include a “best credit card deal,” “5% back on ShoeCoA.com,” “3x points on online shopping,” and so forth. Accordingly, the user may be provided with these deals 1030 (e.g., as a ranked list), and the user may select one of the deals from the list of deals 1030, which may then be entered as part of the payment information 1025.[000120] FIG. 11 illustrates an example editable interface 1100 in a web browser in the context of tracking a shipment, in accordance with example embodiments. For example, a user may enter a prompt in a search box 1105 (e.g., Omnibox in CHROME®) by opening a new tab 1110 and entering the prompt 1115 that states “Create tabs of all my upcoming packages.” In some embodiments, the interface analysis model may determine that multiple tabs are to be opened related to the user’s expected packages. Accordingly, the data item determination model may determine that tracking numbers for all expected packages are data items expected to be entered in each of the multiple tabs. In some embodiments, the multiple tabs may be opened by the user, and autofill may be applied across such multiple tabs. Accordingly, one or more prompts 1120 may be generated such as “track packages,” “track packages ordered in past month,” track packages not delivered yet,” “When is my humidifier coming,” “When is the humidifier being delivered,” create tabs of all packages,” and so forth.[000121] In some embodiments, a tabbed groups and autofill operation 1125 may be performed by the data item suggestion system. For example, the LLM may identify that the user is expecting packages from three shipping companies, and the data item suggestion system may automatically open a tab for each shipping company, such as a first tab 1130 for Co. A, a second tab 1135 for Co. B, and a third tab 1140 for Co. C. Also, for example, a web page corresponding to the first tab 1130 for Co. A may include an editable interface 1145 where one or more tracking numbers may be entered. Based on the output of the LLM, the data item suggestion system may automatically enter the one or more tracking numbers 1150. For example, the LLM may search and find the one or more tracking numbers in content in an email application.[000122] FIG. 12 illustrates an example editable interface 1200 in a web browser in the context of renting an apartment, in accordance with example embodiments. For example, a user may enter a prompt in a search box 1210 (e.g., Omnibox in CHROME®) by opening a new tab 1205 and entering the prompt that states “Create a list of 1 BD apartments to rent near my work.” In some embodiments, the interface analysis model may determine that multiple tabs are to be opened related to the user’s expected apartment rental request. In some embodiments, the data item determination model may determine that the user’s work address is a data item expected to be entered in each of the multiple tabs in order to search for rental apartments close to the user’s workplace. Accordingly, one or more prompts may be generated such as “find user’s work address,” “search in the email app for the work address,” and so forth. In some embodiments, the LLM may search the user’s email app to determine an address 1215 for theuser’s work headquarters. In some embodiments, the multiple tabs may be opened by the user, and autofill may be applied across such multiple tabs.[000123] Subsequently, the data item suggestion system may automatically create a group of tabs 1240 corresponding to rental websites, such as a first tab 1220 for Rental A, a second tab 1225 for Rental B, a third tab 1230 for Rental C, a fourth tab 1235 for Rental D, and so forth. Also, for example, the data item determination model may determine that multiple data items (e.g., a number of bedrooms, a number of baths, a minimum square foot requirement, types of amenities, length of a lease, a type of housing, a price range, and so forth may be expected to be entered. Accordingly, one or more prompts may be generated to cause the LLM to search for such data items. The search may be based on the prompt in a search box 1210 (e.g., Omnibox in CHROME®), the determined work address, past user search history, user’s current lodging parameters, an email or chat conversation indicating user preferences, and so forth. Also, for example, when required as a move-in document, an employment offer letter 1245 may be found and uploaded to fill out a lease form. In some embodiments, the data item suggestion system may automatically pre-populate the search fields 1250 corresponding to such parameters.[000124] FIG. 13 illustrates an example editable interface 1300 in a web browser in the context of searching for a dentist, in accordance with example embodiments. For example, a user may perform a search 1305 by entering a prompt such as “find top 3 dentists that are kind” in a search box 1310 (e.g., Omnibox in CHROME®). Based on the techniques described herein, in some embodiments, multiple tabs may be automatically opened by the data item suggestion system, such as for example, a first tab 1305 for “Philips, DDS,” a second tab 1320 for “James, DDS,” and a third tab 1325 for “Lynelle, DDS.” In some embodiments, the multiple tabs may be opened by the user, and autofill may be applied across such multiple tabs.[000125] In some embodiments, the data item suggestion system may perform an autofill operation 1330 to fill out a new patient intake form 1335. For example, the interface analysis model may identify one or more data fields in the new patient intake form 1335. In some embodiments, the data item determination model may determine that multiple data items such as contact information, emergency contact information, dental insurance, and so forth may be expected to be entered. Accordingly, one or more prompts are generated for an LLM to search and determine the multiple data items. In some embodiments, the contact information may be found from saved favorites in contacts or previously filled forms. Also, for example, the dental insurance information may be found from an email application, or from an image library. Insome embodiments, the data item suggestion system may automatically fill in the new patient intake form 1335 based on the information determined by the LLM.[000126] In some embodiments, the determining of the browsing context may be based on the content of a web page. For example, an analysis of an HTML code for a web page may indicate more detailed browsing context. For example, the content of the web page may relate to rental homes, a shopping webpage, a travel reservations webpage, a payment page, a patient intake form, and so forth. As another example, the content of the web page may relate to a search for airline fares to a location. Also, for example, the content of the web page may relate to an online shopping platform with specific items being displayed.[000127] In some embodiments, the determining of the browsing context may be based on content displayed across a plurality of tabs of the web browser. For example, a first tab may display a web page related to tourist attractions and a second tab may display a web page related to a web page for a search for airline fares to a location. Accordingly, the browsing context may be travel to a destination and tourist attractions for that destination. Accordingly, a data item expected to be entered may include travel related information, dates of travel, number of travelers, travel discounts, and so forth. In some embodiments, a third tab may display a web page related to an online shopping platform with specific clothing items being displayed. Accordingly, the browsing context may be travel to a destination, tourist attractions for that destination, and appropriate attire, and a data item expected to be entered may include shopping related information such as a shoe size, a dress size, a waist length, and so forth.[000128] In some embodiments, the determining of the browsing context may be based on an input into a search field of the web browser. For example, a user may be interacting with web browser window, and a tab may be open. For example, the user may be performing a search based on a search query. For example, the search query may be for “best used SUVs under $30,000," and one or more search results may be displayed. For example, a first search result, a second search result, and so forth, may be displayed. In some embodiments, the data item suggestion system may display a fillable form (e.g., overlaid over the browsing window). The data item suggestion system may present one or more data items expected to be entered. The one or more data items expected to be entered may be based on the browsing context.[000129] In some embodiments, the determining of the browsing context may be based on a browsing history associated with the web browser. For example, the data item suggestion system may present one or more additional data items expected to be entered that are tailored to the user’s search history. In some embodiments, fillable form and the one or more additional data items expected to be entered may be presented together.Proactive and Reactive Personalized Suggestive Multimodal PromptsOverview[000130] In another aspect, this application generally relates to generating suggested prompts that may be input to a large language model (LLM). Information such as user data, device data, along with context from application programs on the device, user interactions, and so forth, may be analyzed to determine an expected user activity. A machine learning model can be trained to generate personalized prompts based on such information that, upon being provided as input to the LLM, cause the LLM to generate an output relevant to the expected user activity.[000131] Some approaches to suggestions for an autofill feature and / or an autocomplete feature are related technologies that provide input textual suggestions based on integrated user data and combining real-time text analysis with contextual actionable items. Generally, inputs and outputs to an artificial intelligence (Al) model can be multimodal (e.g., text, audio, video, image, etc.). Autofill and autocomplete features are generally not configured to utilize operating system awareness, and do not have multimodal capabilities. Although personalized suggestions can be generated based on user activities and interactions with application programs, such suggestions are in the form of tasks, hyperlinks, text, etc., and are not in a form that can be used as a prompt to an LLM. Also, these suggestions are generally not multimodal. [000132] Some existing search engines associated with browser applications may include an Al-powered chat mode. Although such Al-powered chat modes may be configured to generate prompts, these prompts are based on a user’ s browsing activity, such as a search query and / or search history, and the prompts are not configured to seamlessly integrate with the operating system or use context aware information from application programs running on the operating system. Also, for example, context based on content across different tabs of a browser is also not used in generating personalized, contextual prompts. As another example, although existing prompt interfaces allow a user to enter a prompt or to select a suggested prompt, there is no editing interface for the user to edit a suggested prompt (e.g., in a multimodal manner), select from suggested options to complete a partial prompt, and / or to enter a prompt and review suggested prompts.[000133] As described herein, reactive personalized multimodal prompts may be generated by a prompt suggestion system (e.g., an Al assistant) by combining one or more signals based on user interactions with various application programs of an operating system. The user interactions can be used to determine an expected user activity, and this can be used to personalize a prompt relevant to the user’s needs. In some aspects, the one or more signalscan be based on user data from activities (e.g., web and app activity settings, location information, etc.), along with operating system (OS) context (e.g., device model, sensors, app information, etc.). For example, an app category (e.g., weather, shopping, media, transportation, ride-share, news, entertainment, sports, etc.) may be used to identify user activities. Users can be provided with controls for privacy settings that allow them control over what activities are shared with the Al assistant. Also, for example, context from a browser may be used as a signal for generating prompts.[000134] For example, the prompt suggestion system may determine that a user has a concert ticket (e.g., from a messaging platform, an email application, etc.) as part of user data from activities, and has a rideshare application open as part of the OS context. Accordingly, the prompt suggestion system may combine these two signals to generate personalized, suggestive prompts. For example, the prompt can relate to queries for restaurants near the concert venue, queries for directions on a map to pick up another person who is also going to the concert, etc.[000135] As another example, the prompt suggestion system may determine that a user has an airline ticket to visit another country (e.g., from a messaging platform, an email application, etc.) as part of user data from activities, and has a weather application open as part of the OS context. In some embodiments, location data from the OS context may indicate that the user is at a foreign location. Accordingly, the prompt suggestion system may combine these signals to generate personalized, suggestive prompts. For example, the prompt can relate to queries for possible clothing or other accessories to be purchased, queries for outdoor activities that are suited to the weather conditions at the location, etc.[000136] Also, for example, the prompt suggestion system may determine that a user is preparing for a dinner (e.g., from a messaging platform, an email application, a calendar application, etc.) as part of user data from activities, and has a shopping application open as part of the OS context. Accordingly, the prompt suggestion system may combine these two signals to generate personalized, suggestive prompts. For example, the prompt can relate to queries for what attire may be appropriate for the dinner, what type of wine to take, what type of dessert to take, etc.[000137] An LLM may be configured to provide different types of outputs such as Al generated content, learn from user activities, summarize content, compare content, search for information, provide explanations, and / or inspire a user to perform a new action. Accordingly, the one or more signals can be used to generate prompts (e.g., a type of modality, a type of query, etc.) that can be input to the LLM for a desired type of output. The suggested promptscan be partial prompts that may be customized by a user, user-initiated prompts that may be automatically completed, and / or entire prompts that may be provided to a user. In some embodiments, a suggested prompt can be presented, and the user may be provided with editing tools to edit the suggested prompt. For example, the user interface may comprise an editing interface that facilitates multimodal editing of text, audio, video, images, or a combination thereof. Also, for example, the editing interface can be configured to receive and present prompts in different languages.[000138] In some embodiments, the prompt suggestion system can be trained to guide the user to facilitate prompt engineering by highlighting all or portions of a suggested prompt to be editable by the user. The user may additionally interact with suggested prompts in several ways. For example, the user may select a suggested prompt, complete a partial prompt, ignore suggested prompts and / or enter their own prompt (e.g., without selecting any of the suggested prompts). The prompt suggestion system can be trained to learn from a history of such user interactions with previously suggested prompts to deliver more customized and / or personalized prompts in the future. Although the prompts can be reactive in the near term, the prompt suggestion system may be configured to be trained on an ongoing basis to generate proactive prompts in the longer term. The proactive prompts may, in some aspects, be based on a history of user interactions with suggested reactive prompts, such as types of prompts selected by the user and the signals that were combined to generate the selected prompts. Accordingly, when similar signals are observed, a proactive prompt may be generated for the user.[000139] A prompt interface for the prompt suggestion system may be adapted to a flip and fold device. For example, suggested prompts may be provided in a “tabletop” mode. Also, for example, a “drag and drop feature” can be used to allow the user to interact with the prompt interface over the two sides of a flip and fold device. In some embodiments, the prompt interface may be provided as a widget on a cover screen.[000140] Another aspect of the prompt suggestion system may be to integrate it with a web browser (e.g., CHROME® on a desktop device, or a mobile web browsing application). For example, the prompt suggestion system could use the signals from user interactions with application programs, an operating system context, and / or a web context to generate personalized prompts. In some embodiments, the prompt suggestion system may use an address in the web browser to crawl the HTML code for the page and determine web context and / or content being viewed by the user. Also, for example, user inputs into a search box (e.g., Omnibox in CHROME®) and / or web browsing history may be used to determine context. Insome embodiments, data from across different tabs of a web browser may be used to determine context.[000141] The application developer community may also be incentivized to provide additional data about their application programs. For example, the prompt suggestion system may facilitate user navigation of an app, content available within an app may be promoted by designing appropriate prompts for the user, in-app payments may be facilitated, and so forth. Also, for example, the app developers may be incentivized to integrate the prompt suggestion system with their apps to serve as app-specific Al assistants (e.g., chatbots) that may be tailored to a particular app. As more and more apps are integrated, the prompt suggestion system is able to utilize more complex signals to generate more accurate prompts. Also, for example, the prompt suggestion system may be configured to learn from user activities within an app to create a more personalized user experience.[000142] The prompt suggestion system may be configured to create and / or complete actions on an OS platform, such as, for example, silencing notifications, creating alarms, etc. As another example, the prompt suggestion system may be configured to create themes, wallpapers, interface designs, app layouts, etc. on a mobile device. The prompt suggestion system may also be integrated with existing apps (e.g., user media content libraries), Autofill and / or Autocomplete features to generate content across platforms and devices, create social media posts (e.g., using images, audio, and / or video captured by a user to generate new content).[000143] As with suggestions, search results, etc., suggested prompts may be ranked based on various factors such as activities associated with a particular user, the types of signals, and so forth.[000144] In some embodiments, the prompt suggestion system and / or the LLM (e.g., using Gemini Nano) can run locally on a mobile device. Such a smaller LLM can run faster and more efficiently. Once a prompt is generated, and a user selection of a prompt is received, the selected prompt may be provided to the on-device version of the LLM. Also, for example, the selected prompt may be provided to a full version of the LLM residing in the cloud.Example Prompt Suggestion Systems[000145] FIG. 14A is a diagram illustrating an example prompt suggestion system 1400 A, in accordance with example embodiments. An operating system of device 1410 may support a plurality of software application programs (apps) 1410a. Device 1410 may include computing devices such as a laptop, a desktop computer, a smart television, an electronic reading device, a streaming content device, a gaming console, a tablet device, a wearabledevice, or other related computing devices that are configured to execute software instructions and application programs.[000146] In some embodiments, device 1410 may include a user interface 1410b associated with the operating system. User interface 1410b may include a user input mode 1410c that enables a user to interact with device 1410. The user input mode 1410c can be configured for multimodal input, including speech, text, image, video, gestures, and so forth. For example, interface 1410d indicates speech inputs via a microphone and / or image or video inputs via a camera.[000147] In some embodiments, an on-device prompt engineering system 1405 can be configured to run on an operating system of device 1410. On-device prompt engineering system 1405 can include one or more interfaces 1415a (e.g., by an application programming interface (API)) to communicate with the plurality of software application programs 1410a on device 1410. APIs are being used for illustrative purposes only. Generally, the application programs may interact with the operating system via additional, and / or alternative features such as software development kits, function calls, procedures, and so forth. Also, for example, the operating system may interact with one or more aspects of the plurality of software application programs 1410a via cloud interfaces.[000148] Some embodiments involve determining a context associated with the operating system. The at least one of the one or more suggested prompts may be based on the determined context. For example, on-device prompt engineering system 1405 may include a context capture model 1420 configured to capture application data. The application data may include data from the plurality of software application programs 1410a via the one or more interfaces 1415a. For example, the application data may include an indication of an app category (e.g., transportation, navigation, media content, news, game, dating, food, crafts, messaging, calendar, email, and so forth). Also, for example, context capture model 1420 may be configured to capture application data that includes user data, and / or device data 1415b from the operating system, and / or a web browser. The user and / or device data 1415b may include information related to the user, device location, time, history, and so forth.[000149] On-device prompt engineering system 1405 may include a context analysis model 1425 configured to analyze the captured app data, user data, and / or device data. Generally speaking, context analysis model 1425 may use such data to determine an expected user activity. The term “expected user activity” as used herein, may generally refer to a user intent in interacting with device 1410. For example, an expected user activity may be to find and / or visit a restaurant for dinner, find parking for an event, create an invitation for an event,purchase an item, download and / or play media content, find hotels for an upcoming vacation, connect with a colleague for an upcoming professional meeting, search for gift ideas, create and / or share a photo album, and so forth. The context analysis model 1425 may use the captured app data, user data, and / or device data to identify candidate expected user activities. In some embodiments, the captured app data, user data, and / or device data may be represented as weighted signals that are aggregated to determine candidate expected user activities. In some embodiments, the candidate expected user activities may be ranked (e.g., based on user activities, preferences, etc.) and an expected user activity may be determined based on the ranking.[000150] Upon determining the expected user activity, prompt suggestion model 1430 may generate one or more suggested prompts tailored to the expected user activity. In some embodiments, prompt suggestion model 1430 may include a trained machine learning model to generate the one or more suggested prompts for a large language model (LLM), so that when the one or more suggested prompts are provided as input to the LLM, the LLM generates an output relevant to the expected user activity. Some embodiments involve detecting, by the user interface, a partial prompt input by a user. The generating of the one or more suggested prompts may be a suggested completion of the partial prompt. In some embodiments, at least one of the one or more suggested prompts may be a partial prompt. Such embodiments involve receiving, by the user interface, a user completion of the partial prompt. Such embodiments also involve providing, to the LLM, the user-completed prompt. For example, the suggested prompts can be partial prompts that may be customized by a user, user-initiated prompts that may be automatically completed, and / or entire prompts that may be provided to a user. In some embodiments, a suggested prompt can be presented, and the user may be provided with editing tools to edit the suggested prompt.[000151] In some embodiments, the user interface may include an editing interface configured to provide functionality that enables a user to edit the one or more suggested prompts. For example, the user may be able to enter a prompt, interact with the one or more suggested prompts, complete a suggested partial prompt, change the content of a prompt, change a modality of a prompt, and so forth. For example, the suggested prompt may be in audio format, and the user may convert the audio format to textual format and edit the textual formatted version of the prompt.[000152] In some embodiments, the functionality that enables the user to edit the one or more suggested prompts may include functionality that enables multimodal edits. For example, user interface 1410b may comprise an editing interface that facilitates editing of text, audio,video, images, etc. For example, when the prompt is in textual format, the user may edit the text. Also, for example, when the prompt is in image format, the user may be able to edit the image (e.g., using photo editing features such as cropping, changing a resolution, changing one or more image characteristics such as color, brightness, sharpness, contrast, etc.).[000153] On-device prompt engineering system 1405 can interface with one or more hardware components 1435 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), memory, input and / or output devices of device 1410). For example, on-device prompt engineering system 1405 can interface with one or more input devices, such as a keyboard, microphone(s) and / or camera(s) 1440. In some embodiments, the speech input by the user can be captured by microphone(s) 1440. Microphone(s) may be a part of device 1410, or may be an audio input device (e.g., a wired or wireless microphone) separate from device 1410, and communicatively linked to device 1410. Also, for example, on-device prompt engineering system 1405 can interface with an input method editor for text input. The text input may be by a digital or physical keyboard. For example, a touch screen of device 1410 can display a digital keyboard. In some embodiments, different digital keyboards corresponding to different languages may be displayed. Also, for example, different digital keyboards corresponding to different layouts, designs, and so forth, may be displayed. As illustrated, device 1410 may include user interface 1410b associated with the operating system, and keyboard, microphone(s) and / or camera(s) 1440 may be associated with user interface 1410b, user input mode 1410c, interface 1410d and so forth.[000154] FIG. 14B illustrates an overview of an example prompt suggestion system 1400B, in accordance with example embodiments. In some embodiments, application data 1450 may be received, by an operating system of a device, from a plurality of application programming interfaces (APIs) 1445 running on the device. In some embodiments, a trained machine learning model -based prompt generation system 1455 may generate one or more suggested prompts 1460 based on the application data 1450. At least one of the one or more suggested prompts 1460 may be presented by a user interface associated with the operating system.[000155] In some embodiments, the prompt suggestion system 1400B may detect a partial, and / or complete prompt entered by a user, or may detect user interaction with at least one of the one or more suggested prompts 1460. Accordingly, a user-selected prompt may be provided as input to an LLM 1470, causing LLM 1470 to generate an output 1475 relevant to an expected user activity.[000156] FIG. 15 is a diagram illustrating an example prompt suggestion system 1500, in accordance with example embodiments. A plurality of application programs (apps) are shown, however, this list is not exhaustive and is for purposes of illustration only. Generally speaking, the operating system may be able to obtain context from any app that interfaces with the operating system. The plurality of apps may include an electronic mail (email) app 1505, a short messaging system (SMS) app 1510, a weather app 1515, a web browser app 1520, a calendar app 1525, a phone app 1530, a cloud storage app 1535, a camera app 1540, a rideshare app 1545, a travel app 1550, and so forth.[000157] Generally speaking, prompt suggestion system 1500 may be configured with appropriate privacy controls that can be managed by a user. For example, a user may be able to opt-in to the prompt generation service, and select which, if any, apps may be used for purposes of the prompt generation. For example, the user may have opted not to share information from apps shown with dashed boundaries (e.g., SMS app 1510, phone app 1530, camera app 1540, etc.). Accordingly, although information about an app category may be available to prompt suggestion system 1500, the user may elect not to share more detailed information about the app.[000158] For example, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user’s social network, social contacts, or activities, a user’s preferences, or a user’s current location, and so forth), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personal data is removed, secured, encrypted, and so forth. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user. In addition to user controls, where user information is used to generate suggested prompts, such user information may be restricted to the user’s device, and not shared with a server, and / or with other devices. Also, for example, the user information may be deleted after use. For example, in the instance where the user consents to the use of such application data, the application data may be used to determine the suggested prompts, and the application data and / or the user input prompts may then be securely stored on the device, and not shared with other devices, servers, and so forth.[000159] Context capture model 1560 may share one or more aspects in common with context capture model 1420 of FIG. 14A. Context capture model 1560 may be configured to receive application data from the plurality of apps. Context analysis model 1570 may share one or more aspects in common with context analysis model 1425 of FIG. 14A. Context analysis model 1570 may be configured to analyze the received application data to determine an expected user activity. In some embodiments, context analysis model 1570 may include an app determination model 1575 to determine an app category. An app category may be, for example, a social media app, a news app, a business app, a gaming app, an education app, a lifestyle app, a finance app, a productivity app, a travel app, a weather app, a navigation / mapping app, and so forth. So apps may belong to more than one category. The app category may be indicative of a general user intent. In some embodiments, information indicating use of two or more app categories may provide an indication of an expected user activity.[000160] Prompt suggestion model 1580 may share one or more aspects in common with prompt suggestion model 1430 of FIG. 14A. Prompt suggestion model 1580 may be configured to apply a trained machine learning model to generate one or more suggested prompts for large language model (LLM) 1585. The one or more suggested prompts may be input to LLM 1585 to generate an output relevant to the determined expected user activity.[000161] Some embodiments involve determining, based on a respective category associated with the plurality of application programs, the expected user activity. For example, a travel app (e.g., a hotel app and an airline app) may indicate that the user is searching for places to stay at a travel destination. As another example, a calendar app and a food delivery app (e.g., a grocery store app) may indicate that the user is shopping to host a meal. Also, for example, a crafts app, a media content app, and a photo library app may indicate that the user is looking to create an invitation to an event, purchase a gift for an event, etc.[000162] In some embodiments, LLM 1585 may include one or more LLMs, and a particular LLM may be selected based on the expected user activity and / or the one or more suggested prompts. LLM 1585 may include, for example, language representation models that generate natural language outputs, multimodal models that can process and generate content in multiple modalities, such as text, audio, image, video, etc. Also, for example, LLM 1585 may include zero-shot models that are not trained based on task-specific training data and can perform multiple tasks. As another example, LLM 1585 may include fine-tuned models that are trained to perform specific tasks.[000163] LLM 1585 may be configured to provide different types of outputs such as Al generated content, learn from user activities, summarize content, compare content, search forinformation, provide explanations, and / or inspire a user to perform a new action. Accordingly, the one or more signals can be used to generate prompts (e.g., a type of modality, a type of query, etc.) that can be input to the LLM 1585 for a desired type of output.[000164] In some embodiments, a version of the LLM may reside on the device. For example, LLM 1585 can run locally on a mobile device. Such a smaller LLM 1585 can run faster and more efficiently. Once a prompt is generated, and a user selection of a prompt is received, the selected prompt may be provided to the on-device version of LLM 1585. In some embodiments, the selected prompt may be provided to a full version of LLM 1585 residing in the cloud.[000165] Applying LLM 1585 at the device has several advantages. For example, the application data does not leave the device (e.g., is not shared with a server, with other devices, and so forth), and is used for on-device processing. This has several advantages. For example, by restricting content to the device, any user information that may be used for generating suggested prompts is maintained with appropriate privacy and / or security controls. For example, the content may be encrypted, stored in a dedicated memory location on the device, and so forth. Such features also enhance the capabilities of the prompt suggestion system, as the suggested prompts can be suitably personalized to a particular user, user preferences, and so forth, based on user control of an extent of the personalization. Also, for example, the user can retain control over what information is collected about the user, how that information is used, and what information is provided to the user. On-device processing is advantageous as it enables faster processing, less power consumption, lower latency, and less data transmission over a network. Also, for example, in situations where a server may be unavailable, a network may be unavailable, a secured network may be unavailable, and / or network bandwidth may be limited, on-device processing can continue to generate customized prompt suggestions.[000166] Graphical user interface (GUI) 1590 may be associated with the operating system and may share one or more aspects in common with user interface 1410b. In some embodiments, GUI 1590 may include a prompt / LLM interface 1590. For example, in an input mode, prompt / LLM interface 1590 may include user input mode 1410c, and interface 1410d of FIG. 14A. Also, for example, in an output mode, prompt / LLM interface 1590 may be configured to provide the outputs generated by LLM 1585.[000167] In some embodiments, the one or more suggested prompts may be configured as multimodal inputs to the LLM. Some embodiments involve receiving, by GUI 1590, a user selection of a particular prompt of the one or more suggested prompts. The particular prompt may be in textual format, audio format (e.g., speech, music), image format, video format, andso forth. Such embodiments involve generating a feature embedding associated with the particular prompt. The feature embedding generally transforms the particular prompt into an input format compatible with an expected input for LLM 1585. Such embodiments also involve providing, to LLM 1585, the feature embedding. Such embodiments further involve receiving, from LLM 1585, and based on the feature embedding, a particular output associated with the particular prompt. Such embodiments also involve providing, by GUI 1590, the particular output.[000168] In some embodiments, particular prompt may be in a first modality, and the particular output may be in a second modality different from the first modality. For example, the particular prompt may be in textual format, and the particular output may be in image format. As another example, the particular prompt may be in speech format, and the particular output may be in textual format. Additional, and / or alternative embodiments are possible.Example Prompt Generation[000169] FIG. 16A illustrates example suggested prompts in the context of a shopping app 1600 A, in accordance with example embodiments. For example, a user interface may provide the user with one or more suggested prompts. In some embodiments, the one or more suggested prompts may belong to broad categories such as “create,” “explore,” and “understand.” In the shopping app, in the context of food and drink, an example suggested prompt under the “create” category may be “Thanksgiving meal menu” 1605 that can help the user search for a menu for a Thanksgiving meal. Another example of a suggested prompt under the “explore” category may be “delicious dessert recipes” 1610 that can help the user search for dessert recipes (e.g., that may complement a Thanksgiving meal). Also, for example, a suggested prompt under the “understand” category may be “what produce is in season” 1615 that can help the user plan a meal around seasonally available produce. In some embodiments, the expected user activity may be determined based on a variety of factors. For example, application data from a calendar app (e.g., calendar app 1525), an email app (e.g., email app 1505), and / or an SMS app (e.g., SMS app 1510) may indicate that the user is planning a Thanksgiving meal, and is interacting with the shopping app. Accordingly, the expected user activity may be determined to be “planning for a Thanksgiving meal.”[000170] FIG. 16B illustrates example suggested prompts in the context of a transportation app 1600B, in accordance with example embodiments. For example, a user interface may provide the user with one or more suggested prompts. In some embodiments, the one or more suggested prompts may belong to broad categories such as “create,” “explore,” and “understand.” The transportation app may be a rideshare app. An example suggestedprompt under the “create” category may be “weekend itinerary in Brazil” 1620 that can help the user plan a weekend for an upcoming trip to Brazil. Another example of a suggested prompt under the “explore” category may be “things to do in Brazil” 1625 that can help the user search for tourist attractions in Brazil. Also, for example, a suggested prompt under the “understand” category may be “travel safety precautions” 1630 that can help the user plan for appropriate safety and security issues during the trip. In some embodiments, the expected user activity may be determined based on a variety of factors. For example, application data from a calendar app (e.g., calendar app 1525), a travel app (e.g., travel app 1550), and / or a web browser app (e.g., web browser app 1520) may indicate that the user is planning a trip to Brazil (e.g., a business trip) and flight information from the travel app may indicate that the user will be spending a weekend. Accordingly, when the user interacts with the transportation app, the expected user activity may be determined to be “planning for a trip to Brazil.”[000171] FIG. 16C illustrates example suggested prompts in the context of a dating app 1600C, in accordance with example embodiments. For example, a user interface may provide the user with one or more suggested prompts. In some embodiments, the one or more suggested prompts may belong to broad categories such as “create,” “explore,” and “understand.” An example suggested prompt under the “create” category may be “a funny pickup line” 1635 that can help the user interact and initiate contact with another user of the dating app. Another example of a suggested prompt under the “explore” category may be “adventurous date ideas” 1640 that can help the user search for ideas to plan a date (e.g., while interacting over the dating app). Also, for example, a suggested prompt under the “understand” category may be “safe places to meet for a date” 1645 that can help the user plan for appropriate safety and security issues during an upcoming date, or to propose safe venues to another user of the dating app. In some embodiments, the expected user activity may be determined based on a variety of factors. For example, application data from a calendar app (e.g., calendar app 1525), a rideshare app (e.g., rideshare app 1545), a web browser app (e.g., web browser app 1520), and / or a dating app may indicate that the user is planning a date. Accordingly, when the user interacts with the dating app, the expected user activity may be determined to be “planning for a date.”[000172] FIG. 17A illustrates an example user interface for presenting suggested prompts, in accordance with example embodiments. For example, a prompt suggestion system may determine that a user is interacting with a carryout app 1705. The carryout app 1505 may provide various tabs to access services. For example, the tabs may include a ride tab 1710 to secure rides, and a delivery tab 1715 to order food deliveries. Generally speaking, the carryout app 1705 may include a user interface, such as a search box 1720, to receive queries from theuser. In some embodiments, the prompt suggestion system may present a prompt / LLM user interface 1730 to receive multimodal prompts from the user, and to present outputs from the LLM based on a selected prompt.[000173] In many situations, users may not be able to formulate a prompt, and / or not be able to craft a prompt that may be appropriate for the LLM to generate an output that would be relevant to the user’s intent. In some embodiments, the prompt suggestion system may present an “inspire me” feature 1725 to the user. For example, the “inspire me” feature 1725 may be overlaid on search box 1720. Upon selection of the “inspire me” feature 1725 by the user, the prompt suggestion system may present one or more suggested prompts to the user. In some embodiments, the “inspire me” feature 1725 may be presented upon determining that a threshold time has passed, and the user has not entered a prompt into the prompt / LLM user interface 1730. In some embodiments, the “inspire me” feature 1725 may be presented along with the prompt / LLM user interface 1730, and / or search box 1720.[000174] FIG. 17B illustrates another example user interface for presenting suggested prompts, in accordance with example embodiments. For example, a prompt suggestion system may determine that a user is interacting with a content generation app 1735. The content generation app 1735 may provide various tabs to access content. For example, the tabs may include a recent screenshot tab 1740, a sing a song tab 1745, a venue tab 1750, a theme tab 1755, an invitation tab 1760, and a stored video tab 1765. In some embodiments, the prompt suggestion system may present a prompt user interface 1780. The prompt suggestion system may detect user interaction with the prompt user interface 1780, for example, in the form of text input. For example, the user may enter a portion 1770 of a prompt, such as, “create an image for my wedding invitation inspired by...” and the prompt suggestion system may present one or more possible completions 1775 for the incomplete prompt.[000175] For example, the prompt suggestion system may determine, based on user interaction with the content generation app 1735, that the user has played a song in stored sing a song tab 1745. Accordingly, the possible completions 1775 may include “...my first dance song.” As another example, the prompt suggestion system may determine, based on user interaction with the content generation app 1735, that the user has viewed a video in stored video tab 1765. Accordingly, the possible completions 1775 may include “...our first prom together.” Also, for example, the prompt suggestion system may determine, based on user interaction with the content generation app 1735, that the user has played a song in stored sing a song tab 1745 and may want to include the couple’s favorite song in the invitation. Accordingly, the possible completions 1775 may include “...our favorite song.” Anotherexample may be that the prompt suggestion system may determine, based on user interaction with the content generation app 1735, that the user has viewed different genres in the theme tab 1755 and may want to include a song from their birth decade. Accordingly, the possible completions 1775 may include “...my birth decade.”[000176] In some embodiments, the prompt suggestion system may allow the user to select from the one or more possible completions 1775 to complete their input prompt 1770. In some embodiments, the prompt suggestion system may allow the user to edit the one or more suggested prompts.[000177] A prompt interface for the prompt suggestion system may be adapted to a flip and fold device. For example, suggested prompts may be provided in a “tabletop” mode. Also, for example, a “drag and drop feature” can be used to allow the user to interact with the prompt interface over the two sides of a flip and fold device. In some embodiments, the prompt interface may be provided as a widget on a cover screen.Prompt Suggestions in Foldable Devices[000178] FIG. 18A illustrates an example user interface for presenting suggested prompts in a foldable device 1800A, in accordance with example embodiments. For example, foldable device 1800 A may include a first foldable portion 1805 and a second foldable portion 1810. In some embodiments, the prompt suggestion system may be displayed in a tabletop design. For example, an app window may be provided in a first user interface 1815 associated with first foldable portion 1805, and one or more suggested prompts may be provided in a second user interface 1820 associated with second foldable portion 1810.[000179] FIG. 18B illustrates another example user interface for presenting suggested prompts in a foldable device 1800B, in accordance with example embodiments. For example, a foldable device 1800B may include a first foldable portion 1825 and a second foldable portion 1830. In some embodiments, an app window may be provided in a first user interface 1835 associated with first foldable portion 1825, and one or more suggested prompts may be provided in a second user interface 1840 associated with second foldable portion 1830. For example, second user interface 1840 may provide a suggested prompt “create an image of a Corgi on the beach in San Diego during sunset.” Upon selection by the user, an LLM may generate a first image 1845 and a second image 1850. In some embodiments, the user may be using a messaging app displayed in a messaging app window in the first user interface 1835 associated with the first foldable portion 1825. The first user interface 1835 may provide a “drag and drop” interface 1855 that allows a user to drag one or more of the first image 1845 or the second image 1850 displayed in the second foldable portion 1830 and drop them in the“drag and drop” interface 1855 in the first user interface 1835. In some embodiments, the one or more of the first image 1845 or the second image 1850 may subsequently be shared by the user via the messaging app displayed in the messaging app window.[000180] FIG. 18C illustrates another example user interface for presenting suggested prompts in a foldable device 1800C, in accordance with example embodiments. Foldable device 1800C is shown in a folded state with a cover display 1860. In some embodiments, an interactive prompt interface 1865 may be provided on the cover display 1860 configured with one or more features described herein.[000181] Another aspect of the prompt suggestion system may be to integrate it with a web browser (e.g., CHROME® on a desktop device, or a mobile web browsing application). For example, the prompt suggestion system could use the signals from user activities and OS context and add web context to generate personalized prompts. In some embodiments, the prompt suggestion system may use an address in the web browser to crawl the HTML code for the page and determine web context and / or content being viewed by the user. Also, for example, user inputs into a search box (e.g., Omnibox in CHROME®) and / or web browsing history may be used to determine context. In some embodiments, data from across different tabs of a web browser may be used to determine context.Prompt Suggestions in Web Browsers[000182] Generally speaking, the prompt suggestion system may use a combination of application data from the apps, device data from the OS, and / or browsing data from a web browser to generate the one or more suggested prompts.[000183] FIG. 19A illustrates an example user interface for presenting suggested prompts in a web browser window 1900 A, in accordance with example embodiments. In some embodiments, a user may enter a search query in a search field 1905. For example, the search query may be “create an invite for my 21stbirthday yoga party.” In some embodiments, a search suggestion 1910 may be provided for this search query, and selection of search suggestion 1910 may place search suggestion 1910 in search field 1905. In some embodiments, the prompt suggestion system may present a suggested prompt 1915, such as, for example, “create an invite.”[000184] FIG. 19B illustrates another example user interface for presenting suggested prompts in a web browser window 1900B, in accordance with example embodiments. Some embodiments involve determining a browsing context associated with a web browser. The at least one of the one or more suggested prompts may be based on the determined browsing context. In some embodiments, the browsing context may correspond to categories similar toan app category. For example, the browsing context may be transportation, navigation, media content, news, game, dating, food, crafts, messaging, calendar, email, and so forth.[000185] In some embodiments, the determining of the browsing context may be based on the content of a web page. For example, an analysis of an HTML code for a web page may indicate more detailed browsing context. For example, the content of the web page may relate to tourist attractions at a particular location. As another example, the content of the web page may relate to a search for airline fares to a location. Also, for example, the content of the web page may relate to an online shopping platform with specific items being displayed.[000186] In some embodiments, the determining of the browsing context may be based on content displayed across a plurality of tabs of the web browser. For example, a first tab may display a web page related to tourist attractions and a second tab may display a web page related to a web page for a search for airline fares to a location. Accordingly, the browsing context may be travel to a destination and tourist attractions for that destination. Accordingly, a suggested prompt may be “find tourist attractions at <location>.” In some embodiments, a third tab may display a web page related to an online shopping platform with specific clothing items being displayed. Accordingly, the browsing context may be travel to a destination, tourist attractions for that destination, and appropriate attire. Accordingly, a suggested prompt may be “shop for travel accessories for <location>.”[000187] In some embodiments, the determining of the browsing context may be based on an input into a search field of the web browser. For example, a user may be interacting with web browser window 1900B, and a tab 1920 may be open. For example, the user may be performing a search based on a search query 1925. For example, the search query 1925 may be for “best used SUVs under $30,000," and one or more search results may be displayed. For example, a first search result comprising first search result image 1930, a second search result comprising second search result image 1935, and so forth, may be displayed. In some embodiments, the data item suggestion system may display a prompt user interface 1940 (e.g., overlaid over the browsing window in tab 1920). The data item suggestion system may present one or more suggested prompts 1945 in prompt user interface 1940. The one or more suggested prompts 1945 may be based on the browsing context.[000188] In some embodiments, the determining of the browsing context may be based on a browsing history associated with the web browser. For example, the prompt suggestion system may present one or more additional suggested prompts 1950 that are tailored to the user’s search history. In some embodiments, prompt user interface 1940 and the one or more additional suggested prompts 1950 may be presented together. In some embodiments, theprompt user interface 1940 may be a first instance of the browsing window in tab 1920, and the one or more additional suggested prompts 1950 may be presented in a second instance of the browsing window in tab 1920.[000189] FIG. 19C illustrates another example user interface for presenting suggested prompts in a web browser window 1900C, in accordance with example embodiments. For example, one or more search results 1965 may be displayed in response to a search query 1960 entered by a user. In some embodiments, the search query may be related to a “Tokyo trip.” An analysis of the HTML code of the web browser window 1900C may indicate that the user is searching for hotels in Tokyo. Accordingly, the prompt suggestion system may determine that the expected user activity is to find hotels near the airport in Tokyo. Accordingly, a suggested prompt 1965 may be presented, such as, “show me which of these is closest to the airport.” In some embodiments, the prompt suggestion system may also obtain application data from a mapping app indicating a search for a location of Tokyo airport. Accordingly, the prompt suggestion system may determine the expected user activity with a higher level of confidence. Generally speaking, a confidence level associated with the determined expected user activity increases as the number of signals that are used to determine the expected user activity increases.Prompt Suggestions within Apps[000190] The application developer community may also be incentivized to provide additional data about their application programs. For example, the prompt suggestion system may facilitate user navigation of an app, content available within an app may be promoted by designing appropriate prompts for the user, in-app payments may be facilitated, and so forth. Also, for example, the app developers may be incentivized to integrate the prompt suggestion system with their apps to serve as app-specific Al assistants (e.g., chatbots) that may be tailored to a particular app. As more and more apps are integrated, the prompt suggestion system is able to utilize more complex signals to generate more accurate prompts. Also, for example, the prompt suggestion system may be configured to learn from user activities within an app to create a more personalized user experience.[000191] FIG. 20A illustrates an example prompt suggestion system for an application program, in accordance with example embodiments. A first instance 2005 of an app is illustrated (e.g., a crafts app). Generally, the first instance 2005 may include an app user interface 910 where a user may interact with features presented in the app. Arrow 1 indicates a transition to a second instance 2015 of the app, where a prompt user interface 2020 may be provided to enable a prompt suggestion system to suggest prompts based on expected useractivity in the app. As described herein, prompt user interface 2020 may be a multimodal interface configured to receive inputs in a first modality, and presents outputs in a second modality that may be different from the first modality. An example suggested prompt 2025 is shown, “@CraftsApp Help me find handmade gift items for a best friend that arrives by October 26.” As described herein, the prompt suggestion system may use application data from various apps such as a calendar app, an SMS app, a phone app, an email app, user search history, web browsing history, user interactions within the app, and so forth to determine that the expected user activity is to “find a gift for their best friend.” Also, the crafts app may be specializing in handicrafts, and the prompt suggestion system may combine the application data and the app data to generate the suggested prompt 2025.[000192] Generally, app developers and device users may have an ability to opt-in to share application data with the prompt suggestion system. In some embodiments, the prompt suggestion system may be configured to assist users to navigate the app. In some embodiments, the prompt suggestion system may be configured to generate the one or more suggested prompts to direct user activity to an application program of the plurality of application programs. As indicated by arrow 2, suggested prompt 2025 may direct the user back to the app user interface 910 so that the user may further interact with the features of the app. In some embodiments, where the application data from the app provides additional details about app interactions, the suggested prompts may be tailored to be more specific to an expected user activity. In addition to increasing user traffic for an app, such coordination between the prompt suggestion system and the app may save compute resources, time spent by a user, etc. by assisting the user in navigating the app and finding services in an efficient manner. For example, the prompt suggestion system may be configured to generate the one or more suggested prompts to facilitate user navigation of an application program of the plurality of application programs.[000193] FIG. 20B illustrates another example prompt suggestion system for an application program, in accordance with example embodiments. Device 2025 may display graphical user interface 2030. A plurality of icons may be displayed by graphical user interface 2030, each of the icons being associated with an application program. For example, the plurality of icons may be associated with App 1, App 2, App 3, ..., App 6. The graphical user interface 2030 may have features that enable an app developer to provide an overview statement 2035 for the app. For example, when a user interacts with App 5 (e.g., by touching the graphical user interface 2030 at a location of App 5, by hovering over the location of App 5, by sending a voice command, etc.), the overview statement 2035 for App 5 may be displayed,such as, for example, “Tell us about your App that can enable better suggestions for our users when invoking the Al Tool: ABC is an e-commerce company specializing in arts and crafts. The online store offers a variety of items in different categories, including gift items, jewelry, toys, etc.” Generally speaking, app developers may integrate their app with the prompt suggestion system instead of having to develop separate personal Al tools. In some embodiments, the prompt suggestion system may be trained to provide app-specific Al tools to help users interact with the app. For example, the prompt suggestion system may be configured as a chat agent for an application program of the plurality of application programs.Training Machine Learning Models for Generating Inferences / Predictions[000194] The data item suggestion system and / or the LLM may be trained based on a variety of techniques. For example, the data item suggestion system may be trained based on a history of user interactions with previous editable interfaces. For example, a user may interact with one or more editable interfaces in several ways. For example, the user may select a suggested data item from a list of data items, manually enter a data item, ignore suggested data items, and so forth. The data item suggestion system can be trained to learn from a history of such user interactions with editable interfaces to deliver more customized and / or personalized data items in the future. For example, training data may be generated based on the user interactions.[000195] Some embodiments involve training the machine learning model based on a history of user interactions with the plurality of application programs. For example, a user may prefer to enter certain data items within certain apps. As another example, a user may use a particular photo app to save images. As another example, the user may communicate with friends and / or family using one or more preferred apps.[000196] In some examples, a trained machine learning model can work on a variety of computing devices, including but not limited to, mobile computing devices (e.g., smart phones, tablet computers, cell phones, laptop computers), wearable devices, stationary computing devices (e.g., desktop computers), and server computing devices. A machine learning model, such as a convolutional neural network, can be trained using training data described herein to perform one or more aspects of prompt generation, autofill, and so forth, as described herein. In some examples, the neural network can be arranged as an encoder / decoder neural network. [000197] A trained machine learning model can process the editable interface to predict an output data comprising at least one suggested entry. In one example, (a copy of) the trained neural network can reside on a mobile computing device. In some embodiments, the mobile computing device can include a microphone that can capture input voice data, or a camera thatcan capture images and videos. Such input voice data, images, and videos may be used to determine prompts, and / or the at least one suggested entry.[000198] The prompt suggestion system and / or the LLM may be trained based on a variety of techniques. For example, the prompt suggestion system (e.g., machine learningbased prompt generation system 1455 of FIG. 14B) may be trained based on a history of user interactions with previously suggested prompts. For example, a user may interact with one or more suggested prompts in several ways. For example, the user may select a suggested prompt, complete a partial prompt, ignore suggested prompts and / or enter their own prompt (e.g., without selecting any of the suggested prompts). The prompt suggestion system can be trained to learn from a history of such user interactions with previously suggested prompts to deliver more customized and / or personalized prompts in the future. For example, training data may be generated based on the user interactions.[000199] Some embodiments involve training the machine learning model based on a history of user interactions with the plurality of application programs. For example, a user may prefer to perform certain activities using certain apps. As another example, a user may use a particular photo app to edit images, or a particular song app to edit songs. As another example, the user may perform certain daily routines using one or more preferred apps. Also, for example, when app navigation data is available, the prompt suggestion system can be trained to learn from how the user navigates apps.[000200] In some examples, a trained machine learning model (e.g., machine learningbased prompt generation system 155 of FIG. IB) can work on a variety of computing devices, including but not limited to, mobile computing devices (e.g., smart phones, tablet computers, cell phones, laptop computers), wearable devices, stationary computing devices e.g, desktop computers), and server computing devices. A machine learning model, such as a convolutional neural network, can be trained using training data described herein to perform one or more aspects of prompt generation as described herein. In some examples, the neural network can be arranged as an encoder / decoder neural network.[000201] A trained machine learning model can process the input data to predict an output data comprising one or more suggested prompts that are associated with the application data. In one example, (a copy of) the trained neural network can reside on a mobile computing device. In some embodiments, the mobile computing device can include a microphone that can capture input voice data, or a camera that can capture images and videos. Such input voice data, images, and videos may be used to determine suggested prompts.[000202] FIG. 21 shows diagram 2100 illustrating a training phase 2102 and an inference phase 2104 of trained machine learning model(s) 2132, in accordance with example embodiments. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine learning algorithm can be termed as a trained machine learning model. For example, FIG. 21 shows training phase 2102 where one or more machine learning algorithms 2120 are being trained on training data 2110 to become trained machine learning model(s) 2132. Then, during inference phase 2104, trained machine learning model(s) 2132 can receive input data 2130 and one or more inference / prediction requests 2140 (perhaps as part of input data 2130) and responsively provide as an output one or more inferences and / or predict! on(s) 2150.[000203] As such, trained machine learning model(s) 2132 can include one or more models of one or more machine learning algorithms 2120. Machine learning algorithm(s) 2120 may include but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). Machine learning algorithm(s) 2120 may be supervised or unsupervised and may implement any suitable combination of online and offline learning.[000204] In some examples, machine learning algorithm(s) 2120 and / or trained machine learning model(s) 2132 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 2120 and / or trained machine learning model(s) 2132. In some examples, trained machine learning model(s) 2132 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.[000205] During training phase 2102, machine learning algorithm(s) 2120 can be trained by providing at least training data 2110 as training input using unsupervised, supervised, semisupervised, and / or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of training data 2110 to machine learning algorithm(s) 2120 and machine learning algorithm(s) 2120 determining one or more output inferences based on the provided portion (or all) of training data 2110. Supervised learning involves providing a portionof training data 2110 to machine learning algorithm(s) 2120, with machine learning algorithm(s) 2120 determining one or more output inferences based on the provided portion of training data 2110, and the output inference(s) are either accepted or corrected based on correct results associated with training data 2110. In some examples, supervised learning of machine learning algorithm(s) 2120 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of machine learning algorithm(s) 2120.[000206] Semi-supervised learning involves having correct results for part, but not all, of training data 2110. During semi-supervised learning, supervised learning is used for a portion of training data 2110 having correct results, and unsupervised learning is used for a portion of training data 2110 not having correct results. Reinforcement learning involves machine learning algorithm(s) 2120 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 2120 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 2120 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machine learning algorithm(s) 2120 and / or trained machine learning model(s) 2132 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning. [000207] In some examples, machine learning algorithm(s) 2120 and / or trained machine learning model(s) 2132 can use transfer learning techniques. For example, transfer learning techniques can involve trained machine learning model(s) 2132 being pre-trained on one set of data and additionally trained using training data 2110. More particularly, machine learning algorithm(s) 2120 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 2104. Then, during training phase 2102, the pre-trained machine learning model can be additionally trained using training data 2110, where training data 2110 can be derived from kernel and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 2120 and / or the pre-trained machine learning model using training data 2110 of CDl’s data can be performed using either supervised or unsupervised learning. Once machine learning algorithm(s) 2120 and / or the pre-trained machine learning model has been trained onat least training data 2110, training phase 2102 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 2132.[000208] In particular, once training phase 2102 has been completed, trained machine learning model(s) 2132 can be provided to a computing device, if not already on the computing device. Inference phase 2104 can begin after trained machine learning model(s) 2132 are provided to computing device CD1.[000209] During inference phase 2104, trained machine learning model(s) 2132 can receive input data 2130 and generate and output one or more corresponding inferences and / or prediction(s) 2150 about input data 2130. As such, input data 2130 can be used as an input to trained machine learning model(s) 2132 for providing corresponding inference(s) and / or prediction(s) 2150 to kernel components and non-kernel components. For example, trained machine learning model(s) 2132 can generate inference(s) and / or predict! on(s) 2150 in response to one or more inference / prediction requests 2140. In some examples, trained machine learning model(s) 2132 can be executed by a portion of other software. For example, trained machine learning model(s) 2132 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. Input data 2130 can include data from computing device CD1 executing trained machine learning model(s) 2132 and / or input data from one or more computing devices other than CD1.[000210] For example, input data 2130 can include labeled editable interface based on a user interaction (e.g., selections of suggested data items, user input data items, user edits of data items, non-selection of data items) with previously presented data items. Input data 2130 for prompt engineering for an LLM can also include prompts labeled with context data, device data, browsing history, search history, and so forth. Other types of input data are possible as well. Also, for example, input data 2130 can include labeled prompts based on a user interaction (e.g., selections of prompts, user input prompts, user edits of prompts, user selection of prompt completions, non-selection of prompts) with previously presented prompts.[000211] Inference(s) and / or predict! on(s) 2150 can include output at least one suggested entry, numerical values, and / or other output data produced by trained machine learning model(s) 2132 operating on input data 2130 (and training data 2110). In some examples, trained machine learning model(s) 2132 can use output inference(s) and / or predict! on(s) 2150 as input feedback 2160. Trained machine learning model(s) 2132 can also rely on past inferences as inputs for generating new inferences.[000212] A neural network comprising a data item suggestion system or an LLM can be examples of machine learning algorithm(s) 2120. After training, the trained version of theneural network can be an example of trained machine learning model(s) 2132, such as, for example, a neural network comprising a trained data item suggestion system or a trained LLM. In this approach, an example of the one or more inference / prediction request(s) 2140 can be a request to predict a data item and a corresponding example of inferences and / or prediction(s) 2150 can be a predicted data item. As another example, an example of the one or more inference / prediction request(s) 2140 can be a request to predict a prompt and a corresponding example of inferences and / or predict! on(s) 2150 can be a predicted prompt.[000213] In some examples, one computing device CD SOLO can include the trained version of the neural network, perhaps after training. Then, computing device CD SOLO can receive a request to predict a data item and / or a prompt and use the trained version of the neural network to predict the data item and / or the prompt. In some examples, one computing device CD SOLO can include the trained version of the neural network, perhaps after training. Then, computing device CD SOLO can receive a request to predict a prompt, and use the trained version of the neural network to predict the prompt.[000214] In some examples, two or more computing devices CD CLI and CD SRV can be used to provide output images, e.g., a first computing device CD CLI can generate and send requests to predict a data item and / or a prompt to a second computing device CD SRV. Then, CD SRV can use the trained version of the neural network, to predict the data item and / or the prompt, and respond to the requests from CD CLI for the predicted data item and / or prompt. Then, upon reception of responses to the requests, CD CLI can provide the requested predicted prompt to the LLM, and / or the predicted data item (e.g., using a user interface and / or a display). [000215] In some examples, two or more computing devices CD CLI and CD SRV can be used to provide output images, e.g., a first computing device CD CLI can generate and send requests to predict a prompt to a second computing device CD SRV. Then, CD SRV can use the trained version of the neural network, to predict the prompt, and respond to the requests from CD CLI for the predicted prompt. Then, upon reception of responses to the requests, CD CLI can provide the requested predicted prompt (e.g., using a user interface and / or a display).Example Data Network[000216] FIG. 22 depicts a distributed computing architecture 2200, in accordance with example embodiments. Distributed computing architecture 2200 includes server devices 2208, 2210 that are configured to communicate, via network 2206, with programmable devices 2204a, 2204b, 2204c, 2204d, 2204e. Network 2206 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the publicInternet, or any other type of network configured to provide a communications path between networked computing devices. Network 2206 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.[000217] Although FIG. 22 only shows five programmable devices, distributed application architectures may serve tens, hundreds, or thousands of programmable devices. Moreover, programmable devices 2204a, 2204b, 2204c, 2204d, 2204e (or any additional programmable devices) may be any sort of computing device, such as a mobile computing device, desktop computer, wearable computing device, head-mountable device (HMD), network terminal, a mobile computing device, and so on. In some examples, such as illustrated by programmable devices 2204a, 2204b, 2204c, 2204e, programmable devices can be directly connected to network 2206. In other examples, such as illustrated by programmable device 2204d, programmable devices can be indirectly connected to network 2206 via an associated computing device, such as programmable device 2204c. In this example, programmable device 2204c can act as an associated computing device to pass electronic communications between programmable device 2204d and network 2206. In other examples, such as illustrated by programmable device 2204e, a computing device can be part of and / or inside a vehicle, such as a car, a truck, a bus, a boat or ship, an airplane, etc. In other examples not shown in FIG. 15, a programmable device can be both directly and indirectly connected to network 2206.[000218] Server devices 2208, 2210 can be configured to perform one or more services, as requested by programmable devices 2204a-2204e. For example, server device 2208 and / or 2210 can provide content to programmable devices 2204a-2204e. The content can include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content can include compressed and / or uncompressed content. The content can be encrypted and / or unencrypted. Other types of content are possible as well.[000219] As another example, server device 2208 and / or 2210 can provide programmable devices 2204a-2204e with access to software for database, search, computation, graphical, audio, video, World Wide Web / Internet utilization, and / or other functions. Many other examples of server devices are possible as well.Computing Device Architecture[000220] FIG. 23 is a block diagram of an example computing device 2300, in accordance with example embodiments. In particular, computing device 2300 shown in FIG. 23 can be configured to perform at least one function of and / or related to the machine learning models, method 2500, and / or method 2600.[000221] Computing device 2300 may include a user interface module 2301, a network communications module 2302, one or more processors 2303, data storage 2304, one or more camera(s) 2312, one or more sensors 2314, and power system 2316, all of which may be linked together via a system bus, network, or other connection mechanism 2305.[000222] User interface module 2301 can be operable to send data to and / or receive data from external user input / output devices. For example, user interface module 2301 can be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices. User interface module 2301 can also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRT), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, either now known or later developed. User interface module 2301 can also be configured to generate audible outputs, with devices such as a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface module 2301 can further be configured with one or more haptic devices that can generate haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 2300. In some examples, user interface module 2301 can be used to provide a graphical user interface (GUI) for utilizing computing device 2300, such as, for example, a graphical user interface of a mobile phone device.[000223] Network communications module 2302 can include one or more devices that provide one or more wireless interface(s) 2307 and / or one or more wireline interface(s) 2308 that are configurable to communicate via a network. Wireless interface(s) 2307 can include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other type of wireless transceiver configurable to communicate via a wireless network. Wireline interface(s) 2308 can include one or more wireline transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or a similar physical connection to a wireline network.[000224] In some examples, network communications module 2302 can be configured to provide reliable, secured, and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., guaranteed message delivery) can be provided, perhaps as part of a message header and / or footer (e.g.,packet / message sequencing information, encapsulation headers and / or footers, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communications can be made secure (e.g., be encoded or encrypted) and / or decry pted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), a Rivest-Shamir-Adelman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms can be used as well or in addition to those listed herein to secure (and then decry pt / decode) communications.[000225] One or more processors 2303 can include one or more general purpose processors, and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). One or more processors 2303 can be configured to execute computer- readable instructions 906 that are contained in data storage 2304 and / or other instructions as described herein.[000226] Data storage 2304 can include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of one or more processors 2303. The one or more computer-readable storage media can include volatile and / or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with at least one of one or more processors 2303. In some examples, data storage 2304 can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other examples, data storage 2304 can be implemented using two or more physical devices.[000227] Data storage 2304 can include computer-readable instructions 2306 and perhaps additional data. In some examples, data storage 2304 can include storage required to perform at least part of the herein-described methods, scenarios, and techniques and / or at least part of the functionality of the herein-described devices and networks. In some examples, data storage 2304 can include storage for a trained neural network model 2310 (e.g., trained machine learning model, trained LLM, etc.). In particular of these examples, computer-readable instructions 2306 can include instructions that, when executed by one or more processors 2303, enable computing device 2300 to provide for some or all of the functionality of trained neural network model 2310.[000228] In some examples, computing device 2300 can include one or more camera(s) 2312. Camera(s) 2312 can include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, camera(s) 2312 can generate image(s) of captured light. The one or more images can be one or more still images and / or one or more images utilized in video imagery. Camera(s) 2312 can capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.[000229] In some examples, computing device 2300 can include one or more sensors 2314. Sensors 2314 can be configured to measure conditions within computing device 2300 and / or conditions in an environment of computing device 2300 and provide data about these conditions. For example, sensors 2314 can include one or more of: (i) sensors for obtaining data about computing device 2300, such as, but not limited to, a thermometer for measuring a temperature of computing device 2300, a battery sensor for measuring power of one or more batteries of power system 2316, and / or other sensors measuring conditions of computing device 2300; (ii) an identification sensor to identify other objects and / or devices, such as, but not limited to, a Radio Frequency Identification (RFID) reader, proximity sensor, one-dimensional barcode reader, two-dimensional barcode (e.g., Quick Response (QR) code) reader, and a laser tracker, where the identification sensors can be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or object configured to be read and provide at least identifying information; (iii) sensors to measure locations and / or movements of computing device 2300, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser-displacement sensor, and a compass; (iv) an environmental sensor to obtain data indicative of an environment of computing device 2300, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a movement sensor, a microphone, a sound sensor, an ultrasound sensor and / or a smoke sensor; and / or (v) a force sensor to measure one or more forces (e.g., inertial forces and / or G-forces) acting about computing device 2300, such as, but not limited to one or more sensors that measure: forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or locations of the ZMPs. Many other examples of sensors 2314 are possible as well.[000230] Power system 2316 can include one or more batteries 2318 and / or one or more external power interfaces 2320 for providing electrical power to computing device 2300. Each battery of the one or more batteries 2318 can, when electrically coupled to the computingdevice 2300, act as a source of stored electrical power for computing device 2300. One or more batteries 2318 of power system 2316 can be configured to be portable. Some or all of one or more batteries 2318 can be readily removable from computing device 2300. In other examples, some or all of one or more batteries 2318 can be internal to computing device 2300 and so may not be readily removable from computing device 2300. Some or all of one or more batteries 2318 can be rechargeable. For example, a rechargeable battery can be recharged via a wired connection between the battery and another power supply, such as by one or more power supplies that are external to computing device 2300 and connected to computing device 2300 via the one or more external power interfaces. In other examples, some or all of one or more batteries 2318 can be non-rechargeable batteries.[000231] One or more external power interfaces 2320 of power system 2316 can include one or more wired-power interfaces, such as a USB cable and / or a power cord, that enable wired electrical power connections to one or more power supplies that are external to computing device 2300. One or more external power interfaces 2320 can include one or more wireless power interfaces, such as a Qi wireless charger, that enable wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection is established to an external power source using one or more external power interfaces 2320, computing device 2300 can draw electrical power from the external power source the established electrical power connection. In some examples, power system 2316 can include related sensors, such as battery sensors associated with the one or more batteries or other types of electrical power sensors.Cloud-Based Servers[000232] FIG. 24 depicts a cloud-based server system in accordance with an example embodiment. In FIG. 24, functionality of a neural network, and / or a computing device can be distributed among computing clusters 2409a, 2409b, and 2409c. Computing cluster 2409a can include one or more computing devices 2400a, cluster storage arrays 2410a, and cluster routers 2411a connected by a local cluster network 2412a. Similarly, computing cluster 2409b can include one or more computing devices 2400b, cluster storage arrays 2410b, and cluster routers 2411b connected by a local cluster network 2412b. Likewise, computing cluster 2409c can include one or more computing devices 2400c, cluster storage arrays 2410c, and cluster routers 2411c connected by a local cluster network 2412c.[000233] In some embodiments, computing clusters 2409a, 2409b, and 2409c can be a single computing device residing in a single computing center. In other embodiments, computing clusters 2409a, 2409b, and 2409c can include multiple computing devices in asingle computing center, or even multiple computing devices located in multiple computing centers located in diverse geographic locations. For example, FIG. 24 depicts each of computing clusters 2409a, 2409b, 2409c residing in different physical locations.[000234] In some embodiments, data and services at computing clusters 2409a, 2409b, 2409c can be encoded as computer readable information stored in non-transitory, tangible computer readable media (or computer readable storage media) and accessible by other computing devices. In some embodiments, computing clusters 2409a, 2409b, 2409c can be stored on a single disk drive or other tangible storage media or can be implemented on multiple disk drives or other tangible storage media located at one or more diverse geographic locations. [000235] In some embodiments, each of computing clusters 2409a, 2409b, and 2409c can have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. In other embodiments, however, each computing cluster can have different numbers of computing devices, different numbers of cluster storage arrays, and different numbers of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster can depend on the computing task or tasks assigned to each computing cluster.[000236] In computing cluster 2409a, for example, computing devices 2400a can be configured to perform various computing tasks of a neural network, and / or a computing device. In one embodiment, the various functionalities of a neural network, and / or a computing device can be distributed among one or more of computing devices 2400a, 2400b, and 2400c. Computing devices 2400b and 2400c in respective computing clusters 2409b and 2409c can be configured similarly to computing devices 2400a in computing cluster 2409a. On the other hand, in some embodiments, computing devices 2400a, 2400b, and 2400c can be configured to perform different functions.[000237] In some embodiments, computing tasks and stored data associated with a neural network, and / or a computing device can be distributed across computing devices 2400a, 2400b, and 2400c based at least in part on the processing requirements of a neural network, and / or a computing device, the processing capabilities of computing devices 2400a, 2400b, 2400c, the latency of the network links between the computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency, and / or other design goals of the overall system architecture.[000238] Cluster storage arrays 2410a, 2410b, 2410c of computing clusters 2409a, 2409b, and 2409c can be data storage arrays that include disk array controllers configured to manageread and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, can also be configured to manage backup or redundant copies of the data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.[000239] Similar to the manner in which the functions of a conditioned, axial selfattention based neural network, and / or a computing device can be distributed across computing devices 2400a, 2400b, 2400c of computing clusters 2409a, 2409b, 2409c, various active portions and / or backup portions of these components can be distributed across cluster storage arrays 2410a, 2410b, 2410c. For example, some cluster storage arrays can be configured to store one portion of the data of a first layer of a neural network, and / or a computing device, while other cluster storage arrays can store other portion(s) of data of second layer of a neural network, and / or a computing device. Also, for example, some cluster storage arrays can be configured to store the data of an encoder of a neural network, while other cluster storage arrays can store the data of a decoder of a neural network. Additionally, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.[000240] Cluster routers 2411a, 2411b, 2411c in computing clusters 2409a, 2409b, and 2409c can include networking equipment configured to provide internal and external communications for the computing clusters. For example, cluster routers 2411a in computing cluster 2409a can include one or more internet switching and routing devices configured to provide (i) local area network communications between computing devices 2400a and cluster storage arrays 2410a via local cluster network 2412a, and (ii) wide area network communications between computing cluster 2409a and computing clusters 2409b and 2409c via wide area network link 2413a to network 2206. Cluster routers 2411b and 2411c can include network equipment similar to cluster routers 2411a, and cluster routers 2411b and 2411c can perform similar networking functions for computing clusters 2409b and 2409b that cluster routers 2411a perform for computing cluster 2409a.[000241] In some embodiments, the configuration of cluster routers 2411a, 2411b, 2411c can be based at least in part on the data communication requirements of the computing devices and cluster storage arrays, the data communications capabilities of the network equipment in cluster routers 2411a, 2411b, 2411c, the latency and throughput of local cluster networks 2412a, 2412b, 2412c, the latency, throughput, and cost of wide area network links 2413a, 2413b, 2413c, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency and / or other design criteria of the moderation system architecture.Example Methods of Operation[000242] FIG. 25 is a flowchart of a method 2500 for proactive information retrieval, in accordance with example embodiments. Method 2500 can be executed by a computing device, such as computing device 2300. Method 2500 can begin at block 2510, where the method involves detecting a user interaction with an editable interface at a computing device.[000243] At block 2520, the method involves determining, based on the user interaction, a data item expected to be entered in the editable interface.[000244] At block 2530, the method involves generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggested entry associated with the data item from content accessible by the computing device.[000245] At block 2540, the method involves providing the generated prompt to the LLM to determine the at least one suggested entry associated with the data item.[000246] At block 2550, the method involves receiving, from the LLM, the at least one suggested entry.[000247] At block 2560, the method involves providing the at least one suggested entry at the editable interface.[000248] In some embodiments, the editable interface comprises one of a web form, a chat interface, a short messaging service (SMS) interface, or an email interface.[000249] In some embodiments, the generating of the prompt involves predicting, by the ML model, content relevant to the determining of the at least one suggested entry by the trained LLM. Such embodiments also involve providing, by the ML model, the predicted content to the trained LLM.[000250] In some embodiments, the ML model may be located at a web browsing application, and wherein the editable interface is a fillable web form.[000251] In some embodiments, the ML model may be located at an interactive user interface for another LLM.[000252] In some embodiments, the LLM may be a large multimodal model (LMM).[000253] In some embodiments, the editable interface may be configured for multimodal inputs, multimodal outputs, or both.[000254] In some embodiments, the generated prompt may be a multimodal prompt.[000255] In some embodiments, the editable interface may include a form with one or more fillable fields, and wherein the determining of the data item involves determiningrespective data values for the one or more fillable fields, wherein the determining is based on one or more of extracting forms, fields, or field labels.[000256] In some embodiments, the LLM may be trained to determine the at least one suggested entry based on crowdsourcing to predict form and field patterns.[000257] In some embodiments, the receiving of the at least one suggested entry involves receiving a ranked list of a plurality of suggested entries, and wherein the providing of the at least one suggested entry involves providing the ranked list of the plurality of suggested entries. [000258] Some embodiments involve receiving user selection from the ranked list of the plurality of suggested entries as the data item expected to be entered. Such embodiments involve entering the user selection as the data item expected to be entered.[000259] Some embodiments involve automatically entering the at least one suggested entry as the data item expected to be entered.[000260] In some embodiments, a version of the LLM may reside on the computing device.[000261] Some embodiments involve determining a browsing context associated with a web browser. The generating of the prompt may be based on the browsing context.[000262] In some embodiments, the determining of the browsing context may be based on a content of a web page.[000263] In some embodiments, the determining of the browsing context may be based on a browsing history associated with the web browser.[000264] In some embodiments, the determining of the browsing context may be based on content displayed across a plurality of tabs of the web browser.[000265] Some embodiments involve training the ML model based on a history of user interactions with a plurality of editable interfaces.[000266] Some embodiments involve training the ML model based on a history of user interactions with a plurality of application programs.[000267] In some embodiments, the trained machine learning model may be configured as a chat agent for an application program of the plurality of application programs.[000268] In some embodiments, the content accessible by the computing device may include content from one or more application programs on the computing device.[000269] In some embodiments, the content accessible by the computing device may include content from an operating system of the computing device.[000270] FIG. 26 is a flowchart of a method 2600 that exhibits improved computational efficiency, in accordance with example embodiments. Method 2600 can be executed by acomputing device, such as computing device 2300. Method 2600 can begin at block 2610, where the method involves receiving, by an operating system of a device, application data from a plurality of application programs running on the device.[000271] At block 2620, the method involves applying, by the operating system and based on the application data, a trained machine learning model to generate one or more suggested prompts for a large language model (LLM), wherein the one or more suggested prompts, upon being provided as input to the LLM, cause the LLM to generate an output relevant to an expected user activity.[000272] At block 2630, the method involves providing, by the operating system, at least one of the one or more suggested prompts to be presented by a user interface associated with the operating system.[000273] Some embodiments involve determining a context associated with the operating system. The at least one of the one or more suggested prompts may be based on the determined context.[000274] Some embodiments involve receiving, by the user interface, a user selection of a particular prompt of the one or more suggested prompts. Such embodiments involve generating a feature embedding associated with the particular prompt. Such embodiments also involve providing, to the LLM, the feature embedding. Such embodiments further involve receiving, from the LLM, and based on the feature embedding, a particular output associated with the particular prompt. Such embodiments also involve providing, by the user interface, the particular output.[000275] In some embodiments, particular prompt may be in a first modality, and the particular output may be in a second modality different from the first modality.[000276] Some embodiments involve determining, based on a respective category associated with the plurality of application programs, the expected user activity.[000277] In some embodiments, the one or more suggested prompts may be configured as multimodal inputs to the LLM.[000278] Some embodiments involve detecting, by the user interface, a partial prompt input by a user. The generating of the one or more suggested prompts may be a suggested completion of the partial prompt.[000279] In some embodiments, the at least one of the one or more suggested prompts may be a partial prompt. Such embodiments involve receiving, by the user interface, a user completion of the partial prompt. Such embodiments also involve providing, to the LLM, the user-completed prompt.[000280] In some embodiments, the user interface may include an editing interface configured to provide functionality that enables a user to edit the one or more suggested prompts.[000281] In some embodiments, the functionality that enables the user to edit the one or more suggested prompts may include functionality that enables multimodal edits.[000282] In some embodiments, a version of the LLM may reside on the device.[000283] Some embodiments involve determining a browsing context associated with a web browser. The at least one of the one or more suggested prompts may be based on the determined browsing context.[000284] In some embodiments, the determining of the browsing context may be based on a content of a web page.[000285] In some embodiments, the determining of the browsing context may be based on an input into a search field of the web browser.[000286] In some embodiments, the determining of the browsing context may be based on a browsing history associated with the web browser.[000287] In some embodiments, the determining of the browsing context may be based on content displayed across a plurality of tabs of the web browser.[000288] Some embodiments involve training the machine learning model based on a history of user interactions with previously suggested prompts.[000289] Some embodiments involve training the machine learning model based on a history of user interactions with the plurality of application programs.[000290] In some embodiments, the trained machine learning model may be configured as a chat agent for an application program of the plurality of application programs.[000291] In some embodiments, the trained machine learning model may be configured to generate the one or more suggested prompts to facilitate user navigation of an application program of the plurality of application programs.[000292] In some embodiments, the trained machine learning model may be configured to generate the one or more suggested prompts to direct user activity to an application program of the plurality of application programs.[000293] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those enumerated herein, will beapparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.[000294] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.[000295] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and / or communication may represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.[000296] A block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively, or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.[000297] The computer readable medium may also include non-transitory computer readable media such as non-transitory computer-readable media that stores data for short periods of time like register memory, processor cache, and random-access memory (RAM).The computer readable media may also include non-transitory computer readable media that stores program code and / or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.[000298] Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.[000299] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for explanatory purposes and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for proactive information retrieval, comprising: detecting a user interaction with an editable interface at a computing device; determining, based on the user interaction, a data item expected to be entered in the editable interface; generating, during the user interaction, a prompt to be provided to a large language model (LLM), wherein the prompt comprises one or more instructions that cause the LLM to determine at least one suggested entry associated with the data item from content accessible by the computing device; providing the generated prompt to the LLM to determine the at least one suggested entry associated with the data item; receiving, from the LLM, the at least one suggested entry; and providing the at least one suggested entry at the editable interface.

2. The computer-implemented method of claim 1, wherein the editable interface comprises one of a web form, a chat interface, a short messaging service (SMS) interface, or an email interface.

3. The computer-implemented method of claim 1, wherein the generating of the prompt further comprises: predicting, by a machine learning (ML) model, content relevant to the determining of the at least one suggested entry by the trained LLM; and providing, by the ML model, the predicted content to the trained LLM.

4. The computer-implemented method of claim 3, wherein the ML model is located at a web browsing application, and wherein the editable interface is a fillable web form.

5. The computer-implemented method of claim 3, wherein the ML model is located at an operating system of the computing device.

6. The computer-implemented method of claim 3, wherein the ML model is located at an interactive user interface for another LLM.

7. The computer-implemented method of claim 3, further comprising: training the ML model based on a history of user interactions with a plurality of editable interfaces.

8. The computer-implemented method of claim 3, further comprising: training the ML model based on a history of user interactions with a plurality of application programs.

9. The computer-implemented method of claim 8, wherein the trained ML model is configured as a chat agent for an application program of the plurality of application programs.

10. The computer-implemented method of claim 1, wherein the LLM is a large multimodal model (LMM).

11. The computer-implemented method of claim 1, wherein the editable interface is configured for multimodal inputs, multimodal outputs, or both.

12. The computer-implemented method of claim 1, wherein the generated prompt is a multimodal prompt.

13. The computer-implemented method of claim 1, wherein the editable interface comprises a form with one or more fillable fields, and wherein the determining of the data item comprises determining respective data values for the one or more fillable fields, wherein the determining is based on one or more of extracting forms, fields, or field labels.

14. The computer-implemented method of claim 1, wherein the LLM is trained to determine the at least one suggested entry based on crowdsourcing to predict form and field patterns.

15. The computer-implemented method of claim 1, wherein the receiving of the at least one suggested entry comprises receiving a ranked list of a plurality of suggested entries, andwherein the providing of the at least one suggested entry comprises providing the ranked list of the plurality of suggested entries.

16. The computer-implemented method of claim 15, further comprising: receiving user selection from the ranked list of the plurality of suggested entries as the data item expected to be entered; and entering the user selection as the data item expected to be entered.

17. The computer-implemented method of claim 1, further comprising: automatically entering the at least one suggested entry as the data item expected to be entered.

18. The computer-implemented method of claim 1, wherein a version of the LLM resides on the computing device.

19. The computer-implemented method of claim 1, further comprising: determining a browsing context associated with a web browser, and wherein the generating of the prompt is based on the browsing context.

20. The computer-implemented method of claim 19, wherein the determining of the browsing context is based on a content of a web page.

21. The computer-implemented method of claim 19, wherein the determining of the browsing context is based on a browsing history associated with the web browser.

22. The computer-implemented method of claim 19, wherein the determining of the browsing context is based on content displayed across a plurality of tabs of the web browser.

23. The computer-implemented method of claim 1, wherein the content accessible by the computing device comprises content from one or more application programs on the computing device.

24. The computer-implemented method of claim 1, wherein the content accessible by the computing device comprises content from an operating system of the computing device.

25. A computer-implemented method that exhibits improved computational efficiency, comprising: receiving, by an operating system of a device, application data from a plurality of application programs running on the device; applying, by the operating system and based on the application data, a trained machine learning model to generate one or more suggested prompts for a large language model (LLM), wherein the one or more suggested prompts, upon being provided as input to the LLM, cause the LLM to generate an output relevant to an expected user activity; and providing, by the operating system, at least one of the one or more suggested prompts to be presented by a user interface associated with the operating system.

26. The computer-implemented method of claim 25, further comprising: determining a context associated with the operating system, and wherein the at least one of the one or more suggested prompts is based on the determined context.

27. The computer-implemented method of claim 25, further comprising: receiving, by the user interface, a user selection of a particular prompt of the one or more suggested prompts; generating a feature embedding associated with the particular prompt; providing, to the LLM, the feature embedding; receiving, from the LLM and based on the feature embedding, a particular output associated with the particular prompt; and providing, by the user interface, the particular output.

28. The computer-implemented method of claim 27, wherein the particular prompt is in a first modality, and the particular output is in a second modality different from the first modality.

29. The computer-implemented method of claim 25, further comprising: determining, based on a respective category associated with the plurality of application programs, the expected user activity.

30. The computer-implemented method of claim 25, wherein the one or more suggested prompts are configured as multimodal inputs to the LLM.

31. The computer-implemented method of claim 25, further comprising: detecting, by the user interface, a partial prompt input by a user, and wherein the generating of the one or more suggested prompts is a suggested completion of the partial prompt.

32. The computer-implemented method of claim 25, wherein the at least one of the one or more suggested prompts is a partial prompt, and further comprising: receiving, by the user interface, a user completion of the partial prompt; and providing, to the LLM, the user-completed prompt.

33. The computer-implemented method of claim 25, wherein the user interface comprises an editing interface configured to provide functionality that enables a user to edit the one or more suggested prompts.

34. The computer-implemented method of claim 33, wherein the functionality that enables the user to edit the one or more suggested prompts comprises functionality that enables multimodal edits.

35. The computer-implemented method of claim 25, wherein a version of the LLM resides on the device.

36. The computer-implemented method of claim 25, further comprising: determining a browsing context associated with a web browser, and wherein the at least one of the one or more suggested prompts is based on the determined browsing context.

37. The computer-implemented method of claim 36, wherein the determining of the browsing context is based on a content of a web page.

38. The computer-implemented method of claim 36, wherein the determining of the browsing context is based on an input into a search field of the web browser.

39. The computer-implemented method of claim 36, wherein the determining of the browsing context is based on a browsing history associated with the web browser.

40. The computer-implemented method of claim 36, wherein the determining of the browsing context is based on content displayed across a plurality of tabs of the web browser.

41. The computer-implemented method of claim 25, further comprising: training the machine learning model based on a history of user interactions with previously suggested prompts.

42. The computer-implemented method of claim 41, further comprising: training the machine learning model based on a history of user interactions with the plurality of application programs.

43. The computer-implemented method of claim 25, wherein the trained machine learning model is configured as a chat agent for an application program of the plurality of application programs.

44. The computer-implemented method of claim 25, wherein the trained machine learning model is configured to generate the one or more suggested prompts to facilitate user navigation of an application program of the plurality of application programs.

45. The computer-implemented method of claim 25, wherein the trained machine learning model is configured to generate the one or more suggested prompts to direct user activity to an application program of the plurality of application programs.

46. A computing device, comprising: one or more processors; and data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions comprising the computer-implemented method of any one of claims 1-45.

47. The computing device of claim 46, wherein the computing device is a mobile device.

48. A computer program comprising instructions that, when executed by a computer, cause the computer to perform steps in accordance with the method of any one of claims 1-45.

49. An article of manufacture comprising one or more non-transitory computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions that comprise the computer-implemented method of any one of claims 1-45.

50. A system, comprising: means for carrying out the computer-implemented method of any one of claims 1-45.

Images