Context-aware on-device intelligence
On-device AI enhances LLMs by using embedding representations of structured entities to understand user intent, addressing the challenge of inaccurate intent interpretation and reducing resource consumption.
Patent Information
- Application Number
- PCT/US2024/029103
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-05-13
- Publication Date
- 2025-11-20
AI Technical Summary
Existing large language models (LLMs) fail to accurately interpret user intent, leading to operations or recommendations that are not applicable to the user or of no interest, and this often requires communication resources that consume bandwidth and power.
Implementing on-device artificial intelligence that leverages on-screen context and user interactions to understand user intent, using embedding representations of structured entities to enhance LLMs, enabling context-aware responses without relying on external communication resources.
Enhances LLMs by deriving user preferences and generating relevant responses, reducing bandwidth and power consumption, and providing accurate recommendations even in areas without network connectivity.
Smart Images

Figure US2024029103_20112025_PF_FP_ABST
Abstract
Description
CONTEXT-AWARE ON-DEVICE INTELLIGENCEBACKGROUND
[0001] Understanding user intent is key to crafting helpful, engaging experiences. Large language models (LLMs) enable natural language processing (NLP), which is one way to interpret user intent. While existing LLMs enable NLPs, they often fail to accurately interpret user intent, which can result in operations or recommendations that are not applicable to the user or in which the user has no interest.SUMMARY
[0002] The present document describes techniques associated with context-aware on-device intelligence. These techniques provide on-device large language model (LLM) intelligence that can use a context of a user device. The context is determined by collecting textual, audio, image, and video data from the user device and transforming the data into embedding representations. The embedding representations are used to enhance the capabilities of the LLM, enabling the LLM to generate more-relevant and contextually aware responses, such as mapping user journeys through different apps, predicting a user’s next action, and providing search predictions tailored to the user’s current context.
[0003] In an example, a method for context-aware on-device intelligence is disclosed. The method includes capturing on-screen raw data displayed via a display device of an electronic device to provide captured content. The method also includes, at an operating-system level of the electronic device, converting the captured content into structured entities. In addition, the method includes, at the operating-system level, generating embedding representations of the structured entities and storing the embedding representations and the structured entities in a context store for retrieval of information representing a user context.
[0004] This summary is provided to introduce simplified concepts of context-aware on- device intelligence, which are further described below in the Detailed Description. This summary’ is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.BRIEF DESCRIPTION OF DRAWINGS
[0005] The details of one or more aspects of context-aware on-device intelligence are described in this document with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:FIG. 1 illustrates an example electronic device in which aspects of context-aware on-device intelligence can be implemented;FIG. 2 illustrates an example implementation of the electronic device from FIG. 1 in more detail;FIG. 3 illustrates a block diagram of an example computing system in accordance with some implementations;FIG. 4 illustrates an example pipeline for context-aware on-device intelligence;FIG. 5 illustrates an example architecture of a memory Al, which can be implemented for context-aware on-device intelligence;FIG. 6 illustrates an example trainer for training a large language model (LLM) used for the context-aware on-device intelligence;FIG. 7 illustrates an example of a generic transformer;FIG. 8 illustrates an example transfonnation of an input tensor component in a language space;FIG. 9 illustrates an example of a fine-tuning (FT) trainer;FIG. 10 illustrates an example of prompt engineering;FIGs. 11 and 12 depict example methods for context-aware on-device intelligence;FIG. 13 illustrates an example wireless network device that can be implemented in accordance with one or more aspects of a context-aware on-device intelligence as described herein; andFIG. 14 illustrates an example system that includes an example device that implement aspects of context-aware on-device intelligence as described with reference to the previous FIGs. 1 to 13.DETAILED DESCRIPTIONOverview
[0006] Understanding user intent is a key element in creating helpful and engaging experiences. Many products leverage systems that learn user interests based on historical data (e.g., queries and interactions) and attempt to generate personalized results and recommendations based on that historical data. The leveraging of such systems highlights the value of accurately interpreting user intent through interactions of the user.
[0007] Large language models (LLMs) enable performance for some natural language processing (NLP) tasks. While existing LLMs enable NLPs, they often fail to accurately interpret user intent, which can result in operations or recommendations that are not applicable to the user or in which the user has no interest. Many devices can, using the techniques described herein, implement on-device artificial intelligence (Al) that leverages on-screen context and user interactions across different software applications (apps) to better understand user intent and user journeys. By analyzing on-screen context, the on-device techniques can derive or infer short-termand long-term user preferences while preserving privacy. The insights gained from this rich personal data can significantly enhance LLMs for different underlying tasks, such as mapping user journeys to understand how a user traverses different apps to reach their goals, anticipating actions to predict a next user action for greater convenience, and suggesting relevant queries to provide search predictions tailored to the user’s current context.
[0008] For example, consider a user engaging in a text conversation via a messaging app with a friend about different places they like to go hiking and the user, during the conversation, mentioning going to Santa Cruz. This text conversation can be captured by the system and used as context to assist an LLM in generating queries relevant to the user. Some queries for this example might include “what are some hiking areas near Santa Cruz?,” “what are some things I need in order to be prepared for such a hike?,” “what days have good weather for hiking near Santa Cruz?,” and so forth. When the user triggers an app with an LLM application programming interface (API) on the user device, the user can be presented with these prompts for user selection, and upon selection of one of the prompts, the app can provide the selected prompt to the LLM to generate replies. In this way, the system attempts to understand the user’s intent and help the user take the next step of their journey through different apps or through research (e.g., information gathering) of a topic of interest.
[0009] By the system (device) itself generating the prompts and providing the selected prompts to the on-device LLM, communication resources connecting the device to a server or the cloud are not used for carrying context information, prompt suggestions, and / or LLM responses. Refraining from using the communication resources saves bandwidth for use by other services, and it also reduces power usage on the device. It also allows the benefits to be experienced even in the absence of a communication network, for instance in remote areas, underground, etc., or when communication systems are overloaded or otherwise impaired.
[0010] While features and concepts of the described techniques for context-aware on- device intelligence can be implemented in any number of different environments, aspects are described in the context of the following examples.Example Systems and Apparatuses
[0011] FIG. 1 illustrates an example electronic device 100 in which aspects of context- aware on-device intelligence can be implemented. The example electronic device 100 is capable of implementing at least a portion of the context-aware on-device intelligence in accordance with the techniques described herein. The electronic device 100 is illustrated as including user-context data 102, a content-capture application-programming-interface (API) 104, one or more LLM modules 106, and a context store 108. The user-context data 102 can include various data, suchas explicit context 110, implicit context 112, short-term context 114, and long-term context 116. The user-context data can be classified into additional categories.
[0012] The term “context” refers to context of a user of a device. An example of context includes information surrounding a user’s interactions with the device, such as screenshot captures, text messaging conversations, phone calls, app usage, Internet browsing and activities, etc. Context can also include information surrounding the user that does not correspond to the user’s interactions with the device, such as environmental conditions (e.g., weather), location, date, time, etc. The user’s context provides valuable insights for personalization and feature enhancement. Further, this context can be classified as explicit or implicit and further categorized into short-term or long-term.
[0013] Explicit context 110 is data that the user intentionally and explicitly, via a user input, saves for later use. Examples of explicit context 110 include a screen capture initiated by the user, an image downloaded or saved to device memory' storage responsive to a user input, and text copied to clipboard memory.
[0014] Implicit context 112 is data that the user has not intentionally and explicitly saved, such as on-screen data captured by the system (e.g., through a content-capture API) for which the user has not confirmed via a user input to save for later use. Knowledge of the user’s context can be extracted from the raw screen data. Some examples of on-screen data include articles or streaming content (e.g., audio, video) across various apps, communication screens with friends or family (e.g., texts, emails, social media chats), notifications from different apps, user past search queries in different apps (e g., shopping apps, streaming media apps, search apps), and screenshots.
[0015] Short-term context 114 is context that spans a brief period of time and is relevant for immediate actions or responses. Some examples of short-term context 114 include current app usage, recent notifications (e.g., received in the last 10 minutes, 30 minutes, 60 minutes), ongoing conversations, currently displayed content, content displayed within the last X minutes (e.g., 10 minutes, 30 minutes, 60 minutes), and so forth.
[0016] Long-term context 116 is context that is not short-term context 114; for example, it encompasses a longer history of interactions, such as those that are useful for understanding trends and patterns. Some examples of long-term context 116 include frequently used apps, common communication topics, past search queries (e.g., more than one day ago, more than three days ago, more than one week ago, more than two weeks ago), long-standing interests, and so forth.
[0017] Additional categories of time-frame context can also be implemented. A mediumterm context, for example, is not short-term context and not long-term context but is contextbetween the short-term context 114 and the long-term context 1 16, such that the medium-term context does not overlap with either of the short-term context 114 or the long-term context 116. An example of medium-term context relates to user interactions wi th the electronic device 100 within the last 24 hours but longer than the short-term context 114. Some examples of mediumterm context include user-search queries, frequently used apps, conversation topics, and so forth.
[0018] The content-capture API 104 is an API configured to capture various context data, including on-screen data (e.g., data displayed via a display device of the electronic device 100) while the user is interacting with the electronic device 100. For example, the content-capture API 104 can extract knowledge from raw screen data, which can be used by the techniques to determine the user context and infer a user intent.
[0019] The LLM module(s) 106 is a generative Al model that can use the user-context data 102 captured by the content-capture API 104 as input and then output representations of different elements in the data. Example representations can include embedding representations (e.g., embeddings 118) and entity representations (e.g., entities 120). The embedding representations include vector representations (e g., vectors of floating-point numbers) of the data, rather than keyword representations. Embeddings can support semantic search and multimodal use cases. In multimodal cases, multimodal encoders can be used to encode the embeddings. The entity representations include entities extracted from the user context. For example, if a user is viewing a product page on a shopping webpage, specific descriptors related to the product (e.g., price, brand, model) can be stored as entities 120.
[0020] The LLM module(s) 106 can also generate one or more queries 122, for example, by using the embeddings 118 and the entities 120 as input to generate a final query’ . The final query can be provided, through another API, to an application running on the electronic device 100, such as an application that runs a multimodal Al model to generate outputs based on a set of inputs (e.g., text, images, videos, audio). The final query’ represents a prediction of the user’s intent and the user's next anticipated query’ or search. The final query is provided to help the user along their journey. In some examples, a plurality of final queries can be provided for user selection. The user selection of one of the queries can confirm the prediction of the user’s intent as it correlates to the selected query. In an example, a first LLM extracts elements from the raw screen data to generate entities 120 and embeddings 118 for storage in the context store 108, and a second LLM calls into the context store 108 for relevant information and generates queries for the application.
[0021] The context store 108 is a storage device on the electronic device 100 where the user-context data 102 is stored as the embeddings 118 and the entities 120. The context store 108stores a variety of data to provide rich insights to the user’s context. Further details are disclosed below.
[0022] FIG. 2 illustrates an example implementation 200 of the electronic device 100 from FIG. 1 in more detail. The electronic device 100 may include additional components and interfaces omitted from FIG. 1 for the sake of clarity.
[0023] The electronic device 100 can be a variety of consumer electronic devices (e.g., computer systems). As non-limiting examples, the electronic device 100 can be a mobile phone 100-1, atablet device 100-2, a laptop computer 100-3, a desktop computer 100-4, a computerized watch 100-5. a wearable computer 100-6, a video game controller 100-7, and the like.
[0024] The electronic device 100 includes one or more radio frequency (RF) transceiver(s) 202 for communicating over wireless networks (e.g., the cloud). The electronic device 100 can tune the RF transceiver(s) 202 and supporting circuitry (e.g., antennas, front-end modules, amplifiers) to one or more frequency bands defined by various communication standards.
[0025] The electronic device 100 includes one or more integrated circuits 204. The integrated circuits 204 can include, as non-limiting examples, a central processing unit, a graphics processing unit, or a tensor processing unit. A central processing unit generally executes commands and processes needed for the electronic device 100 and an operating system (OS). A graphics processing unit performs operations to display graphics of the electronic device 100 and can perform other specific computational tasks. A tensor processing unit generally performs symbolic match operations in neural-network machine-learning applications. The integrated circuits 204 can be single-core or multiple-core processors. The integrated circuits 204 can include other components (not illustrated), including communication units (e.g., modems), input / output controllers, and system interfaces.
[0026] One or more sensors 206 include sensors or other circuitry operably coupled to at least one integrated circuit 204. The sensors 206 monitor the process, voltage, and temperature of the integrated circuit 204 to assist in evaluating operating conditions of the integrated circuit 204. The sensors 206 can also monitor other aspects and states of the integrated circuit 204. The integrated circuit 204 can utilize outputs of the sensors 206 to monitor its chip state. Other modules can also use the sensor outputs to adjust the system voltage of the integrated circuit 204.
[0027] The electronic device 100 also includes an image processing system 208 that can perform various image processing operations as discussed herein, including pre-processing operations, post-processing operations, and various processing operations therebetween. For example, the image processing system 208 may analyze image data to identify objects, classify objects, identify activities, and store processed image data.
[0028] The electronic device 100 also includes computer-readable storage media (CRM) 210. The CRM 210 is a suitable storage device (e.g., random-access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NVRAM), read-only memory' (ROM), Flash memory’) to store device data of the electronic device 100. The device data can include the operating system 212, one or more applications 214 of the electronic device 100. user data, and multimedia data. The operating system 212 generally manages hardware and software resources (such as the applications 214) of the electronic device 100 and provides common sen-ices for the applications 214. The operating system 212 and the applications 214 are generally executable by the integrated circuits 204 (e.g., a central processing unit) to enable communications and user interaction with the electronic device 100. The CRM 210 can also include the usercontext data 102 and the LLM modules 106 described above, such that the user-context data 102 and the LLM modules 106 are stored locally on the electronic device 100. The techniques for context-aware on-device intelligence can be performed at the OS level, which is a level in a computing system hierarchy that runs system software. The OS level acts as an intermediary' between application programs and the computer hardware by managing the hardware and software resources of the electronic device 100, providing a user interface, and running application programs (e.g., the applications 214). The application programs interact with the operating system 212 by making requests for services through a defined API. The user can interact with the operating system 212 through a user interface, such as a command-line interface (CLI) or a graphical user interface (GUI).
[0029] The electronic device 100 also includes a display device 216 configured to display media content. The display device 216 can include a touch-sensitive display capable of receiving touch input from a user via a finger of the user, a stylus, or another touch-input mechanism. Alternatively, the display device 216 may include a non-touch display such that touches to the display are not detected or sensed as input. The display device 216 can render any suitable displayable content, including images, videos, color designs, icons, text, and so on. Further details of these and other components and features are provided below.
[0030] FIG. 3 illustrates a block diagram of an example computing system 300 in accordance with some implementations. The computing system 300 can be included or implemented in the electronic device 100 of FIG. 1. According to some examples, the computing system 300 may be comprised of various modules, such as an input module 302. The input module 302 can include a user input 304 and a model input 306. The computing system 300 can also include a processing module 308 and a memory' module 310. The memory' module 310 can include an LLM module 312 (e.g., the LLM module(s) 106). In some examples, the memory' module 310 can also include a natural language processing (NLP) module (not shown), acomparison module 314, and an output module 316. These modules may include or be included in various physical systems, such as electronic devices, non-volatile computer-readable media, user input devices (e.g., computer mouse, keyboard, digitizer, touch panel), and similar devices.
[0031] FIG. 4 illustrates an example pipeline 400 for context-aware on-device intelligence. As mentioned, the content-capture API 104 captures content displayed via the display device 216. Such content (e g., user-context data) can include text 402, audio 404, and image 406 content. The text 402 content can include textual data displayed via the display device 216, such as conversations (e.g., via a messaging app), past queries (e.g., search queries entered by the user via an app), articles rendered for the user to read, notifications, text from screenshots, etc. The text 402 content can include essential messages and discussions. Another example of the text 402 content is text-based web activity including screens viewed by the user across multiple web platforms (Chrome, Instagram, Amazon, etc.).
[0032] The audio 404 content can include transcripts displayed via the display device 216, including audio transcripts of phone calls, audio playback, microphone input, audio extracted from video, on-device automatic-speech recognition (ASR) to convert the audio to text, etc. The image 406 content can include, for example, image data from screenshots, information about an image discussed via a chat conversation, etc.
[0033] The text 402, the audio 404, and the image 406 content are used as input to an LLM 408 (e.g., the LLM module 106, the LLM module 312) for parsing, understanding, and indexing the information. In particular, the LLM 408 extracts information from the text 402, the audio 404, and the image 406 content and generates entities and embeddings for storage in the context store 108.
[0034] Due to their capability to understand the content, LLM models can encode the context store 108 and provide strong reasoning capabilities. Such encoding enables a context retriever 410 to more quickly search for and retrieve particular information in the context store 108.
[0035] FIG. 5 illustrates an example architecture 500 of a memory Al, which can be implemented for context-aware on-device intelligence. The example architecture 500 enables capture of relevant memories from user journeys in-app, extraction of the relevant memories into relevant entities, and utilization of the relevant entities and current on-device context to answer queries for clients using the data.
[0036] The example architecture 500 includes a memory extraction path and a memory lookup path. The memory extraction path is a path in which memory' is captured from various ingestion APIs (e.g., content-capture API), batched, and sent to an LLM to distill important chunks. The memory lookup path is a path in which memory is utilized for various downstreamclients. The memory lookup path processes the client query, context, and relevant memory entries to generate an appropriate output. To support the memory lookup path, the memory Al includes various layers, including memory ingestion 502, data batching 504, Al memory extraction 506, memory storage and retrieval 508, and search 510.
[0037] The memory ingestion 502 layer is the start of the memory extraction path. Here, data from ingestion APIs and notifications is captured. There are two major types of memory' ingestion: explicit and implicit. In explicit memory', the user actively selects to add items to the memory'. In implicit memory', data is captured automatically without a user actively adding items to the memory. Example sources of implicit memory include content capture, notification / conversation (e.g., via push notifications), ASR (audio / video) via sound played on-device in videos or audios that the user watches or listens to, and a shared directory via directories (e.g., screenshot, documents folder) to which the LLM 408 has access. In an example, a steady stream of data is collected in the memory ingestion 502 phase.
[0038] Not all data captured is sent to the LLM. The data batching 504 layer batches outputs and shortens the outputs before sending the shortened batch outputs to the LLM. Accordingly, the data batching 504 provides a mechanism to break up the steady stream of data collected in the memory ingestion 502 phase. This helps to avoid overloading the model and to maintain system health. The data batching 504 is a data parsing layer that includes a filter, transformation, cleansing, and bandwidth throttling.
[0039] The Al memory extraction 506 layer is a shared layer between the LLM 408 and an Al app. The Al memory extraction 506 layer converts raw natural language data into usable data, for example, using an Al app with a generative Al model and a sentence embedder. On the memory extraction side, raw data is converted into structured entities. On the memory' lookup side, natural language query data plus memory is converted to structured output and sent back to clients. The Al extraction 506 layer includes two major components: (i) a fine-tuned Al model that takes the batched output from above and converts the batched output into memory entities; and (ii) an encoder (e.g., sentence encoder) that generates embedding vectors for the entities generated above.
[0040] The memory' storage and retrieval 508 layer is an efficient database for storing extracted memory data, along with their embedding vectors. The indexed memory' entries along with their embedding vectors are stored in a vector database for efficient retrieval.
[0041] The search 510 layer performs a final search and retrieval (RAG), taking a user prompt from clients, retrieving relevant memories, and sending the relevant memories to an LLM to generate final suggestions. The search 510 layer includes a fine-tuned Al model tuned specifically for taking a prompt that includes memory entities and generating output suggestions.In the search 510 phase, the user prompt is first run through an encoder to generate a corresponding embedding vector. Then, the top N matching memory entities are obtained by, for example, performing a nearest-neighbor search based on cosine similarity (e.g., search for matching memory entities that are within a similarity threshold of the embedding vector of the user prompt). The top N results can be post-filtered to the top K entities via scoring. The post-filtered results can be combined with a client-specific final prompt to send to an LLM to generate the final suggestions. Then, the LLM output is converted into an API that client applications can read. After the Al memory retrieves the relevant client-side output, the data is piped to the client (e.g., application) in a privacy-preserving manner.Large Language Models
[0042] Generally, large language models (LLMs) are a class of artificial intelligence (Al). LLMs are trained on enormous amounts of data to provide foundational capabilities, which can be used and reused, often through fine-tuning for particular applications and tasks. Other software applications, in contrast, are often built and trained in a specific domain for each use case. In this way, LLMs are considered a type of foundational model.
[0043] Some LLMs use a machine-learned (ML) computer model that is able to parse language and provide context-aware outputs, such as to mimic a human response. This mimic of a human response is ty pi cal ly to a prompt, such as from a user asking a question. The question “ask how to get to the train station in French,” for example, can be used as a prompt by which an LLM provides a translation service, namely a human response in French to the English prompt. An example of context-aware on-device intelligence may use first context information captured from a textual conversation between the user and a friend via a messaging app that related the user’s desire to buy a new watch, second context information captured from a website the user visited that includes an article about fishing gear, and third context information from a calendar app of the user about a scheduled fishing trip to McCall, Idaho. Using the first, second, and third context information as input, the LLM may generate a prompt in the form of “what are good waterproof watches for fishing?” or “what stores near McCall, Idaho sell watches that can get wet?”
[0044] FIGs. 6-10 illustrate different aspects of LLMs, such as atrained model that can be implemented on the electronic device 100 or implemented at a remote server and accessed by the electronic device 100 via an API request. Examples of the LLMs described in FIGs. 6-10 include the LLM module 106, the LLM module 312, and the LLM 408.
[0045] By way of example, consider FIG. 6, which illustrates a trainer 600 by which to train an LLM used for context-aware on-device intelligence. The trainer 600 receives trainingdata as training inputs, such as an input 602. This training data may be of many different types, such as textual conversations, queries, articles, transcripts of telephone calls, locally stored images, and screenshots images with text. In this example, the training input 602 is a phrase, though it may instead be a word, a long text passage (e.g., a book, article, or web-page), an image with descriptive text, or any other data containing comprehensible text. In a process called “tokenization,” the trainer 600 breaks the training input 602 into tokens, marked as tokens 602-1, 602-2, 602-3, and 602-4. Here the training input 602 has a missing next word, marked as a blank 602-5. The goal of the trainer 600 is to predict the blank 602-5.
[0046] The trainer 600 encodes the tokens (602-1, 602-2. etc.) into an input tensor x 604 through a mapping procedure. For instance, the token “It” 602-1 is mapped to a first component 604-1 of the input tensor x 604, the token “’s” is mapped to a second component 604-2 of the input tensor x 604, the token “character” is mapped to a third component 604-3 of the input tensor x 604, and the token “ize” is mapped to a fourth component 604-4 of the input tensor x 604. Though the tokens “It” 604-1 and “’s” 604-2 are shown as two portions of the word “It’s,” other mapping schemes exist, such as mapping based on discrete words or phonemes. In some instances, an ML model or an ML component of the trainer 600 performs the tokenization and / or mapping of the training input 602 into the input tensor x 604 (e.g., a feature-extracting convolutional neural network (CNN)). The mapping of the tokenized training input 602 into the input tensor x 604 may involve a lookup table, which maps each possible token (e.g., 602-1, 602- 2, etc.) to a known tensor object in a language space of the training data.
[0047] A transformer 606 takes the input tensor x 604 as an input, with the goal of predicting the blank 602-5 by transforming the input tensor x 604 into a transformed tensor x' 608. The transformation process is mathematically represented as follows:Tx — x' Eq. (1)
[0048] T in Eq. (1) represents the transformer 606. The transformed tensor x' 608 includes components 608-1, 608-2, 608-3, 608-4, and 608-5. The component 608-1 is a transformation of the component 604-1 by the transformer 606 (similar for component pairs 608-2 / 604-2, 608- 3 / 604-3, and 608-4 / 604-4). The component 608-5 corresponds to the blank 602-5, and thus the component 608-5 is a prediction for the blank 604-5. The final transformed tensor x' 608 component 608-5 is derived as part of the transformation process in addition to the contextualization of the components 604-1 through 604-4.
[0049] Inputs such as the input tensor x 604 and / or the training input 602 generally include multiple tokens. For instance, the training input 602 includes the tokens 602-1 through 602-4. The trainer 600 converts a single training input (e.g., the training input 602) into multiple training inputs. For example, by removing the token 602-4, the blank 602-5 “shifts left” as the traininginput 602 calls for the trainer 600 to predict the token 602-4, thus creating a new training input from the original training input 602. As the value for the token 602-4 is known in this example, the new' input is a labeled input, which allows it to be used by a supervised ML training algorithm (it should be noted that such an input is also able to be used by an unsupervised ML training algorithm). In this way. a single text containing multiple tokens (e.g.. a book, a research paper, etc.) is used as multiple training inputs for the trainer 600.
[0050] An example transformer 700 is shown in FIG. 7. The transformer 700 is used to both contextualize words within an input prompt and to predict a next word from the input prompt. The transformer 700 includes an attention 702, a multi-head attention 704, a multi-layer perceptron (MLP) 706, and an output 708. The attention 702 and the multi-head attention 704 take tokenized and mapped inputs (e.g., the input tensor x 604 of FIG. 6) and contextualize them, similarly to how a speaker of a language w ill understand the meaning of a word in the context of the rest of the words in a sentence in which the word is found. The contextualization employs known correlation operators, such as matrix operators, normalization, dot product operators, etc. to characterize correlations in components (e.g., the components 604-1, 604-2, etc. of FIG. 6) of an input tensor (e.g., the input tensor x 604 of FIG. 6). The output 708 is a prediction based on a transformation of the input tensor (e g., the component 608-5 of the transformed tensor x' 608 of FIG. 6).
[0051] In some examples, the transformer 700 also includes ML components, such as the MLP 706. The MLP 706 is used to derive additional correlations in the input tensor (e.g., the input tensor x 604 of FIG. 6) apart from the contextualization done by the attention 702 and / or the multi-head attention 704. Although a single MLP 706 is pictured as a final step of the transformer 700 before the output 708, this need not be the case. Multiple MLPs may be included, the components may be connected in a different order (e.g., an additional MLP in between the attention 702 and the multi -head attention 704, a CNN prior to the attention 702, etc.), or other components not listed may also be included. Components of the transformer 700 may run in series, in parallel, or in a combination of both series and parallel operations. Additionally or alternately, the transformer 700 may have a different makeup of attention components. There may be multiple or no single attentions (e.g., the attention 702), multiple or no multi-head attentions (e.g., the multi-head attention 704) or any combination of attentions and multi-head attentions. In general, the transformer 700 is not limited to the specific makeup shown in FIG. 7 and may have different components than those pictured.
[0052] FIG. 8 illustrates an example transformation 800 in a language space 802-1 of an input tensor component 804-1 (e.g., the token 602-1 of the input tensor x 604 of FIG. 6). The language space 802-1 is a multi-dimensional mathematical space, which includes specificlanguage components codified as tensors within the multi-dimensional mathematical space. The term "tensor" is a mathematical object of any dimensionality, including scalar, vector, and matrix quantities. The language space 802-1 is therefore a mathematical vocabulary, and mapped tokens (e.g., token 602-1 of FIG. 6) are tokens that have been translated into the mathematical vocabulary. For ease of illustration, the language space 802-1 is shown in FIG. 8 as a three-dimensional space with orthogonal basis vectors ilsi2, and i3. However, this should not be seen as limiting. In general, the language space 802-1 has the dimensionality of the mapped tokens from an input tensor. For example, the input tensor x 604 of FIG. 6, whose tensor components 604-1 through 604-4 each contain n members, correspond to an ^-dimensional language space.
[0053] The input tensor component 804-1 is plotted in the language space 802-1, shown in FIG. 8 as a vector in three-dimensional space. In some examples, the plotting is the product of a lookup table, a CNN feature mapping, or any other mapping from the token into the language space 802-1. The input tensor component 804-1 is transformed by the transfonnation 800. Consider a language space 802-2, identical to the language space 802-1, and an input tensor component 804-2, identical to the input tensor component 804-1. The transformation 800 is based on transformation operators 806 and 808 and performed by a transformer (e.g., the transformer 700 of FIG. 7). The transformation operators 806 and 808 are illustrated as vector addition operators, resulting in a remapped tensor 810.
[0054] As an illustration of this transfonnation, let the input tensor component 804-2 represent a mapped (e.g., translated into the mathematical vocabulary' of the language space 802- 2) token of “rodent’' and let the transfonnation operators 806 and 808 be generated by contextualizing mapped tokens “large” and "eared” from an input prompt, which includes the phrase “large-eared rodent.” Contextualizing is defined as characterizing the correlations between “rodent,” “large,” and “eared” from the input prompt (e.g., the input 602 of FIG. 6) in a way that corresponds with how a speaker of the input prompt’s language would understand the word “rodent” as it appears in the input prompt along with “large” and “eared.” In this illustration, the transformed tensor 810 maps to an area of the language space 802-2 containing the word “chinchilla.”
[0055] Though the transformation of the input tensor component 804-2 to the transformed tensor 810 has been shown as two transformations using the transformation operators 806 and 808, this should not be seen as limiting. Any number of transformation operations may be employed, including more than two or a single transformation operation. Transformation operators (e.g., the transformation operator 806) may also take forms other than vector / tensor addition, such as multiplication (e.g., scaling, matrix multiplication, dot product, cross product, tensor product), normalization, orthogonalization, or any combination of these or othertransformation operations known to a person of ordinary skill in the art. Thus, the transformation operators 806 and 808 of FIG. 8 are meant to be illustrative, not limiting.
[0056] Sophisticated LLMs may have a very large number of trained parameters, with modem LLMs boasting hundreds of billions of parameters in their employed models. Because of this, it is often advantageous not to train an LLM from scratch but rather to fine-tune an already- trained model. To give a human analogy, this is much like teaching a person who already knows a language how to write in the American Psychological Association (“APA”) style. It takes an entire upbringing for the person to master the language, but a single university course suffices to learn the APA writing style.
[0057] By way of example, consider FIG. 9, which illustrates a fine-tuning (FT) trainer 900. The FT trainer 900 takes an LLM 902 (e.g., an LLM previously trained by7the trainer 600) and FT data 904 as training inputs. The FT trainer 900 includes an FT training module 906 and a final output in the form of an FT LLM 908.
[0058] The FT training module 906 includes a language space 906-1. Though the language space 906-1 has here been illustrated as a three-dimensional space with orthogonal basis axes ii, i2, and i3, this should not be seen as limiting. In general, the language space 906-1 may have the dimensionality of its input data, such as an input tensor 906-2. The language space 906- 1, in some examples, may be of a lower dimension than a language space used to train the LLM 902 (e g., the language space 802-1 of FIG. 8). The input tensor 906-2 (e.g., the component 604- 2 of FIG. 6) is mapped into the language space 906-1 and transformed by transformation operators 906-3 and 906-4. The transformation performed by the transformation operators 906-3 and 906-4 is by way of the LLM 902, giving a resultant transformed tensor 906-5 (e.g., the transformation 800 of FIG. 8). In some examples (not pictured), the transformation process may include a change of basis into another language space or a mapping into a smaller language space.
[0059] The FT training module 906 includes an additional transformation operator 906-6, resulting in a final tensor 906-7. This may be represented mathematically as follows:F(Tx) = x' + 8x Eq. (2)
[0060] T, as in Eq. (1), is the transformation performed by7the LLM 902 (e.g., an LLM trained by the trainer 600 of FIG. 6), / ' is the additional transformation operator 906-6, x' is the transformed tensor 906-5 (e.g., the remapped tensor 810 of FIG. 8), and 8x is a perturbation component. The perturbation component 6x is shown in the language space 906-1 as the additional transformation operator 906-6, giving the final tensor 906-7. The perturbation component 8x is based on the FT data 904. The final tensor 906-7 is the mapped representation of x' + 8x. This gives the FT LLM 908 all of the capabilities of the LLM 902 with the additional context of the FT data 904.
[0061] Consider, as before, an example of the input tensor 906-2 representing a mapping of a token “rodent” into the language space 906-1, the transformation operator 906-3 representing a mapping of a token “large” into the language space 906-1, and the transformation operator 906- 4 representing a mapping of a token “eared” into the language space 906-1. The transformed tensor 906-5 represents a region of the language space 906-1 containing “chinchilla.” In another example, the input tensor 906-2 represents an encoded token of “hiking,” the transformation element 906-3 represents a contextualized encoded token of “trails,” and the transformation element 906-4 represents a contextualized encoded token of “Santa Cruz.”
[0062] By way of example, suppose a veterinarian office wishes to fine-tune (FT) train the LLM 902 to associate types of medications with animal types. Training the entire LLM 902 is logistically prohibitive, and a different LLM containing the correlation between the types of medication and the animal types may be unavailable. Instead, the veterinarian office employs the FT trainer 900. where the FT data 904 includes the correlations between the types of medication and the animal types. The additional transformation operator 906-6 associates a chinchilla with a specific type of medication. Thus, the final tensor 906-7 contains the type of medication corresponding with, for example, rodents, which is not found in the LLM 902. Continuing with the hiking example, the additional transformation operation 906-6 may be an association with hiking and equipment suggested for safe hiking. Thus, the final tensor 906-7 may contain suggested equipment for the user to take on a hike that may not otherwise be generated using the LLM 902 without such additional contextual information.
[0063] While the FT training module 906 has been shown as performing the additional transformation operator 906-6 as a single operation, this need not be the case. The additional transformation operation 906-6 may include several operations, a change in dimensionality, or any other of a number of transformations known to a person of ordinary skill in the art. Further, the FT training module 906 may include other components not pictured, such as MLP or CNN components, additional feature mapping, etc. The illustration of a single operation for the additional transformation operator 906-6 is shown for brevity and to aid in understanding, not to express a limitation on the functionality of the FT training module 906.
[0064] In some instances, it is desirable to guide the output of an LLM without fine-tuning the LLM. For example, by including context information about a user's intent or interest, a programmer may wish to add the functionality of an already -trained LLM to an application via an API call, including all of the up-to-date functionality of the LLM with no additional training or upkeep needed from the programmer. In such instances, the only avenue to guide the output of the LLM is the input prompt. Tailoring the input prompt to obtain a desired output is known as prompt engineering.
[0065] By way of example, consider FIG. 10, which illustrates prompt engineering 1000. An entry prompts is an attempt to obtain a desired result B using an LLM (e.g., the LLM 902 of FIG. 9). The LLM includes different computational avenues leading to results (“paths’") based on a form of the entry’ prompt ^. There may exist, at least in concept, an ideal path 1002 leading from the entry prompt^ to the desired result B in the most-efficient manner possible. There are also other paths, such as a false path 1004 leading to an undesired result !), an intermediate paths 1006 leading to an intermediary’ results C, another intermediate path 1008 leading from the intermediate result C to the desired result B, and an inefficient path 1010 leading from the entry' prompt^ to the desired result B. Many paths may exist, limited only by a scope of the LLM and a scope of the entry’ prompts. The ideal path may be expressed mathematically as follows:
[0066] S' in Eq. (3) represents the ideal path 1002 and L represents the entry prompt^, which is characterized by a language space cp (e.g., the language space 802-1 of FIG. 8) and a function 9(<p) for the path propagation in the language space (p. It may be difficult to distinguish the various paths 1004-1010 from the ideal path 1002. In order to reach or reasonably approximate the ideal path 1002, variations are made to the entry prompt £ as follows:
[0067] S in Eq. (4) represents a path, which deviates from the ideal path 1002 (S) by the function ^ (p). where 27(e) is characterized by the parameter e. During the prompt engineering 1000, various iterations of the entry7prompt A are input into the LLM in an attempt to find an acceptable path. By way of example, a form of the entry prompt A does not reach the desired result B (e.g., the false path 1004) and is discarded. In another example, another form of the entry7prompt^ gives the intermediate path 1006, arriving at the intermediate result C, and a subsequent prompt gives the intermediate path 1008 from the intermediate result C to the desired result B. Though this reaches the desired result B, multiple steps are taken, which is less efficient than a single, direct prompt. In another example, another form of the entry7prompts gives the inefficient path 1010, which arrives at the desired result B. In another example, another form of the entry7prompt A gives the ideal path 1002 (or an acceptable approximation) to the desired result B. Characterization of variations in the form of the entry prompt 4 may be expressed mathematically as follows:
[0068] Eq. (5) illustrates the variation of the deviation path S by the parameter e, with p being a maximum acceptable value. In an ideal scenario, for instance, ip = 0. Consider the example of tasking the LLM to list day-hiking trails near Santa Cruz with the entry prompt A ofthe form '‘list some hiking areas near Santa Cruz.” Suppose the LLM generates a list of hiking trails that are for experienced backpackers, tens of miles long, with steep elevation climbs, but the user has little experience backpacking and is only interested in a day hike. The generated list of more-extreme hiking trails from the entry prompt A is then analogous to the LLM taking the false path 1004, with the list of more-extreme hiking trails represented by the undesired result / ), which is not an acceptable path for the prompt engineering 1000.
[0069] Consider another example of the above form of the prompt A, but now suppose there is a follow-up step of entering another prompt of the form “generate a list of day -hiking trails near Santa Cruz,” resulting in a list of hiking trails that can be hiked in a single day. The other prompt can be generated from context information including the amount of time scheduled for the hike on the user’s calendar app, or a screenshot the user saved that includes a packing list having light equipment more suitable for a day hike. With the initial list of more-extreme hiking trails represented by the intermediate result C, the resultant list of day -hiking trails is represented by the desired result B. This process is represented by the additive paths 1006+ 1008. Though the desired result B is reached, it is possible that the deviation is still above the maximum acceptable value ip. which may be due to a computation cost, an inability to enter multiple prompts, or other limitations.
[0070] Consider an example of the entry prompt A having the form “list day -hike areas near Santa Cruz,” with this form of the entry prompts being a result of the prompt engineering 1000 process (in aspects, gained through the knowledge of unsuccessful forms of the entry prompt A, such as by the previous form of the entry prompt A generating the list of more-extreme hiking trails). With this form of the entry prompt A, the LLM returns a list of day -hike trails near Santa Cruz. Suppose the list of day-hike trails includes trails with steep elevation climbs that are not deemed acceptable. This again represents too high of a deviation from the maximum acceptable value ip. This is illustrated by the inefficient path 1010. Suppose the entry7prompts is, based on this result, reformatted to the fonn “make a list of day-hike trails for less-experienced hikers nearSanta Cruz.” The generated list of hiking trails now have a value under the maximumacceptable value ip, which is represented by the ideal path 1002. The prompt engineering 1000 leverages the failed paths from all of the forms of the input prompt A to arrive at an acceptably efficient and succinct prompt, which results in the desired result B.
[0071] Each of the forms of the entry prompt A in the previous several examples may be used to train the application agent to parse the entry prompt A in order to reformat it in a way that is in line with an intent of the entry prompt A. The prompt engineering 1000 leverages the failed paths from all of the forms of the entry prompt A to arrive at an acceptably efficient and succinctprompt, which results in the desired result B. In some examples, the parsing of the entry prompt A in order to determine the intent may be done by the LLM. The LLM may also be used in the prompt engineering 1000 to generate additional forms of the entry prompt A, to find follow-up prompts, to classify intents, or to perform other aspects of the prompt engineering 1000.Example Methods
[0072] FIGs. 11 and 12 depict example methods 1100 and 1200, respectively, for context- aware on-device intelligence. The method 1100 can be performed by the electronic device 100. At least some of the method 1100 can be performed by operating-system level components of the electronic device 100. The method 1100 is shown as a set of blocks that specify operations performed but are not necessarily limited to the order or combinations shown for performing the operations by the respective blocks. Further, any of one or more of the operations may be repeated, combined, reorganized, or linked to provide a wide array of additional and / or alternate methods. In portions of the following discussion, reference may be made to the example electronic device 100 of FIG. 1 or to entities or processes as detailed in Figs. 2-10, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities operating on one device.
[0073] FIG. 11 depicts a method 1100 for implementing context-aware on-device intelligence. At 1102, on-screen raw data displayed via a display device of an electronic device is captured to provide captured content. For example, text, audio, and image data displayed via the display device 216 of the electronic device 100 is captured.
[0074] At 1104, the captured content is converted, at the OS level of the electronic device, into structured entities. For example, the captured content (e.g., user-context data 102) is transformed into entities 120 by the LLM module 106.
[0075] At 1106, embedding representations of the structured entities are generated, at the OS level of the electronic device. For example, the LLM module 106 generates the embeddings 118 corresponding to the entities 120.
[0076] At 1108, the embedding representations and the structured entities are stored, at the OS level, in a context store for retrieval of information representing a user context. For example, the embeddings 118 and entities 120 are stored in the context store 108 of the electronic device 100.
[0077] At 1110, the information representing the user context is categorized into a plurality of time-frame categories based on a duration of time that the embedding representations and the structured entities are stored in the context store. For example, a timestamp can be associated with the captured content when captured, with the embedding representations whengenerated, or with the structured entities when generated. When the information representing the user context (e.g., the embedding representations and the structured entities) is retrieved, the timestamp can be used to determine a time relevance of the information by categorizing the information into time-frame categories, such as the short-term context 114 or the long-term context 1 16.
[0078] Continuing in FIG. 12, at 1202, a user prompt is received. For example, the user query' can be received via an app running on the electronic device 100, such as an Al app that interfaces with an LLM (e.g., using an API).
[0079] At 1204, the user prompt is converted, at the OS level, into an embedding vector. For example, the user prompt is converted into an embedding representation using the LLM module 106, the LLM module 312, or the LLM 408.
[0080] At 1206, a subset of embedding representations that are within a similarity' threshold to the embedding vector of the user prompt are determined, at the OS level. For example, the top N matching memory entities can be obtained by, for example, performing a nearest-neighbor search based on cosine similarity'.
[0081] At 1208, a set of final suggestions is determined, at the OS level, based on the user prompt combined with the determined subset of the embedding representations. For example, the top N results can be post-filtered to the top K entities via scoring.
[0082] At 1210, the set of final suggestions is provided via a display device for user selection as an LLM input. For example, the final suggestions (e.g., queries 122) can be provided to the user via the display device 216, such as through an Al app (e.g., an app with an API for interfacing with an LLM).
[0083] At 1212, context data corresponding to a user-selected suggestion of the set of final suggestions is retrieved. For example, responsive to a user input selecting one of the final suggestions, the electronic device 100 can retrieve particular user-context data 102, such as particular entities 120, that correspond to the user-selected final suggestion.
[0084] At 1214, the retrieved context data is appended to the user-selected suggestion as extra context for the LLM input. For example, the electronic device 100 can append the retrieved user-context data 102 to the user-selected final suggestion for the LLM input to provide additional context information for the LLM to enhance the capabilities of the LLM.Example Environments and Devices
[0085] FIG. 13 illustrates an example wireless network device 1300 (e.g., the electronic device 100) that can be implemented in accordance w ith one or more aspects of context-aw are on- device intelligence as described herein. The device 1300 can be integrated with electroniccircuitry, microprocessors, memory, input / output (I / O) logic control, communication interfaces and components, and other hardware, firmware, and / or software to implement the device 1300 in ahome areanetwork. Further, the wireless network device 1300 can be implemented with various components, such as with any number and combination of different components as further described with reference to the example device shown in FIG. 13.
[0086] In this example, the wireless network device 1300 includes a low-power microprocessor 1302 and a high-power microprocessor 1304 (e.g., microcontrollers or digital signal processors) that process executable instructions. The device 1300 also includes an inputoutput (VO) logic control 1306 (e.g., to include electronic circuitry). The microprocessors can include components of an integrated circuit, a programmable logic device, a logic device formed using one or more semiconductors, and other implementations in silicon and / or hardware, such as a processor and memoij' system implemented as a system-on-chip (SoC). Alternatively or in addition, the device 1300 can be implemented with any one or combination of software, hardware, firmware, or fixed logic circuitry that may be implemented with processing and control circuits. The low-power microprocessor 1302 and the high-power microprocessor 1304 can also support one or more different device functionalities of the device 1300. For example, the high-power microprocessor 1304 may execute computationally intensive operations, whereas the low-power microprocessor 1302 may manage less complex processes such as detecting a hazard or temperature from one or more sensors 1308. The low-power microprocessor 1302 may also wake or initialize the high-power microprocessor 1304 for computationally intensive processes.
[0087] The one or more sensors 1308 can be implemented to detect various properties such as acceleration, temperature, humidity, water, supplied power, proximity, external motion, device motion, sound signals, ultrasound signals, light signals, fire, smoke, carbon monoxide, global-positioning-satellite (GPS) signals, radio frequency (RF), other electromagnetic signals or fields, or the like. As such, the sensors 1308 may include any one or a combination of temperature sensors, humidity sensors, hazard-related sensors, other environmental sensors, accelerometers, microphones, optical sensors up to and including cameras (e.g.. charged coupled-device or video cameras), active or passive radiation sensors, GPS receivers, and radio frequency identification detectors. In implementations, the wireless network device 1300 may include one or more primary sensors, as well as one or more secondary sensors, such as primary' sensors that sense data central to the core operation of the device (e.g.. sensing a temperature in a thermostat or sensing smoke in a smoke detector), while the secondary sensors may sense other types of data (e.g., motion, light or sound), which can be used for energy -efficiency objectives or automation objectives.
[0088] The wireless netw ork device 1300 includes a memory' device controller 1310 and a memory device 1312, such as any type of a nonvolatile memory and / or other suitable electronicdata storage device. The wireless network device 1300 can also include various firmware and / or software, such as an operating system 1314 that is maintained as computer-executable instructions by the memory and executed by a microprocessor. The device software may also include one or more applications 1316 (e.g., applications 214) that implement various functionalities of the wireless network device 1300. The wireless network device 1300 also includes a device interface 1318 to interface wi th another device or peripheral component and includes an integrated data bus 1320 that couples the various components of the wireless network device 1300 for data communication between the components. The data bus 1320 in the wireless network device 1300 may also be implemented as any one of or a combination of different bus structures and / or bus architectures.
[0089] The device interface 1318 may receive input from a user and / or provide information to the user (e.g., as a user interface), and a received input can be used to determine a setting. The device interface 1318 may also include mechanical or virtual components that respond to a user input. For example, the user can mechanically move a sliding or rotatable component, or motion along a touchpad may be detected, and such motions may correspond to a setting adjustment of the device 1300. Physical and virtual movable user-interface components can allow the user to set a setting along a portion of an apparent continuum. The device interface 1318 may also receive inputs from any number of peripherals, such as buttons, a keypad, a switch, a microphone, and an imager (e.g., a camera device).
[0090] The wireless network device 1300 can include network interfaces 1322, such as a home area network interface for communication with other wireless network devices in a home area network, and an external network interface for network communication, such as via the Internet. The wireless network device 1300 also includes wireless radio systems 1324 for wireless communication with other wireless network devices via the home area network interface and for multiple, different wireless communications systems. The wireless radio systems 1324 may include Wi-Fi, Bluetooth™, Mobile Broadband, BLE, and / or point-to-point IEEE 1302.16.4. Each of the different radio systems can include a radio device, antenna, and chipset that is implemented for a particular wireless communications technology. The wireless network device 1300 also includes a powder source 1326, such as a battery7and / or a cable to connect the device 1300 to line voltage. An AC power source may also be used to charge the battery of the device 1300.
[0091] FIG. 14 illustrates an example system 1400 that includes an example device 1402 that implements aspects of context-aware on-device intelligence as described with reference to the previous FIGs. 1 to 13. The example device 1402 can be implemented as the wireless network device 1300 (in FIG. 13). the electronic device 100 (in FIG. 1). or other device as described herein.The example device 1402 may be any type of computing device, client device, mobile phone, tablet, communication device, entertainment device, gaming device, media playback device, and / or another type of device. Further, the example device 1402 may be implemented as any other type of wireless network device that is configured for communication over a network, such as a thermostat, hazard detector, camera, lighting unit, commissioning device, router, border router, joiner router, joining device, end device, leader, access point, and / or other wireless network devices.
[0092] The device 1402 includes communication devices 1404 that enable wired and / or wireless communication of device data 1406, such as data that is communicated between devices in a home area network, data that is being received, data scheduled for broadcast, data packets of the data, data that is synchronized between the devices, etc. The device data 1406 can include any type of communication data, as well as audio, video, and / or image data that is generated by applications executing on the device 1402. The communication devices 1404 can also include transceivers for cellular phone communication and / or for network data communication.
[0093] The device 1402 also includes input / output (I / O) interfaces 1408, such as data network interfaces that provide connection and / or communication links between the device 1402, data networks (e.g., a home area network, external network, etc.), and other devices. The I / O interfaces 1408 can be used to couple the device 1402 to any type of components, peripherals, and / or accessory devices. The I / O interfaces 1408 also include data input ports via which any type of data, media content, and / or inputs can be received, such as user inputs to the device 1402, as well as any type of communication data, as well as audio, video, and / or image data received from any content and / or data source.
[0094] The device 1402 includes aprocessing system 1410 (e.g., integrated circuit(s) 204) that may be implemented at least partially in hardware, such as with any type of microprocessors, controllers, and the like that process executable instructions. The processing system 1410 can include components of an integrated circuit, a programmable logic device, a logic device fonned using one or more semiconductors, and other implementations in silicon and / or hardware, such as at least one processor and at least one memory system implemented as a system-on-chip (SoC). Alternatively or in addition, the device 1402 can be implemented with any one or combination of software, hardware, firmware, or fixed logic circuitry that may be implemented with processing and control circuits. The device 1402 may further include any type of a system bus or other data and command transfer system that couples the various components within the device 1402. A system bus can include any one or combination of different bus structures and architectures, as well as control and data lines.
[0095] The device 1402 also includes computer-readable storage memory 1412 (e.g., CRM 210), such as data storage devices that can be accessed by a computing device and that provide persistent storage of data and executable instructions (e.g., software applications, modules, programs, functions, and the like). The computer-readable storage memory described herein excludes propagating signals. Examples of computer-readable storage memory 1412 include volatile memory and non-volatile memory, fixed and removable media devices, and any suitable memory device or electronic data storage that maintains data for computing device access. The computer-readable storage memoiy 1412 can include various implementations of random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable readonly memoiy7(EPROM), electrically erasable programmable read-only memory (EEPROM), and other types of storage memory7in various memory7device configurations.
[0096] The computer-readable storage memory 1412 provides storage of the device data 1406 and various device applications 1414 (e g., applications 214), such as an operating system (e.g., operating system 212) that is maintained as a software application with the computer- readable storage memory 1412 and executed by the processing system 1410. The device applications 1414 may also include a device manager, such as any form of a control application, a software application, a signal processing and control module, code that is native to a particular device, a hardware abstraction layer for a particular device, and so on.
[0097] The device 1402 also includes an audio and / or video system 1416 that generates audio data for an audio device 1418 and / or generates display data for a display device 1420 (e.g., display device 216). The audio device 1418 and / or the display device 1420 include any devices that process, display, and / or otherwise render audio, video, display, and / or image data, such as the image content of a digital photo. In implementations, the audio device 1418 and / or the display device 1420 are integrated components of the example device 1402. Alternatively, the audio device 1418 and / or the display device 1420 are external, peripheral components to the example device 1402. In aspects, at least part of the techniques described for context-aware on-device intelligence may be implemented in a distributed system, such as over a '“cloud” 1422 (e.g.. cloud 114) in a platform 1424. The cloud 1422 includes and / or is representative of the platform 1424 for services 1426 and / or resources 1428.
[0098] The platform 1424 abstracts underlying functionality7of hardware, such as server devices (e.g., included in the services 1426) and / or software resources (e.g., included as the resources 1428), and connects the example device 1402 with other devices, servers, etc. The resources 1428 may also include applications and / or data that can be utilized while computer processing is executed on servers that are remote from the example device 1402. Additionally, the services 1426 and / or the resources 1428 may facilitate subscriber network services, such asover the Internet, a cellular network, or a Wi-Fi network. The platform 1424 may also serve to abstract and scale resources to service a demand for the resources 1428 that are implemented via the platform 1424, such as in an interconnected device aspect with functionality distributed throughout the system 1400. For example, the functionality may be implemented in part at the example device 1402 as well as via the platform 1424 that abstracts the functionality of the cloud 1422.Conclusion
[0099] Although aspects of context-aware on-device intelligence have been described in language specific to features and / or methods, the subject of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of the techniques, and other equivalent features and methods are intended to be within the scope of the appended claims. Further, various aspects are described, and it is to be appreciated that each described aspect can be implemented independently or in connection with one or more other described aspects.
Claims
CLAIMSWhat is claimed is:
1. A method for context-aware on-device intelligence, the method comprising: capturing on-screen raw data displayed via a display device for an electronic device to provide captured content; at an operating-system level of the electronic device, converting the captured content into structured entities; at the operating-system level, generating embedding representations of the structured entities; and at the operating-system level, storing the embedding representations and the structured entities in a context store for retrieval of information representing a user context.
2. The method of claim 1, further comprising: receiving a user prompt; at the operating-system level, converting the user prompt into an embedding vector; at the operating-system level, determining a subset of the embedding representations that are within a similarity threshold to the embedding vector of the user prompt; at the operating-system level, generating a set of final suggestions; and providing the set of final suggestions via the display device for user selection.
3. The method of claim 1 or claim 2, wherein the on-screen raw data includes one or more of push notifications, automatic-speech recognition, and data accessed from a shared directory.
4. The method of claim 1 or claim 2, wherein the captured content includes one or more of textual data, audio transcripts, and image data.
5. The method of any one of the preceding claims, wherein the context store is a vector database.
6. The method of claim 5, wherein the context store is stored locally on the electronic device.
7. The method of any one of the preceding claims, further comprising categorizing the information presenting the user context into a plurality of time-frame categories based on a duration of time that the embedding representations and the structured entities are stored in the context store.
8. The method of claim 7, wherein the plurality of time-frame categories include short-term context and long-term context, the short-term context and the long-term context being non-overlapping with one another.
9. The method of any one of the preceding claims, wherein the captured content represents implicit context of the user, the implicit context including on-screen data for which the user has not confirmed via a user input to save for later use.
10. The method of any one of claims 1 to 8, wherein the captured content represents explicit context, the explicit context including content explicitly saved by a user input.
11. The method of any one of the preceding claims, further comprising batching the captured content prior to the converting of the captured content into the structured entities.
12. The method of any one of the preceding claims, wherein the embedding representations are generated by a first large language model stored locally on the electronic device.
13. The method of claim 12, further comprising providing one or more of the embedding representations as input to a second large language model to use as context for the user to enable the second large language model to generate one or more contextually aware responses.
14. An electronic device comprising: at least one memory system comprising computer program code; and at least one processor configured to execute the computer program code in the at least one memory is configured to cause the electronic device to perform the method of any of claims 1-13.
15. A computer program product comprising instructions that, when run on an electronic device, cause the electronic device to perform the method of any of claims 1-13.
Citation Information
Patent Citations
Modeling personal entities
US20170032257A1
