Cascading method for detecting and interpreting user activity

Abstract

Various implementations disclosed herein include devices, systems, and methods for detecting and interpreting user activity using a resource-heavy process triggered or directed by a determination made by a resource-lightweight process. For example, a method may include performing a first process to produce an output. The first process may include: detecting an event based on a first set of sensor data; identifying a subset of events as human-related events corresponding to one or more predetermined categories depicted in the first set of sensor data; and collecting information about human-related events based on the first set of sensor data. Based on the output of the first process, a second process may be performed to interpret user activity. The second process may include obtaining a second set of sensor data and using the second set of sensor data to interpret the user activity.

Description

Technical Field

[0001] This disclosure relates in general to systems, methods, and apparatuses for detecting and interpreting user activity using resource-intensive processes that are triggered or guided by analysis obtained from an initial resource-lightweight process. Background Technology

[0002] Existing technologies for detecting user activity can be improved in terms of accuracy, power consumption, and / or the type of user activity detected (e.g., activities involving a combination of motion, sound, gaze, etc.). Summary of the Invention

[0003] The various specific embodiments disclosed herein include devices, systems, and methods for detecting and interpreting user activity via resource-intensive processes triggered and / or guided by determinations and decisions initially performed by resource-lightweight processes. For example, compared to resource-intensive processes that utilize sensors and data (such as cameras, camera settings, large language models (LLMs), etc.) requiring more power and computational resources, resource-lightweight processes can operate continuously (or nearly continuously) using sensor data (e.g., hand position data, gaze data, audio data, inertial measurement unit (IMU) data) from fewer sensors and computational resources.

[0004] In some implementations, resource-intensive and resource-light processes can be configured to acquire different multimodal inputs. For example, a resource-light process can detect and collect data associated with human-related events, such as sounds, actions associated with interaction with objects, user movement, etc. Similarly, a resource-light process can detect and classify the current event and / or the object of interest interacting with it to trigger and / or guide improved resource-intensive process decisions. In some implementations, when a triggering event is detected, the LLM (of the resource-intensive process) can be activated to further analyze the triggering event via high-power multimodal processing that can acquire inputs such as images, audio, and / or contextual data. The event can then be further processed. The resource-intensive process can be configured to generate further outputs or predictions, such as a textual description of user activity. In some implementations, multimodal inputs can include, in particular, user voice input, user gaze input, user hand or finger input, body language input, etc.

[0005] In some implementations, a first subset of the resource-lightweight process (e.g., performed by an audio sensor) can be configured to control the operation of a second subset of the resource-lightweight process, such that the second subset (e.g., performed by a camera) can be analyzed by the LLM after the audio signal obtained from the LLM analysis. For example, the first subset of the resource-lightweight process may include audio detection sensing (e.g., detecting speech), which triggers the limited capabilities of the LLM (e.g., viewing audio data without viewing image data) to interpret the speech, and based on the interpreted speech, it can be determined whether a resource-intensive process is necessary. If it is determined that a resource-intensive process is necessary, the use of a second subset of the resource-lightweight process (e.g., a camera) can be triggered to capture images for use in the resource-intensive process, such as the LLM combined with a video encoder, image processing, etc., to analyze hand / gaze posture, as described with reference to the following example: In some implementations, the user can verbally command, "When was car type A manufactured?" In this case, only a limited-capacity LLM is needed to analyze the audio data, as there is no need to enable the camera to perform any resource-intensive processes (such as hand / gaze detection algorithms), since the verbal command does not include any words that would indicate the user is referring to an object in the current physical environment.

[0006] Alternatively, if the user utters the verbal command "when was that car manufactured," the LLM can first be used with limited capabilities (requiring only audio data) to interpret the verbal command (e.g., the user is requesting information related to a car in their environment), and based on the LLM's output, the process can determine if resource-intensive processes (e.g., image capture and object detection) are necessary to identify the car in the physical environment. Thus, a camera can be activated to capture an image, and more computationally intensive processes such as image detection can be performed. In response, if only one car is detected, the process can assume the user is referring to that car. However, if two cars are present in the physical environment, the process is configured to enable another resource-intensive process (such as, for example, hand / gaze recognition) to determine which car the user is referring to when uttering the term "that."

[0007] In some embodiments, a device has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes. In some embodiments, the method performs a first process to produce output. The first process includes: detecting an event based on a first set of sensor data; identifying a subset of the event as human-related events corresponding to one or more predetermined categories depicted in the first set of sensor data; and collecting information about the human-related events based on the first set of sensor data. Based on the output of the first process, the method performs a second process to interpret user activity. The second process includes obtaining a second set of sensor data and using the second set of sensor data to interpret user activity.

[0008] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and components for performing or causing to perform any of the methods described herein. Attached Figure Description

[0009] To enable those skilled in the art to understand this disclosure, more detailed descriptions can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.

[0010] Figure 1 Exemplary electronic devices operating in a physical environment according to some specific implementations are illustrated.

[0011] Figure 2A and Figure 2B An example view of a system including a low-power system according to some specific implementations is shown, which outputs information via a human-related module to trigger a high-power system.

[0012] Figure 3 This illustrates a view of a system, based on some specific implementations, that combines a low-level signal processing system with a high-power multimodal language modeling system to capture detailed behavior and events in real time.

[0013] Figure 4 An example process for locating a car in a garage is illustrated according to some specific implementations of the system shown in Figure 2.

[0014] Figure 5 This illustrates a view of a system that uses low-level signals as triggers to activate the system, based on some specific implementations.

[0015] Figure 6 It is a flowchart representation of an exemplary method for detecting and interpreting user activity via a resource-intensive process, which is triggered and / or guided by determinations and decisions enabled by a resource-lightweight process.

[0016] Figure 7 It is a block diagram based on some specific implementations of electronic devices.

[0017] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0018] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0019] Figure 1 An exemplary electronic device 105 operating in physical environment 100 is illustrated. Figure 1 In the example, physical environment 100 is a room including table 120. Electronic device 105 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture and evaluate information about physical environment 100 and objects within it, as well as information about user 102 of electronic device 105. Information about physical environment 100 and / or user 102 can be used to provide visual and audio content, and / or identify the current location of physical environment 100 and / or the location of the user within physical environment 100.

[0020] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via an electronic device 105 (e.g., a wearable device such as an HMD). Such an XR environment may include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100, and a representation of user 102 based on camera images and / or depth camera images of user 102. Such an XR environment may include virtual content positioned at a 3D location relative to a 3D coordinate system associated with the XR environment, which may correspond to the 3D coordinate system of the physical environment 100.

[0021] In some specific implementations, the system including electronic device 105 may be configured (e.g., using low-computation and / or low-power sensors, etc.) to perform a first process such as a resource-lightweight process to produce an output, which may be configured to trigger a second process or may include information for guiding the second process.

[0022] In some specific implementations, the first process may include detecting an event based on a first set of sensor data. For example, events may include, in particular, hand position events, gaze direction events, audio events, inertial measurement unit (IMU) events, etc.

[0023] In some implementations, the first process may also include identifying a subset of events as human-related events corresponding to one or more predetermined categories depicted in the first set of sensor data. For example, identifying a subset of events as human-related events may include using a Human Basic Model (HFM) and a relevance decoder to identify events such as hand and gaze events, hand-object interaction events, visual attention events to objects, text reading events, user-initiated speech or sound events, human movement events, etc.

[0024] In some implementations, the first process may also include collecting information related to human-related events based on the first set of sensor data. Information collection can be performed by continuously collecting the first set of sensor data.

[0025] In some implementations, a second process is executed or triggered based on the output of the first process. The second process is configured to interpret user activity by acquiring a second set of sensor data (e.g., visual sensor data, higher frame rate data, higher resolution data, etc.) and using that second set of sensor data to interpret user activity. For example, interpreting user activity may include LLM processing.

[0026] Figure 2A and Figure 2BA view of system 200, including a low-power system 210, is illustrated according to some specific implementations. This low-power system outputs information via a human relevance module 202 to trigger a high-power (multimodal language model) system 225, including an LLM 228. In some implementations, the low-power system 210 may include a low-power, event-driven module to serve as an initial detector. Similarly, the high-power system 225 can only be activated when a relevant signal is triggered (e.g., from the human relevance module 202). System 200 ensures that computationally expensive models are used only when necessary, thereby optimizing both power consumption and processing efficiency of system 200.

[0027] The low-power system 210 is configured to use multiple sensing modules (e.g., modules 211, 212, 215, and 216) to detect, track, and interpret various events or interactions in the environment (identifying events 205 via the HFM 206 of the human relevance module 202 and the relevance decoder 204), without relying on a full-scale language model. For example, the low-power system 210 can be configured to operate with minimal computational resources, focusing on processing basic low-dimensional signals from sensors (such as cameras, in particular (for object, hand, and environment tracking), audio sensors (for sound classification, speech-to-text, etc.) from modules 211, 212, 215, and 216) as well as additional sensor inputs (e.g., gaze tracking, motion sensing, etc.).

[0028] In some embodiments, modules 211 and 212, along with associated sensors 217 and 218, are included in a first group 222 of low-power / low-computation modules / sensors. In some embodiments, modules 215 and 216, along with associated sensors 221 and 220, are included in a second group 223 of low-power / low-computation modules / sensors, which has lower power consumption / computation than the first group 222 of low-power / low-computation modules / sensors. In some embodiments, operation of the first group 222 of low-power / low-computation modules / sensors can be triggered by a high-power system 225 upon detection of, for example, audio (e.g., speech), which triggers the limited ability of LLM 228 to interpret the audio. Based on the interpreted audio, the first group 222 of low-power / low-computation modules / sensors (e.g., a camera) can be triggered to capture images for resource-intensive processes, such as assessing hand / gaze posture.

[0029] Module 211 is a visual perception module configured to perform object detection activities. For example, module 211 can be configured to use an outward-facing camera (OFC) 217 ​​to detect objects, hands, people, and environmental features such as lighting and space. Similarly, module 211 can be configured to perform saliency detection, such as identifying, for example, the area within the user's field of vision most likely to be relevant to the user, such as objects with which they interact.

[0030] Module 212 is configured to enable gaze detection. For example, module 212 can be configured to enable inward-facing camera (IFC) 218 ​​to monitor user behavior, such as the direction the user is looking, focusing on or paying attention to objects or the surrounding environment.

[0031] Module 215 is an audio perception module configured to use sensor 221 (e.g., a microphone) to enable sound classification, speech-to-text, and behavior prediction processes. For example, the sound classification process can be configured to recognize ambient sounds or speech, classify activities such as chewing or eating, etc. Similarly, the behavior prediction process is configured to perform audio-based behavior recognition in conjunction with other sensory inputs, such as, for example, distinguishing whether a person is speaking while walking or sitting (e.g., spoken words).

[0032] Module 216 is configured (e.g., via IMU sensor 220) to detect user activities such as walking, running, standing, sitting and / or transitions between these activities.

[0033] In some implementations, the low-power system 210 can operate continuously to identify potentially relevant events or changes in the environment or user behavior, such as loud sounds, pointing, movement, or interaction with objects. Therefore, once a relevant event is detected, the low-power system 210 can generate a low-power output (e.g., trigger event) to signal the occurrence of the relevant event.

[0034] In some specific implementations, audio signals from an audio sensor (e.g., sensor 221) can be used to trigger limited use of LLM 228, and based on the results of analysis performed by LLM 228 on the audio signals, low-power sensors (e.g., sensor 217) such as cameras (e.g., cameras are higher power sensors than audio sensors) can be activated, enabling LLM 228 to perform multimodal processing.

[0035] For example, a second set 223 of low-power / low-computation modules / sensors (associated with a subset of resource-lightweight processes) can be configured to control the operation of a first set 222 of low-power / low-computation modules / sensors (another subset of resource-lightweight processes), such that processes performed by the first set 222 of low-power / low-computation modules / sensors (e.g., performed by a camera) can be performed by the LLM 228 after the associated audio signal has been analyzed by the LLM 228. Therefore, the second set 223 of low-power / low-computation modules / sensors may include triggering the limited capabilities of the LLM 228 (e.g., viewing audio data without viewing image data) to interpret speech for audio detection functions (e.g., detecting speech). Based on the interpreted speech, it can be determined whether resource-intensive processes should be implemented via the full capabilities of the LLM 228. If it is determined that a resource-intensive process should be implemented, the use of other sensors (e.g., cameras of the first set 222 of low-power / low-computation modules / sensors) can be triggered to capture images for use in resource-intensive processes such as the LLM 228 in conjunction with the video encoder of the imaging system 236, as described with reference to the following example: In some implementations, the user can verbally command, "Car type A was manufactured in what year?" In this case, the LLM 228 is enabled with limited capabilities to analyze only audio data, as there is no need to enable the camera to perform resource-intensive processes (such as hand / gaze detection algorithms), since the verbal command does not include any words indicating that the user is referring to an object in the current physical environment.

[0036] Alternatively, if the user utters the verbal command “when was that car manufactured,” the LLM 228 can be initialized with limited capabilities (to analyze audio data) to interpret the verbal command (e.g., the user is requesting information related to a car in the current environment), and based on the output of the LLM 228, the process can determine if resource-intensive processes (e.g., image capture and object detection) are necessary to identify the car in the physical environment. In response, a camera can be activated to capture an image, and more computationally intensive processes such as image detection can be performed. In response, if only one car is detected, the process can assume the user is referring to that car. Similarly, if two cars are present in the physical environment, the process can be configured to enable another resource-intensive process (such as, for example, hand / gaze recognition) to determine which car the user is referring to when uttering the term “that.”

[0037] In some implementations, when a triggering event is detected, system 200 activates LLM 228 to further analyze the event via high-power multimodal processing, which takes inputs such as images (e.g., video frames) from image system 236, audio from audio system 238, and context data from context system 234. For example, if low-power system 210 detects hand-object interaction, high-power system 225 can be configured to process the entire image sequence to generate predictions such as, for example, “the user is picking up a coffee cup.”

[0038] In some implementations, the high-power system 225 can be configured to process multiple input types, such as, for example, image sequences (from a camera) or audio clips (for speech or sound analysis of spoken words), to generate a more detailed understanding of events. Multiple input types can be processed via modules 226, 227, 229, 230, and 232 to be input into the LLM 228.

[0039] In some implementations, the high-power system 225 can be configured to use additional contextual data, such as location (from room detection), historical patterns (e.g., typical actions at specific times), and calendar data, to enhance analysis. This additional contextual data can enable event interpretation, such as determining, for example, that at 8:00 AM, the user is in the kitchen and that the user typically drinks coffee at this time.

[0040] After processing the event, the high-power system 225 can generate further output or predictions including a text description, such as: "The user picked up a coffee cup in the kitchen at 8:15 a.m." Similarly, further outputs or predictions can include higher-level feature embeddings, such as embeddings from image or video analysis, embeddings for audio-based analysis, behavioral embeddings for detecting time-based or action-based patterns, etc. These embeddings can be stored for later analysis, future queries, or integration into other systems.

[0041] Therefore, system 200 implements a process that enables an always-on perception layer (low-power system 210) that conserves power by generating low-dimensional signals using only lightweight perception models (e.g., modules 211, 212, 215, and 216) and triggering higher-power computational models only when relevant events occur. This ensures that computationally intensive resources (e.g., full image processing or multimodal analysis) are used only when necessary. System 200 can continuously evolve its understanding of the user's environment and activities, moving from low-power initial detection to deeper analysis if needed.

[0042] Figure 3An example view is shown of a system 300 that combines a low-level signal processing system 302 with a high-power multimodal language modeling system 315 to capture detailed behavior and events in real time, according to some specific implementations.

[0043] The low-level signal processing system 302 can be configured to operate continuously to detect basic scene-level signals and patterns from sensors (such as motion detectors, microphones, etc.) to capture a coarse scene-level description, such as, for example, that a user is eating breakfast or going to work.

[0044] The high-power multimodal language modeling system 315 can only be activated when a significant event is detected, thereby providing a detailed explanation of the significant event.

[0045] For example, a multi-level process can be enabled, allowing the low-level signal processing system 302 to continuously monitor user behavior and trigger the high-power multimodal language modeling system 315 only when a specific event is detected. Thus, fine-grained, moment-by-moment behavior, such as "you took medication" or "you left your coffee cup on the table," can be captured.

[0046] The multi-level process described above generates a semantic index or log of daily activities, in which each detected event is tagged and stored in real time, allowing for the tracking of detailed behaviors and relationships between events. This can be useful for various applications such as health monitoring, personal diaries, and productivity analytics.

[0047] Therefore, system 300 can provide real-time, fine-grained logging of behavior, while balancing power efficiency by leveraging low-level signal processing system 302 to trigger a more powerful, expensive, high-power multimodal language modeling system 315 on demand.

[0048] Figure 4 An example process 400 for locating a car in a garage using the system 200 of Figure 2, according to some specific implementation, is illustrated. Process 400 is configured to help the user remember events by intelligently capturing and processing key interactions. Process 400 implements a combination of real-time data collection and event-driven processing to balance power consumption with functionality.

[0049] In some implementations, process 400 is configured to create a software assistant that operates by logging interactions or events in real time for future queries. In other implementations, process 400 is configured to collect data related to user behavior, preferences, and the environment, while logging only relevant events to conserve battery life and processing power. Therefore, a selective data collection process can be implemented, allowing process 400 to trigger data collection only at specific moments that may be important or useful to the user, rather than continuously recording video or monitoring every action or event.

[0050] For example, if a user is driving to the airport and glances at a sign 402 stating "Parking on this floor is full, please go to floor 3," process 400 can recognize that the user is reading the sign and can capture a snapshot 403 of the event. Similarly, when a user parks their car, process 400 can record the location 405 based on the detection of a disconnection from a wireless system (e.g., Bluetooth), GPS, or other sensors 407, and subsequently collect contextual data 409, 411, and 412, such as signs 414 or 416 viewed by the user, or buttons 418 that the user has activated.

[0051] In some implementations, process 400 can be configured to process multiple forms of data, such as, in particular, images, text, GPS signals, and object interaction data. Multiple forms of data can be processed by detecting and identifying key moments (such as reading a stop sign, pressing an elevator button, or leaving a car). In some implementations, the detected and identified key moments can be sent to a multimodal language model (e.g., LLM228 in the figure) to provide answers to questions such as "Where did I park my car?". Subsequently, associated images, text, and interaction logs can be combined into a query, which is then passed to the multimodal language model to provide the correct answer to the question.

[0052] In some implementations, interactions associated with process 400 can be stored in logs or databases that can be queried later. For example, interactions can be associated with moments when a user interacts with the world in a meaningful way (such as looking at a sign, pressing a button, etc.). Therefore, if the user subsequently asks a question such as, "Where did I park my car?", process 400 can use a voice query to trigger a search of logs about the key event and retrieve snapshots or contextual data points relevant to the question. Thus, process 400 is configured to detect when a user engages in meaningful activity (e.g., reading a sign, interacting with an object, etc.) to minimize power consumption while collecting enough data to provide useful insights.

[0053] In some implementations, sensors such as gaze tracking, wireless signal disconnection tracking, or object interaction tracking can be used as low-power, always-on components that signal when more detailed data is to be collected. Therefore, process 400 can filter and retrieve snapshots of the most relevant events instead of generating continuous data feeds (e.g., starting from the last few days). For example, process 400 can retrieve only the moments when a user interacted with a specific stop sign or parked a car, rather than reviewing video from the last few hours, thus streamlining the query process and making it faster and more efficient to obtain useful information without unnecessary data overload.

[0054] In some specific implementations, process 400 may incorporate machine learning (ML) to predict relevant user moments and use relevant user moments to enable ML to improve over time regarding pattern recognition and predict when data capture may be useful.

[0055] In some implementations, process 400 can perform complex queries, such as requesting a summary of a day or week from a user by stitching together the aforementioned snapshots into a coherent timeline of key events.

[0056] Therefore, Process 400 can intelligently balance data collection and processing efficiency by capturing critical moments in real time through the use of sensors to detect key moments related to user activity, while minimizing power consumption. This selective, event-driven approach allows Process 400 to act as an efficient personal assistant, providing timely assistance and recall based on past interactions.

[0057] In some implementations, process 400 enables cascaded sensor and processing operations such that a first subset of low-power sensors (e.g., audio sensors) is initialized to capture data such as audio data for input into an LLM (operating with limited unimodal capabilities) configured to evaluate the audio data. Subsequently, additional higher-power sensors (such as cameras) can be enabled to provide input to the LLM (operating with multimodal capabilities) to interpret environmental context and hand and gaze postures. For example, the operation of the first low-power / low-computation sensor can be triggered upon detecting, for example, audio data such as speech. This operation can be configured to trigger the LLM's limited capabilities to interpret the speech, and based on the interpreted speech, a second, different (and higher-power) sensor (e.g., a camera) can be triggered to capture images for evaluating hand and gaze postures, thereby refining the outcome of indicative user requests or commands.

[0058] Figure 5 A view of system 500 is illustrated, illustrating how low-level signals are used as triggers to activate the system according to some specific implementations. In some specific implementations, low-level signals obtained from sensors 509 (e.g., gaze detection sensors, motion detection sensors, wireless signal (long-range and short-range) disconnection sensors, etc.) can be used (via human base model 510 and adapter 514) to trigger the activation of higher-power components (such as cameras or language models such as LLM 522).

[0059] In some implementations, when a triggering event is detected, system 500 may activate LLM 522 to further analyze the event via high-power multimodal processing, which takes input such as an image 511 (e.g., a video frame) for processing via video encoder 512 and adapter 517 and text 519 for processing via text tagger 518.

[0060] In some implementations, system 500 activates high-power components (e.g., LLM 522 or camera) only when it detects a relevant human-centered event occurring (e.g., a user asking "What is that?" while pointing at an object such as hat 506a). For example, when a high-level component such as camera or LLM 522 is triggered, low-level signals obtained from sensor 509 (e.g., gaze focus, gesture, object interaction, etc.) are configured to provide additional context to guide the high-level system (system 500) to ensure it does not process unnecessary data. For example, system 500 is configured to process only the relevant portion 504 of image 502 associated with such as gaze, hand gesture, and / or audible requests such as "What is that?", rather than analyzing the entire scene of image 502. Therefore, selectively capturing information (e.g., portion 504 of image 502) reduces the amount of data that needs to be processed, thereby contributing to the efficiency and latency of system 500.

[0061] In some implementations, system 500 enables cascaded sensor and processing operations such that a low-power sensor (e.g., the audio sensor of sensor 509) is initialized to capture data such as audio data for input into an LLM 522 (operating with limited single-modal capabilities) configured to evaluate the audio data. Subsequently, a higher-power sensor (e.g., the camera of sensor 509) can be enabled to provide input to the LLM 522 (now operating with multimodal capabilities) to interpret the environmental context and hand and gaze postures. For example, the operation of a first low-power / low-computation sensor can be triggered upon detecting, for example, audio data such as speech. This operation can be configured to trigger the limited capabilities of the LLM 522 to interpret the speech, and based on the interpreted speech, a second, different (and higher-power) sensor (e.g., a camera) can be triggered to capture images to evaluate hand and gaze postures, thereby refining the outcome of a user request or command, such as answering a question like "What is that?".

[0062] Figure 6 This is a flowchart representation of an exemplary method 600 for detecting and interpreting user activity via a resource-intensive process, according to some specific implementations. This resource-intensive process is triggered and / or guided by determinations and decisions enabled by a resource-lightweight process. In some implementations, method 600 is performed by a device, such as a mobile device, desktop computer, laptop computer, HMD, or server device. In some implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as... Figure 1(Device 105). In some embodiments, method 600 is performed by processing logic components, including hardware, firmware, software, or combinations thereof. In some embodiments, method 600 is performed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory). Each block in method 600 can be enabled and executed in any order.

[0063] At box 602, method 600 performs a first process (e.g., a resource-light process implemented via, for example, the low-power system 210 of FIG2) to produce an output for triggering a second process (such as a resource-intensive process implemented via, for example, the high-power system 225 of FIG2). In some specific implementations, the first process includes: 1. An event is detected based on a first set of sensor data (such as hand position, gaze data, audio data, IMU data, etc.) obtained via sensors (such as OFC 217, IFC 218, sensor 221 and IMU sensor 220, etc.), as described with respect to Figure 2.

[0064] 2. Identify a subset of events as human-related events corresponding to one or more predetermined categories depicted in the first set of sensor data. For example, this could involve using HFM 206 and the correlation decoder 204 to identify events involving: hands and gaze, hand-object interaction, visual attention to an object, reading text, user-initiated speech or sound, human movement, etc., as described with respect to Figure 2.

[0065] 3. Based on this first set of sensor data, information about human-related events is collected.

[0066] In some specific implementations, the first set of sensor data includes data selected from the following groups: hand position data, gaze data, audio data, and IMU data.

[0067] At box 604, based on the output of the first process (e.g., triggered by it or using information from it), method 600 performs a second process to interpret user activity. The second process may include obtaining a second set of sensor data (e.g., such as information about...). Figure 1 The second set of sensor data (described as a visual sensor, higher frame rate, higher resolution, etc.) is used to interpret user activity. For example, interpreting user activity may include computationally intensive processes, such as the LLM228 processing described with respect to Figure 2.

[0068] In some implementations, a second process can be triggered based on the detection of human-related events by the first process. In some implementations, human-related events can include events such as, in particular, audible sounds, interactions with objects, and user movement.

[0069] In some implementations, the second process may use the output of the first process (e.g., information about human-related events) to interpret user activity.

[0070] In some implementations, interpreting user activity may include categorizing current events into subsets of events.

[0071] In some implementations, interpreting user activity involves combining user gaze, posture, body movement, body language, or facial expressions to interpret spoken words.

[0072] In some specific implementations, the second set of sensor data may include data such as, for example, visual sensor data, frame rate data, video resolution data, etc.

[0073] In some specific implementations, using a second set of sensor data to interpret user activity may include using LLM processing.

[0074] Figure 7 This is a block diagram of example device 700. Device 700 illustrates... Figure 1 An exemplary device configuration of electronic device 105. Although certain specific features have been illustrated, those skilled in the art will recognize from this disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure further relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 700 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, and / or processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, output devices (e.g., one or more displays) 712, one or more internal and / or external image sensor systems 714, memory 720, and one or more communication buses 704 for interconnecting these components and various other components.

[0075] In some embodiments, one or more communication buses 704 include circuitry for interconnecting system components and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 706 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), one or more cameras (e.g., an inward-facing camera and an outward-facing camera of an HMD), one or more infrared sensors, one or more thermal sensors, and / or the like.

[0076] In some embodiments, one or more displays 712 are configured to present a view of a physical environment, a graphical environment, an extended reality environment, etc., to a user. In some embodiments, one or more displays 712 are configured to present content to a user (determined based on the determined user / object position within the physical environment). In some embodiments, one or more displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more displays 712 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. In one example, device 700 includes a single display. In another example, device 700 includes displays for each of the user's eyes.

[0077] In some embodiments, one or more image sensor systems 714 are configured to acquire image data corresponding to at least a portion of the physical environment 100. For example, one or more image sensor systems 714 include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various embodiments, one or more image sensor systems 714 also include an illumination source emitting light, such as a flash. In various embodiments, one or more image sensor systems 714 also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0078] In some embodiments, device 700 includes an eye-tracking system for detecting eye positioning and eye movement (e.g., eye gaze detection). For example, the eye-tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye-tracking camera (e.g., a near-infrared (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the user's eyes. Furthermore, the illumination source of device 700 may emit NIR light to illuminate the user's eyes, and the NIR camera may capture images of the user's eyes. In some embodiments, the images captured by the eye-tracking system may be analyzed to detect the positioning and movement of the user's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. Furthermore, the gaze point estimated from the eye-tracking images enables gaze-based interaction with content displayed on a near-eye display of device 700.

[0079] Memory 720 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 720 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 720 optionally includes one or more storage devices remotely located to one or more processing units 702. Memory 720 includes a non-transitory computer-readable storage medium.

[0080] In some embodiments, memory 720 or a non-transitory computer-readable storage medium of memory 720 stores an optional operating system 730 and one or more instruction sets 740. Operating system 730 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 740 includes executable software defined by binary information stored in charge. In some embodiments, instruction set 740 is software executable by one or more processing units 702 to implement one or more of the techniques described herein.

[0081] Instruction set 740 includes data element generation instruction set 742, first process execution instruction set 744, and second process execution instruction set 746. Instruction set 740 can be represented as a single software executable file or multiple software executable files.

[0082] The first process execution instruction set 742 is configured with instructions that can be executed by the processor to perform a resource-light process with respect to initial sensor data (e.g., continuously) to trigger a second process for a more detailed interpretation of user activities.

[0083] The second process is executed. [Should "execution" be here? Should it be in the diagram? – Note] Figure 7 [Differences between 742 and 744] Instruction set 744 configures instructions that can be executed by the processor to perform resource-intensive processes (triggered by resource-light processes) to interpret user activity based on additional sensor data that differs from the initial sensor data.

[0084] The discourse interpretation instruction set 746 [I can't see this in the diagram] is configured with instructions that can be executed by the processor to interpret discourse using a subset of data elements based on timing attributes.

[0085] Although instruction set 740 is shown as residing on a single device, it should be understood that in other specific implementations, any combination of elements may reside in separate computing devices. Furthermore, Figure 7 This is intended more as a functional description of various features present in a particular implementation than as a structural diagram of the specific implementation described herein. As will be appreciated by those skilled in the art, the items shown individually can be combined, and some items can be separated. The actual number of instruction sets and how features are allocated therein will vary depending on the specific implementation and may depend in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0086] Return to Figure 1A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, and / or virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. As an example, an XR system can detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in a physical environment. For example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) presenting an XR environment, and in response, adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in the physical environment. In some cases (e.g., for accessibility reasons), an XR system can adjust the characteristics of the graphical content in the XR environment in response to representations of physical movement (e.g., voice commands).

[0087] Many different types of electronic systems enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection techniques that project graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0088] Those skilled in the art will understand that well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure more relevant aspects of the specific embodiments of the examples described herein. Furthermore, other effective aspects and / or variations do not include all the details in the specific details described herein. Therefore, several details are described to provide a thorough understanding of the exemplary aspects illustrated in the accompanying drawings. Moreover, the drawings only illustrate some exemplary embodiments of this disclosure and should not be considered limiting.

[0089] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of different embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while certain features may be described above as functioning in certain combinations and even initially claimed in this manner, one or more features of a claimed combination may be removed from that combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0090] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in a sequential order or the specific order shown, or requiring all illustrated operations to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the partitioning of the various system components in the above embodiments should not be construed as requiring such partitioning in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0091] Therefore, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

[0092] The embodiments of the subject matter and operation described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware (including the structures disclosed in this specification and their equivalents) or in a combination thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0093] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or many or combinations of the foregoing. The apparatus may include special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program under consideration, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as "processing," "computing," "calculating," "determining," and "identifying" refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, that manipulate or convert data represented as physical electronic or magnetic quantities within the memory, registers, or other information storage devices, transmitting devices, or display devices of a computing platform.

[0094] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein may be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0095] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the examples above can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Some boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0096] The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Furthermore, the use of "based on" implies openness and inclusivity, as processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be restrictive.

[0097] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node may be called a second node, and similarly, a second node may be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0098] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the term “comprising,” as used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0099] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrase "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" can be interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.

Claims

1. A method comprising: At a device that has a processor and one or more sensors: Perform a first process to produce output, the first process including: The event is detected based on the first set of sensor data; A subset of the events is identified as human-related events corresponding to one or more predetermined categories depicted in the first set of sensor data; and Information about the human-related events is collected based on the data from the first set of sensors; and Based on the output of the first process, a second process is performed to interpret user activity, the second process including obtaining a second set of sensor data and using the second set of sensor data to interpret the user activity.

2. The method of claim 1, wherein the second process is triggered based on the detection of a human-related event by the first process.

3. The method of claim 2, wherein the human-related events include events selected from the group consisting of: audible sounds, interactions with objects, and user movement.

4. The method of claim 1, wherein the second process uses the output of the first process to interpret the user activity, the output including the information about the human-related events.

5. The method of claim 1, wherein interpreting the user activity includes classifying current events in the subset of the events.

6. The method of claim 1, wherein interpreting the user activity includes interpreting spoken words in conjunction with the user's gaze, posture, body movement, body language, or facial expressions.

7. The method of claim 1, wherein the first set of sensor data includes data selected from the group consisting of: hand position data, gaze data, audio data, and IMU data.

8. The method of claim 1, wherein the second set of sensor data includes data selected from the group consisting of: visual sensor data, frame rate data, and video resolution data.

9. The method of claim 1, wherein using the second set of sensor data to interpret the user activity includes using Large Language Model (LLM) processing.

10. An electronic device, comprising: One or more sensors; Non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the electronic device to perform operations including: Perform a first process to produce output, the first process including: The event is detected based on the first set of sensor data; A subset of the events is identified as human-related events corresponding to one or more predetermined categories depicted in the first set of sensor data; and Information about the human-related events is collected based on the data from the first set of sensors; and Based on the output of the first process, a second process is performed to interpret user activity, the second process including obtaining a second set of sensor data and using the second set of sensor data to interpret the user activity.

11. The electronic device of claim 10, wherein the second process is triggered based on the detection of a human-related event by the first process.

12. The electronic device of claim 11, wherein the human-related events include events selected from the group consisting of: audible sounds, interactions with objects, and user movement.

13. The electronic device of claim 10, wherein the second process uses the output of the first process to interpret the user activity, the output including the information about the human-related events.

14. The electronic device of claim 10, wherein interpreting the user activity includes classifying current events in the subset of the events.

15. The electronic device of claim 10, wherein interpreting the user activity includes interpreting spoken words in conjunction with the user's gaze, posture, body movement, body language, or facial expressions.

16. The electronic device of claim 10, wherein the first set of sensor data includes data selected from the group consisting of: hand position data, gaze data, audio data, and IMU data.

17. The electronic device of claim 10, wherein the second set of sensor data includes data selected from the group consisting of: visual sensor data, frame rate data, and video resolution data.

18. The electronic device of claim 10, wherein using the second set of sensor data to interpret the user activity includes using Large Language Model (LLM) processing.

19. A non-transitory computer-readable storage medium storing program instructions, the program instructions being executable via one or more processors to perform operations, the operations including: Perform a first process to produce output, the first process including: The event is detected based on the first set of sensor data; A subset of the events is identified as human-related events corresponding to one or more predetermined categories depicted in the first set of sensor data; and Information about the human-related events is collected based on the data from the first set of sensors; and Based on the output of the first process, a second process is performed to interpret user activity, the second process including obtaining a second set of sensor data and using the second set of sensor data to interpret the user activity.

20. The non-transitory computer-readable storage medium of claim 19, wherein the second process is triggered based on the detection of a human-related event by the first process.

Images