User activity predictions for wearable device users based on user activity context tracking

Abstract

Devices, systems, and methods that facilitate a user activity prediction based on a user's current context (e.g., based on determining that the user is sitting in their office chair looking at the monitor on the desk). The prediction may be facilitated based on storing and aggregating prior user activity data (e.g., data identifying that, while sitting at the chair looking at the monitor on the desk, the user was working on 15 prior occasions and surfing the Internet on 3 prior occasions). The activity data includes contextual data based on one or more wearable device sensors that generally provides more information about the user relative to the environment than the sensors on prior devices.

Description

Attorney Docket No. 097425-01485(P68856WOl)USER ACTIVITY PREDICTIONS FOR WEARABLEDEVICE USERS BASED ON USER ACTIVITYCONTEXT TRACKINGTECHNICAL FIELD

[0001] The present disclosure generally relates to systems, methods, and devices that predict user activities based on tracking user activities and associated contextual information during use of wearable electronic devices such as head-mounted devices (HMDs).BACKGROUND

[0002] Existing devices and systems may not efficiently or effectively predict user activities based on tracking user activities and associated contextual information during use of wearable electronic devices such as HMDs.SUMMARY

[0003] Various implementations disclosed herein include devices, systems, and methods that facilitate a user activity prediction based on a user’s current context (e.g., based on determining that the user is sitting in their office chair looking at the monitor on the desk). The prediction may be facilitated based on storing and aggregating prior user activity data (e.g., data identifying that, while sitting at the chair looking at the monitor on the desk, the user was working on 15 prior occasions and surfing the Internet on 3 prior occasions). The activity data includes contextual data based on one or more wearable device sensors that generally provides more information about the user relative to the environment (e.g., relationships with objects in the physical environment and / or positional data within the physical environment) than the sensors on prior devices.

[0004] Some implementations involve storing activity data comprising contextual details of a user’s activities, the contextual details providing information about what the user was doing in relation to one or more elements (e.g., fixed objects, walls, floors,Attomey Docket No. 097425-01485(P68856WOl) windows, doorways, etc.) and / or positions (e.g., in the middle of the living room) in the user’s physical environment determined based on sensor data (e.g., using computer vision-based assessments of image data from environment-facing and / or user- facing cameras on a wearable device). Some implementations aggregate (e.g., cluster) the activity data based on the contextual details, e.g., clustering data about similar events as corresponding to the same or similar user activities. User activities may be clustered based on their relationship to the physical environment. Such a relationship may relate to proximity to one or more particular elements, the user being within a defined boundary in the environment, the user having a particular similar view direction or view direction within a range of view directions, the user viewing the same element of the physical object, and / or other contextual information. Such a relationship may relate to the user interacting with the physical environment, e g., determining that the user is chopping ingredients on a cutting board, holding a book to read, etc.

[0005] Some implementations provide the aggregated data to facilitate a prediction associated with a current user activity. In some implementation, the prediction may predict, given that the user’s current circumstances correspond to a particular context, that the user is performing and / or about to perform a particular activity with a predicted probability (e.g., given the current context, there is an 85% chance the user is working, a 10% chance the user is reviewing personal e-mails, etc.). Predictions associated with a user’s current activity can be used for numerous purposes. In some cases, such predications are used to enable the device to perform certain functionality, such as recommending an application appropriate for the user’s current activity, automatically opening up an application / tool, changing setting parameters of one or more sensors, displays, and / or other hardware or software components of the device, etc.

[0006] In some implementations, an electronic device has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes.

[0007] The method involves storing user activity data comprising contextual information for a plurality of activities of a user in one or more physical environments. The contextual information may be based on sensor data obtained via one or more sensors of a wearable device worn by the user. The contextual information for each respectiveAttorney Docket No. 097425-01485(P68856WOl) activity of the activities corresponds to a respective relationship between the user and the one or more physical environments during the respective activity. For example, the contextual information may correspond to the user sitting on the couch watching the TV, the user is standing at the kitchen counter mixing cookie dough, etc.).

[0008] The method further involves generating aggregated data based on aggregating (e.g., clustering) activities of the activity data based on similarities in the contextual information. There may be a taxonomy of different levels of granularity, e g., sitting in the living room, sitting near the desk in the living room, sitting in the chair in front of the desk in the living room, sitting in the chair in front of the desk in the living room while looking at the monitor on the desk, etc. The level of granularity used in aggregating may be selected based on the particular use case, based on what is known (and unknown) about the current user context, and / or using various other criteria.

[0009] The method further involves providing the aggregated data to facilitate a user activity prediction associated with a current user activity based on contextual information about a current user state.

[0010] Some implementations involve a wearable device that obtains current contextual information and uses that information along with the aggregated prior activity data (and contextual data associated therewith) to generate a current user activity prediction. In some implementations, an electronic device has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes. The device performing the method may be a wearable device such as an HMD (e.g., a head- worn device that provides passthrough video, a head-worn device that includes a see- through portion such as augmented reality glasses (AR glasses), etc.), an optical see through (OST) device, artificial intelligence (A / I) glasses (e.g., integrated with generative Al assistance for providing real time translation, contextual answers and summarization), (AR) glasses, mixed reality glasses, connected glasses, etc.). The device may be used on conjunction with, but separate a wearable device being worn by a user.

[0011] The method involves obtaining current contextual information corresponding to a current user activity of a user with respect to one or more elements in a physicalAttorney Docket No. 097425-01485(P68856WOl) environment. The current contextual information may be based on sensor data obtained via the one or more sensors of a wearable device. The method further involves generating a current user activity prediction based on the current contextual information and aggregated prior activity data for a plurality of prior activities of the user in one or more physical environments. The aggregated prior activity data may have been aggregated based on prior contextual information. The prior contextual information for each respective prior activity may have been generated based on sensor data obtained via one or more sensors of one or more devices worn by the user and corresponding to a respective relationship between the user and the one or more physical environments during the respective prior activity. As example, generating the current user activity prediction may involve predicting that the user is “cooking breakfast” based on the current context of the user is standing at the counter mixing cookie dough matching the context of a cluster associated with a user “cooking breakfast” activity, where that cluster is generated based on identifying that activity occurring in that context on multiple prior occasions. The method further involves enhancing a user experience based on the current user activity prediction.

[0012] In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.Attorney Docket No. 097425-01485(P68856WOl)BRIEF DESCRIPTION OF THE DRAWINGS

[0013] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be had by reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings.

[0014] Figure 1 illustrates an exemplary electronic device operating in a physical environment, in accordance with some implementations.

[0015] Figures 2A-B illustrate an exemplary view of passthrough video of a physical environment and an example of virtual content to be added to such a view to provide a view of an XR environment, in accordance with some implementations.

[0016] Figure 3 illustrates a view of an XR environment including passthrough video of a physical environment and virtual content, in accordance with some implementations.

[0017] Figure 4 illustrates locations of detected user activities within a physical environment, in accordance with some implementations.

[0018] Figure 5 illustrates aggregating of the detected user activities of Figure 4, in accordance with some implementations.

[0019] Figure 6 illustrates contextual information that may be used to cluster associated user activities, in accordance with some implementations.

[0020] Figure 7 illustrates contextual information that may be used to cluster associated user activities, in accordance with some implementations.

[0021] Figure 8 illustrates contextual information related to view direction during user activities, in accordance with some implementations.

[0022] Figure 9 is a flowchart illustrating an exemplary method for facilitate a user activity prediction based on storing and aggregating user activity information, in accordance with some implementations.

[0023] Figure 10 is a flowchart illustrating an exemplary method for predicting a user activity based on current contextual information and prior contextual information associated with prior user activities, in accordance with some implementations.Attorney Docket No. 097425-01485(P68856WOl)

[0024] Figure 11 is a block diagram of an electronic device adjusting a camera during passthrough in accordance with some implementations.

[0025] In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.DESCRIPTION

[0026] Numerous details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings merely show some example aspects of the present disclosure and are therefore not to be considered limiting. Those of ordinary skill in the art will appreciate that other effective aspects and / or variants do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the example implementations described herein.

[0027] Figure 1 illustrates an example environment 100 including an exemplary electronic device 105 operating in a physical environment 100. In the example of Figure 1 , the physical environment 100 is a room that includes a desk 120 and a window 150 on wall 130. The electronic device 105 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about and evaluate the physical environment 100 and the elements within it, as well as information about the user 102 of the electronic device 105.

[0028] In some implementations, sensor information from the electronic device 105 provides information about the physical environment 100 and / or user 102 that may be used to identify the current location of the physical environment 100, the positional relationship between the user and the physical environment 100 (e.g., coordinates within a mapping of the physical environment 100, positional relationships relative to particular objects or otherAttorney Docket No. 097425-01485(P68856WOl) elements of the physical environment 100, the view direction relative to the physical environment 100 and / or contextual information related to user activities). In some implementations, sensor information from the electronic device 105 provides information about the user 102 relative to the elements in physical environment 100. For example, sensor data from one or more outward and / or user facing cameras may be used to identify elements within the physical environment, the user’s positional relationship to such elements, how the user is interacting with those elements (e.g., sitting on the couch, standing next to the kitchen table, etc.). Sensor data may similarly be used to obtain contextual information while a user engages in one or more activities in the physical environment 100.

[0029] In some implementations, the electronic device 105 does not include a display and the user views the physical environment 100 directly, e.g., using optical see-through components and / or through transparent lenses on the electronic device 105. In some implementations, such lenses are not configured to display content. In some implementations, such lenses are configured to display content (e.g., presenting an extended reality (XR) environment by displaying augmentations or other virtual content (that augments the user’s view of the physical environment 100) using optical waveguides that transmit light to display content on the lenses). In some implementations, the electronic device 105 includes a display that presents views of the physical environment 100 that are based on images captured by outward- facing cameras on the electronic device 105, e.g., by providing passthrough video, with or without added virtual content. In some implementations, the display presents an XR environment by displaying augmentations or other virtual content (that augments displayed depictions of the physical environment 100).

[0030] In various implementations, views of an XR environment may be provided via the electronic device 105. Such an XR environment may include passthrough video views of a 3D environment (e.g., the proximate physical environment 100) that are generated based on camera images and / or depth camera images of the physical environment 100. Such an XR environment may include virtual content that is positioned at 3D locations relative to a 3D coordinate system associated with the XR environment, which may correspond to a 3D coordinate system of the physical environment 100.Attorney Docket No. 097425-01485(P68856WOl)

[0031] Figure 2A illustrates an exemplary view 205 of an XR environment in which the view 205 provides only passthrough video of a physical environment 100, including a depiction 220 of desk 120 and a depiction 250 of window 150. In one example, electronic device 105 (Figure 1) may have one or more outward- facing cameras or other image sensors that captures images (e.g., video) of the physical environment 100 that are sequentially displayed on a display of the electronic device 105. The video may be displayed in real-time and thus can be considered passthrough video of the physical environment 100. The video may be modified, e.g., warped or otherwise adjusted, to correspond to a viewpoint of an eye of the physical environment, e g., so that the user 102 sees the passthrough video of the physical environment from the same viewpoint that user would view the physical environment from if not wearing the electronic device 105 (e g., seeing physical environments directly with their eye). In some implementations, passthrough video is provided to each of the user’s eyes, e.g., with a single outwardfacing camera capturing images that may be warped / altered to provide a video from the viewpoint of each eye or from multiple outward-facing cameras that provide image data (warped or un- warped) to each eye’s viewpoint respectively.

[0032] The view 205 may be provided by a device such as electronic device 105 having a display that provides substantially all of the light visible by an eye of the user. For example, the electronic device 105 may be an HMD having a light seal that blocks ambient light of the physical environment 100 from entering an area between the device 105 and the user 102 while the device is being worn such that the device’s display provides substantially all of the light visible by the eye of the user. A device’s shape may correspond approximately to the shape of the user’s face around the user’s eyes and thus, when worn, may provide an eye area (e.g., including an eye box) that is substantially sealed from direct / ambient light from the physical environment.

[0033] In some implementations, a view of an XR environment includes only depictions of a physical environment such as physical environment 100. A view of an XR environment 100 may be entirely passthrough video. A view of an XR environment may depict a physical environment based on image, depth, or other sensor data obtained by the device, e.g., generating a 3D representation of the physical environment based on such sensor data and then providing a view of that 3D representation from a particularAttomey Docket No. 097425-01485(P68856WOl) viewpoint. In some implementations, the XR environment includes entirely virtual content, e.g., an entirely virtual reality (VR) environment that includes no passthrough or other depictions of the physical environment 100. In some implementations, the view of the XR environment includes depictions of both virtual content and depictions of a physical environment 100.

[0034] Figure 2B illustrates an example of virtual content 230 to be added to the view 205 of the XR environment. In this example, the virtual content 230 includes a user interface 232 (e g., of an app) that includes a background area 235 and icons 242, 244, 246, 248. In this example, the virtual content 230 is approximately planar. In other implementations, virtual content 230 may include non-planar content such as 3D elements.

[0035] Figure 3 illustrates a view 305 of an XR environment including passthrough video of the physical environment 100 and virtual content 230. In this example, the view 305 includes passthrough video including depiction 220 of the desk 120 of the physical environment, as well as virtual content including user interface 232. The virtual content (e.g., user interface 232) may be positioned within the 3D coordinate system as the passthrough video such that the virtual content appears at a consistent position (unless intentionally repositioned) within the XR environment. For example, the user interface 232 may appear at a fixed position relative to the depiction 220 of the desk 120 as the user changes their viewpoint and views the XR environment from different positions and viewing directions. Thus, in some implementations, virtual content is given a fixed position within the 3D environment (e.g., providing world-locked virtual content). In some implementations, virtual content is provided at a fixed position relative to the user (e.g., user-locked virtual content), e.g., so that the user interface 230 will appear to the user remain a fixed distance in front of the user, even as the user moves about and views the environment with virtual content added from different viewpoints.

[0036] Providing a view of an XR environment may utilize various techniques for combining the real and virtual content. In one example, 2D images of the physical environment are captured and 2D content of virtual content (e.g., depicting 2D or 3D virtual content) is added (e.g., replacing some of the 2D image content) at appropriate places in the images such that an appearance of a combined 3D environment (e.g.,Attorney Docket No. 097425-01485(P68856WOl) depicting the 3D physical environment with 2D or 3D virtual content at desired 3D positions within it) is provided in the view. The combination of content may be achieved via techniques that facilitate real-time passthrough display of the combined content. In one example, the display values of some of the real image content is adjusted to facilitate efficient combination, e.g., changing the alpha values of real image content pixels for which virtual content will replace the real image content so that a combined image can be quickly and efficiently produced and displayed.

[0037] Some implementations obtain contextual information, during a user experience while the user is wearing a wearable electronic device, based on sensor data provided by sensors on the wearable device. Contextual information may be obtained and used to predict the user’s current activity. Some implementations interpret wearable device sensor data using computer vision, algorithms, or machine learning techniques to detect what the user is doing in their environment based on the user’s relationships to elements that are identified in the environment. Predicting the user’ s current activity may be based on the user’s prior activities in the same physical environment. Contextual information about the user’s prior activities in the physical environment may be tracked, stored, and / or aggregated to facilitate such predictions. This aggregating of prior user activity information may involve clustering prior user activities (i.e., events) based on their contextual similarities, e.g., same or similar user activity types, events occurring while the user is in similar locations relative to elements in the environment, events occurring when the user is looking at the same elements or in the same directions, events occurring when the user is making similar body movements, events occurring when the user has a similar state (e.g., sitting, standing, walking, etc.), events occurring at similar times of day or days of the week, etc.

[0038] Context” or “contextual information” may be specific to spatial user activities and may be collected, stored, and used to make predictions to improve the user experience or modify the device during such a user experience (e.g., optimizing the device to enhance the activity (or activities) that the user is predicted as most likely to be doing). The contextual information may be specific to spatial user activities relative to the physical environment, which may be distinguished from user actions intended to be input to the wearable device, i.e., for purposes of tracking, the user’s activity relative toAttorney Docket No. 097425-01485(P68856WOl) environmental elements may be distinguished from user actions (e.g., the user using gaze, other gestures, speech, etc. as intentional user input to the wearable device). The contextual information may be limited to environmental characteristics related to the user’s activities related to environmental elements. For example, the contextual information about a user using a wearable device to look up a recipe may identify that the user did so while the user was in the kitchen, standing next to the counter, with the refrigerator door opened, etc.

[0039] Some implementations provide an architecture for storing, aggregating, and using contextual information about user activities during wearable device usage. Such an architecture may involve one or more producers, one or more aggregators, and / or one or more consumers that provide information to one another to facilitate collection and user of a robust data set of user information that achieves various benefits. Such an architecture may facilitate various benefits including, but not limited to, improving efficiency, improving accuracy, enhancing user experience, and / or protecting user privacy.

[0040] In such an architecture, the producers can include one or more devices in a physical environment (wearable or not) that obtain sensors data about user activities in that physical environment. Such information may identify user activities (e.g., events) and the contextual information associated with those activities. Producers can be wearable devices including, but not limited to, HMDs, watches, rings, bracelets, earphones, headsets, clothing, buttons, shoes, etc. Producers can be environment devices including, but not limited to, speakers, televisions, laptops, desktops, webcams, appliances, smart furniture, smart thermostats, smart lights, security cameras, baby / person monitors, smart doorbells, cameras, etc. Producers can be applications executing on devices, e.g., an exercise application, a productivity application, etc. that are associated with particular types of activities, e.g., exercising, working, etc. When the user is using such applications, the associated user activities may be identified as being associated with contextual information about the user’s state and environmental circumstances during that use (e.g., where the user is spatially, what time it is, the user state, how the user is moving, etc.).Attorney Docket No. 097425-01485(P68856WOl)

[0041] Various kinds of components may be producers. As examples, a calendar application may be a producer by contributing information about user activities. A device may be a contributor by identifying which app or apps are open and / or in use. Devices may contribute by contributing any information that is useful in understanding a user’s activity and / or associated context. Information may be available from many sources throughout the user’s day (and / or night) about a user’s activities. In another example, communication applications (e.g., messaging apps, social media apps, etc.) provide information about a user, e.g., based on what a user is texting, reading, communicating, etc., including not limited to information about what a user is currently doing or about to do.

[0042] It may be beneficial to filter information based, for example, on relevancy criteria, to improve system efficiency and performance. For example, information about only certain types of activities and / or only certain types of information from certain devices may be stored. In some implementations, the system determines (e.g., via machine learning or non-learning-based algorithm) over time which types of user activities are frequently performed or otherwise important to the user and selectively stores information about only those features. In some implementations, the system selectively stores information about a user activity only after determining that the user activity occurred in a similar context more than a threshold number of times. The system may intelligently determine what is most important to the user and may evolve over time as the user’s interests, activities, and environment change.

[0043] The aggregator processes may include one or more devices (in the physical environment or elsewhere and wearable or not) configured to cluster or otherwise aggregate such data. For example, an aggregator may aggregate stored user activity information via a matching or clustering process (e.g., using spatial information and / or statistics to determine that a user tends to do x, y, and z activities (with certain respective probabilities) in certain contexts). The aggregation processes may assess stored user activity information and associated contextual information to better understand the user and / or enable predictions about what the user is doing. It may determine statistics, e.g., determining that statistically when the user is at a first desk they are working 3% of the time and when the user is at a second desk they are working 98% of the time. It mayAttorney Docket No. 097425-01485(P68856WOl) aggregate information about the user activity in ways that enable determining statistically if and / or when particular user activities are more likely to be occurring. It may use dense / robust information about prior user activities to enable predictions about user behaviors in a way that would not be possible if extensive data about the user’s prior activities in the physical environment where not available.

[0044] Predictions of a user’s current activity may be used for various purposes. Thus, the consumers may include one or more devices (wearable or not) that predict a user’s current activity given the current context of the user using a wearable device in the physical environment. Examples of consumers include point of view correction (POVc) algorithms, environment hazard / collision warning algorithms, automatic application launching algorithms, applications that provide user experiences, etc.

[0045] Characteristics of a wearable device may be optimized or configured based on knowing (e.g., with a determined degree of confidence) what the user is doing. For example, some wearable device may provide a point of view POVc that is optimized based on some understanding of the user context and / or what the user is doing. For example, POVc processes may balance use of subprocesses that improve image quality (e.g., providing higher resolution corrections) and improve user comfort (e.g., providing corrections that reduce warping, improving registration between virtual / physical objects, etc.). The POVc processes may be adjusted / configured to favor one or the other (quality v. comfort) based on the predicted current user activity. For example, the resolution of the depth map used in such processes can be lowered or raised based on what the user is doing, resulting in smoother or more edgy appearances respectively. It may be selectively utilized and / or configured depending upon the expected importance of such appearance features to the user in different user activities.

[0046] POVc parameters affecting comfort, image quality, or other characteristics may be adjusted based on user activity. If the predicted current user activity, for example, is one in which the user tends to look at or otherwise focus most of their attention on the user’s desk and objects thereon, user comfort may be prioritized. Conversely, if the predicted current user activity, for example, is one in which the user is expected to be relatively more preoccupied with the aesthetics of the environment, the POVc process may be tuned to prioritize image quality. POVc processes may be computationallyAttorney Docket No. 097425-01485(P68856WOl) expensive and adapting sub-processes to appropriately balance image quality with comfort for different user activities can provide significant advantages with respect to their perceived effectiveness and efficiency.

[0047] Some implementations may generate, store, and / or aggregate user activity contextual information by spatializing information about the user relative to the user’s physical environment. The sensors on one or more wearable devices may provide information that enables determining, for example, where the user is relative to elements in the physical environment and what the user is doing while in those locations. For example, the contextual information may identify where the user is in the physical environment when the user opens or performs a particular interaction with a particular app provided via an HMD. The contextual information may identify whether the user is sitting or standing when the user opens the app or performs the particular interaction with the app. User activity information can be determined and stored as a form of spatial “breadcrumbs” in the space. Over time, such spatialized contextual information about user activities provides data from which statistical or other predictions about user activities can be based. Such spatialized contextual information may reveal that specific user events occur more often in certain positions within and / or relative to certain elements of the physical environment.

[0048] Relationships with elements of a physical environment may be determined based on understanding the physical environment based on sensor data, e.g., image and / or depth sensor data from one or more wearable or other electronic devices in the physical environment. In some implementations, a scene understanding and / or semantic understanding are generated to identify the types of objects and the characteristics of those objects of the physical environment. A user’s relationships to furniture and other elements can then be captured as contextual information when user activities are detected.

[0049] Signals or other information from multiple devices in a physical environment may be fused to provide a more robust set of contextual information than would otherwise be available from a single device. For example, smart home devices (e.g., smart speakers, smart TVs, laptops, desktops, etc.) may provide additional contextual information aboutAttorney Docket No. 097425-01485(P68856WOl) the physical environment while user activities occur with a user wearing a wearable device in that environment.

[0050] Some implementations utilize biometric information about the user that can be associated with the user activities, e.g., using information about the user’s heartrate, body temperature, stress level, etc., from a watch device while the user uses an HMD.

[0051] Some implementations aggregate prior user activity and the associated contextual information using aggregations criteria that identifies similar contextual circumstances. Some implementations cluster such data based on similarities of one or more factors, e.g., location, proximity to environmental elements of a particular type, proximity to a particular environmental element, user / device view direction, user state, type of user motion, amount of user motion, proximity to activity in the physical environment, etc.

[0052] Some implementations facilitate storage and aggregation of user activities data (including contextual information) by providing a consistent format in which multiple producers can use to contributor information. Such a format may, for example, require spatial locations of user activities be specified in a particular way or relative to a particular mapping. Such a format may, for example, specify that user activities be specified using a predetermined list of user activity types (e.g., sitting, standing, walking, working, cooking, cleaning, watching, exercising, meditating, etc.), a predetermined list of relationship types (e.g., sitting on, looking at, facing, near, within distance X of, etc.), a predetermined list of object types (e.g., table, chair, refrigerator, desk, laptop, etc.), and / or a predetermined list of object characteristics (e.g., red, yellow, wood, type of table, etc.). Such a format may enable a user activity to be represented in a graph or other data structure that enables the user activities to be assessed, compared, clustered, and otherwise aggregated with other user activities.

[0053] Information may, but need not, necessarily be stored at a central location (e.g., on a HMD, server, laptop, home hub, etc.). Such a central location may ensure privacy of the information, e.g., providing only the central device or process with the abilities to have access to information provided by various contributors and / or aggregate such information.Attorney Docket No. 097425-01485(P68856WOl)

[0054] The architecture may specify which types of user activities are to be tracked and which types are not to be tracked (e.g., accepting information about the user watching TV, interacting with an appliance, manipulating physical objects, engaging in an exercise pattern / routine, etc. but not accepting information about the user simply walking down the hallway, waving their handjumping up and down once, etc.). The architecture and / or data format requirements may ensure consistent spatial-temporal contextual information is stored and available for aggregation.

[0055] Some implementations facilitate collection and storage of information that is associated with spatial context event anchors to enable such information to be aggregated spatially and / or using other contextual information later. Such spatial context event anchors may identify a location within a physical space (or location relative to an element of the physical space) at various levels of specificity, which may depend upon the type of the device that is contributing the device. For example, an HMD may have access to detailed information about the 3D geometry and / or spatial locations of elements within a physical environment, as well as the user’s location relative to that environment or the elements within in. Conversely, the user’s watch may have more limited information about location or may rely on other devices to provide the location but can still contribute information that will be associated with a use activity (e.g., the watch knows the user’s heartrate and / or exercise type at a given point in time). Another device worn by the user or in the environment may provide the position of the user at that point in time, and the information together can be contributed as a user activity event with contextual information that specifies the user’s positional relationship to one or more elements in the environment. Some implementations identify a user’s spatial position based on identifying the closest object to the user in any direction and / or the closest object to the user in a particular direction, such as in the direction the user is facing.

[0056] Some implementations store historical information about spatial and other contextual information about a user’s activities in a physical environment to enable inferenced to be drawn about the user’s likely activities in later circumstances. Based on a user’s current context, predictions can be generated regarding the likelihood(s) that the user is currently engaging in one or more user activities that the user had engaged in previously.Attorney Docket No. 097425-01485(P68856WOl)

[0057] Figure 4 illustrates locations of detected user activities within a physical environment. In this example, spatial context event anchors 410a-d, 420a-c, 430a-d, and 440a-d are depicted at positions relative to elements of the physical environment 400. These spatial context event anchors 410a-d, 420a-c, 430a-d, and 440a-d each correspond to a different user activity detected in the physical environment 400. In this example, the physical environment includes multiple rooms (e.g., home office 415, living room 405, kitchen 435, etc.) and physical elements (e.g., furniture items, windows, countertops, appliances, rugs, walls, corners, doorways, etc.). Electronic devices, such as electronic device 105, may have generated user activity data for user activities (e g., events) that identifies user states, motions, actions, etc., that are associated with the spatial context event anchors 410a-d, 420a-c, 430a-d, and 440a-d. Thus, the detected user activities may be associated with spatial information identifying 2D or 3D spatial positions at which the user activities occurred using coordinates, e.g., xy or xyz coordinates, and / or using relationships to elements of the physical environment, e g., identifying that the activity occurred next to the couch, on the rug, etc. The spatial information provides contextual information about the user activities. Additional contextual information may be tracked and associated with the user activities, e.g., time of day, lighting conditions, user view direction, whether environmental devices are on or off, content or applications active on nearby devices, the presence of other persons, the presence of other device users, the presence of one or more pets, the ambient noise level, the type of ambient noise, whether other persons are talking, etc. The user’s state, motions, and / or actions may be used to classify the user’s activity (e.g., exercising, working, watching TV, conversing, etc.), which may be stored with the user activity information. The user’s state, motions, and / or actions may additionally, or alternatively, be stored as contextual information, e.g., identifying the user’s heart rate, gaze direction, EEG data, hand movements, hand gestures, body movements, body gestures, whether the user is sitting, standing, walking, etc. Contextual information for the user activities associated with spatial context event anchors 410a-d, 420a-c, 430a-d, and 440a-d may be generated based on sensor data from one or more wearable devices worn by the device and / or based on sensor data from other devices in the physical environment 400, as described herein.Attorney Docket No. 097425-01485(P68856WOl)

[0058] Figure 5 illustrates clustering of the detected user activities of Figure 4. In this example, the user activities associated with spatial context event anchors 410a-d are aggregated into cluster 510, the user activities associated with the spatial context event anchors 420a-c are aggregated into cluster 520, the user activities associated with the spatial context event anchors 430a-d are aggregated into cluster 530, and the user activities associated with the spatial context event anchors 440a-d are aggregated into cluster 540. Such clustering may utilize clustering criteria that accounts for spatial context event anchor positioning (e.g., coordinates and / or relationships to particular elements of the physical environment 400 or positions therein) and / or other contextual information. For example, the user activities associated with spatial context event anchors 410a-d may be clustered into cluster 510 based on determining that each activity involved the spatial context of the user sitting on the couch 511 on Saturday mornings. As another example, the user activities associated with spatial context event anchors 420a-c may be aggregated into cluster 520 based on determining that each activity involved the user working while sitting at desk 521. As another example, the user activities associated with spatial context event anchors 430a-d may be aggregated into cluster 530 based on determining that each activity involved the user exercising while laying or sitting on rug 531.

[0059] Figure 6 illustrates contextual information that may be used to cluster associated user activities. In this example, the contextual information for user activities includes information about biome events (i.e., living area events) that are associated with spaces in which they occur. In some implementations, based on an area being identified as a biome (i.e., living area), a graph representing elements and / or activities associated with the biome is created. Information about the user activities may be stored and / or aggregated to build up a rich, personalized spatial context dataset.

[0060] As illustrated in Figure 6, such data may provide information about user state during activities that occur while the user has a positional relationship relative to a particular element of the physical environment 400, e.g., activities that occur while the user is sitting on a sofa 511 (e.g., the user activities associated with spatial context event anchors 410a-d). As illustrated in Figure 6, such information may identify the user’s posture during those user activities (e.g., how frequently the user is laying, sitting, orAttorney Docket No. 097425-01485(P68856WOl) standing), the user’s motion (e.g., how frequently the user is active, moderately active, or stationary), and / or the type of activity (e.g., how frequently the user is eating, conversing, and / or reading).

[0061] Figure 7 similarly illustrates contextual information used to cluster associated user activities. As illustrated in Figure 7, such data may provide information about user state during activities that occur while the user has a positional relationship relative to a particular element of the physical environment 400, e.g., activities while the user is near the desk 521 (e.g., the user activities associated with spatial context event anchors 420a- c). As illustrated in Figure 7, such information may identify the user’s posture during those user activities (e.g., how frequently the user is laying, sitting, or standing), the user’s motion (e.g., how frequently the user is active, moderately active, or stationary), and / or the type of activity (e.g., how frequently the user is eating, conversing, and / or reading).

[0062] Figure 8 illustrates contextual information related to view directions during user activities. In this example, user activities 805, 810, 815, 820, 825 have been identified at locations within a physical environment and corresponding view directions are associated with those events / locations. View direction 806 is associated with user activity 805, view direction 811 is associated with user activity 810, view direction 816 is associated with user activity 815, view direction 821 is associated with user activity 820, and view direction 826 is associated with user activity 825. A view direction may be based on a user’s actual gaze direction and / or the direction in which the user’s head / eyes are facing (e.g., where an HMD is facing). In cases where the user activity occurs where the user is wearing an HMD, the view direction may be determined based on sensors on the HMD, e.g., motion sensors and / or image sensors. Information from other devices may replace or supplement this information. In some implementations, the HMD tracks its own pose (position and orientation), which is used to determine the view direction during user activities. In circumstances in which the user is not wearing an HMD, the information regarding view direction may come from other devices, e.g., image sensors on other devices that capture images depicting the head and / or eyes of the user from which a view direction can be predicted.Attorney Docket No. 097425-01485(P68856WOl)

[0063] View direction may be used to facilitate aggregation of user activity information. In the example of Figure 8, user activities 805, 825 involve view directions 806, 826 that are both directed at a common environment element 840 and may be aggregated together based at least in part on this commonality. Similarly, user activities 815, 820 involve view directions 816, 821 that are both directed at a common environment element 850 and may be aggregated together based at least in part on this commonality. Two user activities may be aggregate together based on spatial and / or view direction criteria, e.g., aggregate if within X feet of one another and viewing the same object, aggregate if sitting on a common object and viewing a desk area, etc. In some implementations, spatial context event anchors are generated based on user activities and view dependent anchors are built on top of those spatial context event anchors, e.g., storing additional view direction information for user activities when such information is available. The inclusion of view direction information may be used to enable the aggregation of user activities even when a user’s spatial relationships are not matched, e.g., aggregating activities when a user is on opposite sides of the room but that involve the user looking at the same object.

[0064] In some implementations, user activity contextual information is selected based on identify salient objects and / or objects that are the target of user focus (or interaction) within a physical environment.

[0065] In some implementations, the system or processes that collect, manage, and / or aggregate user activity contextual data are separate from consumers (e.g., applications or processes) that use information therefrom. The applications or processes that use the information may have access to limited information (e.g., without details of user prior activities) to preserve the user’s privacy while still enabling those applications to provide enhanced services. For example, a mindfulness application may query the system for information about the user’s current activity and receive a response that there is an 80% chance the user is about to meditate. The mindfulness application may then trigger notifications, begin a meditation function, etc. without the mindfulness application itself ever having access to details about the user’s current activities or environment (e.g., without it knowing that the user is sitting on a rug in the family room with arms on kneesAttorney Docket No. 097425-01485(P68856WOl) at 3pm on a Sunday with eyes closed) and / or the user’s prior activities (e.g., without it knowing that the user previously meditated in a similar context on 7 prior occasions).

[0066] In some implementations, a system or process that collects, manages, and / or aggregates user activity contextual data uses data received from a separate application (e.g., a consumer) to determine how to aggregate its stored data to respond to a query (e.g., a query from the separate consumer application, process, or device). For example, the consumer application may ask to be notified when the user is likely performing a particular activity in a particular context. For example, a mindfulness application could query the system to identify (i.e., be notified) when a user likely about to use the application to meditate based on contextual circumstances in which the user previously used the medication application. The system or process that collects, manages, and / or aggregates user activity contextual data may then aggregate that prior user activity data and monitor the user’s current circumstance over time for similarities and, when similarities are identified, notify the requesting application. As another example, the mindfulness application may query the system to identify circumstances in which the user is sitting on a living room sofa and is exhibiting biometric indications that a break may be beneficial, and receive a response when the user’s current activity matches those criteria.

[0067] An application may provide information that identifies what kinds of user activities and / or context are relevant to it. An application may provide information in a variety of formats that may be used to aggregate prior user activity contextual information to make predictions about the user. It may provide rules regarding how spatial temporal clustering should be applied. Different applications may provide different information (e.g., rules, aggregation criteria, query formats, etc.) that can be accommodated by the system. For example, the un-aggregated data may be stored and available for aggregation in response to application queries, e.g., in real time, using such information.

[0068] Information about a user’s state, activity, and / or physical environment may be provided to an application (e.g., to a consumer requesting such information) based on the user having given prior permission for such information to be provided to the application.Attorney Docket No. 097425-01485(P68856WOl)An event handling framework and distribution framework can be used to facilitate use of user activity predictions in a way that corresponds to the user’s expectations.

[0069] Some implementations disclosed herein collect spatial usage information about a user, including user activities and / or biometric information at particular locations in the physical environment and use statistical capabilities to aggregate that information over time and / or for particular purposes. The purposes that the information will be used for need not be known when the prior user activity and associated context information is collected, stored, and / or compiled. The statistical analysis / clustering may use pooling and filtering to distinguish relevant events from irrelevant events for particular purposes.

[0070] Some implementations provide a separated architecture (e.g., providing a central (or separated) location that stores spatial information from multiple sources. For example, such an architecture may involve: producers who contribute info about user activities / state in particular spatial locations; aggregators that determine statistics / clustering about which activities / states the user does / has in particular spatial locations; and / or consumers that use that info to change system settings or provide / enhance user experiences.

[0071] Some implementations utilize semantic statistics / clustering of spatial user state / activity information (e.g., clustering based on proximity to particular type of object, in particular type of room, etc.).

[0072] Some implementations provide view-dependent features, e.g., using or aggregating view-dependent anchors built on top of location anchors.

[0073] Some implementations enable querying / aggregating the stored user information based on consumer needs.

[0074] Some implementations use stored spatial user activity information including contextual information to predict what the user is about to do based on location and other context clues.

[0075] In some implementations, storing user activity information that combines information about time, pose (e.g., view direction), and the physical environment (e.g.,Attorney Docket No. 097425-01485(P68856WOl) semantic mapping-based info and relationships) enables more robust understanding of the user that can be used to enable numerous desirable user experiences.

[0076] User activity predictions can be used to improve the efficiency, effectiveness, and usability of many features provided by electronic devices.

[0077] POVc corrections may be initiated and / or customized to account for the user’s current activity.

[0078] Similarly, collision avoidance may be customized to account for the user’s current activity. For example, based on determining that the user is sitting on the couch, the system may reduce collision avoidance sensitivity on an HMD since there is less chance that a user collision (e.g., with a wall or another person) will occur while the user is sitting on the couch.

[0079] User activity information may additionally be used to improve user experiences with respect to using applications. The system may proactively adapt what applications are loading and / or set up specific configurations for those applications based on the user’s current activity.

[0080] User activity information may be used to improve device features related to health and wellness, for example, using information about the user currently climbing the stairs or putting on running shoes on the sofa to launch activity tracking functionality.

[0081] User activity information may be used to conserve device power, e.g., enabling only functionality that is likely to relevant for the user’s current (and / or upcoming) activities. Moreover, the system may be able to predict upcoming activities and initiate resources that support or are otherwise appropriate for those user activities at appropriate times before they occur, so the user need not wait for the features to initialize when they are ready to use them.

[0082] User activity information may be used to enhance device interaction capabilities. For example, a device may respond to verbal inquires from a user. Digital assistants have historically required users to provide relatively precise and detailed queries because limited or no contextual information may be available to help interpret brief or generic queries. Tn contrast, systems and techniques disclosed herein enableAttorney Docket No. 097425-01485(P68856WOl) better understanding of the user’s current activities and associated contextual information. Such information may enable a personal / digital assistant device or process to respond to inquires from a user with less information being provided by the user and / or in ways that reflect a better understanding of the context of the user’s query. In one example, a user can simply ask the digital assistant “please help me with this” or just “help me,” and the system may understand the user’s current activity and, accordingly, provide appropriate assistance. Implicit contextual information that is recognized in human-to-human interactions may be available to the digital assistant to enable it to provide a response that is more focused on what a user is really asking for and thus provides a user interaction that may be a more natural / human experience for the user. A generative Al user experience may similarly be enhanced, e.g., by supplementing a generative prompt provided by a user with supplemental information about the user’s current activities and / or associated contextual information.

[0083] Some implementations track user activity information to better understand multiple users in a physical environment. Some implementations track information to understand multiple users doing similar activities, e.g., within a factory multiple workers may perform the same tasks again and again. Techniques used herein may be used to collect contextual information about such user activities, from which insights may be gleaned. For example, accumulated data may be used to optimize workflow, identify where things are not working as expected, or identify environmental changes that can be implemented to improve results.

[0084] In some implementations, one or more electronic devices identify user activities and contextual information associated therewith. These devices may use various techniques to identify similar things the user is doing that will be classified as the same activity (e.g., sitting, sitting at a desk, sitting in front of a user, etc.). Such determinations may involve an activity classifier that uses a limited set of activity types, e.g., only tracking the top X number of activities in which the user is most frequently involved.

[0085] Some implementations utilize a semantic understanding of environment, e.g., determining that the user is working at the desk by understanding that there is a chair under the desk in the environment, that the user’s position corresponds to the chair’sAttorney Docket No. 097425-01485(P68856WOl) location, that the user’s posture corresponds to the user sitting, that the user’s view direction corresponds to the user looking at something on or near the desk, etc.

[0086] A classifier (learning-based on non-learning based process) used to identify a user’s activity for tracking purposes may utilize a taxonomy of user activities of different levels of granularity, e.g., identifying whether the user is sleeping or awake, which is subdivided into whether the user is still or moving, which is subdivided into whether the user is running, walking, etc.

[0087] Some implementations involve one or more devices (e g., wearable, nonwearable, or both) capturing sensor data about fine-level user motions (e.g., position of the user, position of the user’s hand, etc. changing over time). Such information may be used to identify higher level user activities and how those activities are associated with one or more physical environment elements and other contextual information. Over time user activity information is compiled and used to make predictions about the user’s current activities, which may be used for numerous purposes, including feeding back into the user activity tracking to improve tracking accuracy. In other words, the information may serve to enable a feedback-mechanism for an activity recognizer used to generate the stored historical data, identifying that the user is most likely doing activity A rather than B, C, D activities at their current location and, therefore, this activity event should be stored as an activity A event.Exemplary Processes

[0088] Figure 9 is a flowchart illustrating an exemplary method for facilitate a user activity prediction based on storing and aggregating user activity information. In some implementations, the method 900 is performed by a device, such as an HMD (e.g., device 105 of Figure 1), desktop, laptop, mobile device, or server device. In some implementations, the method 900 is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method 900 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Each of the blocks in the method 900 may be enabled and executed in any order.Attorney Docket No. 097425-01485(P68856WOl)

[0089] At block 902, the method 900 involve storing activity data comprising contextual information for a plurality of activities of a user in one or more physical environments, the contextual information based on sensor data obtained via one or more sensors of a wearable device worn by the user, the contextual information for each respective activity of the activities corresponding to a respective relationship between the user and the one or more physical environments separate from the wearable device during the respective activity. The contextual information for each respective activity of the activities may correspond to a relationship between the user and one or more elements / objects in the one or more physical environments separate from the wearable device. The contextual information for each respective activity of the activities may correspond to one or more positions in the one or more physical environments.

[0090] The contextual information for one or more of the plurality of activities may comprise a relative location of the user with respect to the one or more elements / objects of the physical environment. The contextual information for one or more of the plurality of activities may comprise an identification of whether the user satisfies a positional classification (e.g., near) or threshold (e.g., within 3 feet of) of one or more elements / objects of the physical environment.

[0091] The contextual information for one or more of the plurality of activities may comprise viewing direction of the user relative to the one or more elements (e.g., identifying that the user is looking at the TV).

[0092] The contextual information for one or more of the plurality of activities may comprise information obtained from another device in the physical environment, the other device separate from the wearable device (e.g. information from a home assistant device, a watch, and / or other ecosystem devices).

[0093] The contextual information for one or more of the plurality of activities may comprise information identifying a movement of the user determined based on a motion sensor of the wearable device.

[0094] The contextual information for one or more of the plurality of activities may comprise information based on a scene understanding of a physical environmentAttorney Docket No. 097425-01485(P68856WOl) generated via computer vision using image data obtained via an image sensor of the wearable device.

[0095] The contextual information for one or more of the plurality of activities may comprise information based on a semantic understanding of a physical environment generated via computer vision using image data obtained via an image sensor of the wearable device.

[0096] The contextual information for one or more of the plurality of activities may comprise a classification of a user state. The classification of the user state corresponds a user posture, gesture, whether the user is sitting or standing, whether the user is standing or walking, or whether the user is stationary or mobile. The classification of user state is based on sensor data from a biometric sensor of the wearable device (e.g., heartrate, EEG, temperature, etc.).

[0097] At block 904, the method 900 involves generating aggregated data based on aggregating (e.g., clustering) activities of the activity data based on similarities in the contextual information. The aggregating may be based on a taxonomy of user activities in which user activities are classified and related using multiple levels of granularity. The aggregating may be based on selecting a level of granularity based on the contextual information about the current user state.

[0098] At block 904, the method 900 involves providing the aggregated data to facilitate a user activity prediction associated with a current user activity based on contextual information about a current user state. The prediction may identify one or more user activities as likely being currently performed or about to be performed by the user, the prediction comprises a likelihood that the current user activity corresponds to a user activity associated with a cluster of user activities identified via the aggregating. The prediction may comprise likelihoods that the current user activity corresponds to each user activity associated with multiple clusters of user activities identified via the aggregating.

[0099] In some implementations, a response to a user query is enabled based on the prediction associated with the current user activity. For example, a response to a userAttorney Docket No. 097425-01485(P68856WOl) query to a digital assistant “please help me” may be interpreted and / or interpreted more accurately based on understanding the user’s current activity and / or context.

[0100] In some implementations, a user interface function is provided based on determining that the user interface function is likely to be used by the user based on the prediction associated with the current user activity. For example, the method 900 may recognize that the user makes breakfast 85% of the time that they are standing in front of the kitchen counter at 8am looking at a pancake box and, in that context, frequently uses a particular user interface cook timer function and, based on recognizing this, automatically initiates the cook timer user interface function and presents an option (e.g., cook timer function icon) for the user to select to initiate its use.

[0101] In some implementations, a point of view (PoV) correction is automatically initiated based on the prediction associated with the current user activity.

[0102] In some implementations, a collision avoidance function is automatically initiated based on the prediction associated with the current user activity.

[0103] Figure 10 is a flowchart illustrating an exemplary method for predicting a user activity based on current contextual information and prior contextual information associated with prior user activities. In some implementations, the method 1000 is performed by a device, such as an HMD (e.g., device 105 of Figure 1), desktop, laptop, mobile device, or server device. In some implementations, the method 1000 is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method 1000 is performed by a processor executing code stored in a non-transitory computer- readable medium (e.g., a memory). Each of the blocks in the method 1000 may be enabled and executed in any order.

[0104] At block 1002, the method 1000 involves obtaining current contextual information corresponding to a current user activity of a user in a physical environment, the current contextual information based on sensor data obtained via the one or more sensors of the wearable device.

[0105] At block 1004, the method 1000 involves generating a current user activity prediction based on the current contextual information and aggregated prior activity data for a plurality of prior activities of the user in one or more physical environments, theAttorney Docket No. 097425-01485(P68856WOl) aggregated prior activity data having been aggregated based on prior contextual information, the prior contextual information for each respective prior activity having been generated based on sensor data obtained via one or more sensors of one or more devices worn by the user and corresponding to a respective relationship (e.g., positional, interaction, etc.) between the user and the one or more physical environments during the respective prior activity. For example, this may involve predicting that the user is “cooking breakfast” based on the current context of the user is standing at the counter mixing cookie dough matching the context of a cluster associated with a user “cooking breakfast” activity, where that cluster is generated based on identifying that activity occurring in that context on multiple prior occasions.

[0106] The prior contextual information for each respective prior activity of the activities may correspond to a relationship between the user and one or more objects in the one or more physical environments separate from the one or more devices worn by the user during the respective prior activity.

[0107] The prior contextual information for each respective prior activity of the activities may correspond to one or more positions in the one or more physical environments.

[0108] The contextual information for one or more of the plurality of prior activities may comprise a relative location of the user with respect to one or more elements of the physical environment.

[0109] The contextual information for one or more of the plurality of prior activities may comprise an identification of whether the user satisfies a positional classification (e.g., near) or threshold (e.g., within 3 feet of) one or more elements of the physical environment.

[0110] The contextual information for one or more of the plurality of prior activities may comprise viewing direction of the user relative to one or more elements of the physical environment (e.g., looking at the TV).

[0111] The contextual information for one or more of the plurality of prior activities may comprise information obtained from another device in the physical environment, the another device separate from the one or more wearable devices.Attorney Docket No. 097425-01485(P68856WOl)

[0112] The contextual information for one or more of the plurality of prior activities may comprise information identifying a movement of the user determined based on a motion sensor of the one or more wearable devices.

[0113] The contextual information for one or more of the plurality of prior activities may comprise information based on a scene understanding of a physical environment generated via computer vision using image data obtained via an image sensor of the one or more wearable devices.

[0114] The contextual information for one or more of the plurality of prior activities comprises information based on a semantic understanding of a physical environment generated via computer vision using image data obtained via an image sensor of the one or more wearable devices.

[0115] The contextual information for one or more of the plurality of prior activities may comprise a classification of a user state. The classification of the user state may correspond to a user posture, gesture, whether the user is sitting or standing, whether the user is standing or walking, or whether the user is stationary or mobile. The classification of user state is based on sensor data from a biometric sensor of the one or more wearable devices (e.g., heartrate, EEG, temperature, etc.).

[0116] The aggregating may be based on a taxonomy of user activities in which user activities are classified and related using multiple levels of granularity. The aggregating may be based on selecting a level of granularity based on the contextual information about the current user state.

[0117] The prediction may comprise a likelihood that the current user activity corresponds to a user activity associated with a cluster of user activities identified via the aggregating. The prediction may comprise likelihoods that the current user activity corresponds to each user activity associated with multiple clusters of user activities identified via the aggregating.

[0118] At block 1006, the method involves enhancing a user experience based on the current user activity prediction. A response to a user query may be enabled based on the prediction associated with the current user activity. A user interface function may be provided based on determining that the user interface function is likely to be used by theAttorney Docket No. 097425-01485(P68856WOl) user based on the prediction associated with the current user activity. A point of view (PoV) correction may be automatically initiated based on the prediction associated with the current user activity. A collision avoidance function may be automatically initiated based on the prediction associated with the current user activity.

[0119] Figure 11 is a block diagram of an example device 1100. Device 1100 illustrates an exemplary device configuration for electronic device 105 of Figures 1. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the device 1100 includes one or more processing units 1102 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and / or the like), one or more input / output (I / O) devices and sensors 1104, one or more communication interfaces 1108 (e g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.1 lx, IEEE 802.14x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or the like type interface), one or more programming (e.g., I / O) interfaces 1110, output devices (e.g., one or more displays) 1112, one or more interior and / or exterior facing image sensor systems 1114, a memory 1120, and one or more communication buses 1104 for interconnecting these and various other components.

[0120] In some implementations, the one or more communication buses 1104 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I / O devices and sensors 1106 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), one or more cameras (e.g., inward facing cameras and outward facing cameras of an HMD), one or more infrared sensors, one or more heat map sensors, and / or the like.

[0121] In some implementations, the one or more displays 1112 are configured to present a view of a physical environment, a graphical environment, an extended realityAttorney Docket No. 097425-01485(P68856WOl) environment, etc. to the user. In some implementations, the one or more displays 1112 are configured to present content (determined based on a determined user / object location of the user within the physical environment) to the user. In some implementations, the one or more displays 1112 correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot lightemitting diode (QD-LED), micro-electromechanical system (MEMS), and / or the like display types. In some implementations, the one or more displays 1112 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. In one example, the device 1100 includes a single display. In another example, the device 1100 includes a display for each eye of the user.

[0122] In some implementations, the one or more image sensor systems 1114 are configured to obtain image data that corresponds to at least a portion of the physical environment 100. For example, the one or more image sensor systems 1114 include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, and / or the like. In various implementations, the one or more image sensor systems 1114 further include illumination sources that emit light, such as a flash. In various implementations, the one or more image sensor systems 1114 further include an on-camera image signal processor (ISP) configured to execute a plurality of processing operations on the image data.

[0123] In some implementations, sensor data may be obtained by device(s) (e.g., device 105 Figure 1) during a scan of a room of a physical environment. The sensor data may include a 3D point cloud and a sequence of 2D images corresponding to captured views of the room during the scan of the room. In some implementations, the sensor data includes image data (e.g., from an RGB camera), depth data (e.g., a depth image from a depth camera), ambient light sensor data (e.g., from an ambient light sensor), and / or motion data from one or more motion sensors (e.g., accelerometers, gyroscopes, IMU, etc.). In some implementations, the sensor data includes visual inertial odometry (VIO) data determined based on image data. The 3D point cloud may provide semanticAttorney Docket No. 097425-01485(P68856WOl) information about one or more elements of the room. The 3D point cloud may provide information about the positions and appearance of surface portions within the physical environment. In some implementations, the 3D point cloud is obtained over time, e.g., during a scan of the room, and the 3D point cloud may be updated, and updated versions of the 3D point cloud obtained over time. For example, a 3D representation may be obtained (and analyzed / processed) as it is updated / adjusted over time (e.g., as the user scans a room).

[0124] In some implementations, sensor data may be positioning information, some implementations include a VIO to determine equivalent odometry information using sequential camera images (e.g., light intensity image data) and motion data (e.g., acquired from the IMU / motion sensor) to estimate the distance traveled. Alternatively, some implementations of the present disclosure may include a simultaneous localization and mapping (SLAM) system (e.g., position sensors). The SLAM system may include a multidimensional (e.g., 3D) laser scanning and range-measuring system that is GPS independent and that provides real-time simultaneous location and mapping. The SLAM system may generate and manage data for a very accurate point cloud that results from reflections of laser scanning from objects in an environment. Movements of any of the points in the point cloud are accurately tracked over time, so that the SLAM system can maintain precise understanding of its location and orientation as it travels through an environment, using the points in the point cloud as reference points for the location.

[0125] In some implementations, the device 1100 includes an eye tracking system for detecting eye position and eye movements (e.g., eye gaze detection). For example, an eye tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye tracking camera (e.g., near-IR (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the eyes of the user. Moreover, the illumination source of the device 1100 may emit NIR light to illuminate the eyes of the user and the NIR camera may capture images of the eyes of the user. In some implementations, images captured by the eye tracking system may be analyzed to detect position and movements of the eyes of the user, or to detect other information about the eyes such as pupil dilation or pupil diameter. Moreover, the point of gaze estimated fromAttorney Docket No. 097425-01485(P68856WOl) the eye tracking images may enable gaze-based interaction with content shown on the near-eye display of the device 1100.

[0126] The memory 1120 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory 1120 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 1120 optionally includes one or more storage devices remotely located from the one or more processing units 1102. The memory 1120 includes a non- transitory computer readable storage medium.

[0127] In some implementations, the memory 1120 or the non- transitory computer readable storage medium of the memory 1120 stores an optional operating system 1130 and one or more instruction set(s) 1140. The operating system 1130 includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the instruction set(s) 1140 include executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set(s) 1140 are software that is executable by the one or more processing units 1102 to carry out one or more of the techniques described herein.

[0128] The instruction set(s) 1140 include a prior user activity instruction set 1142 and a current user activity instruction set 1144 performing functions to identify, store, aggregate, and use user activity information, including contextual information, as described herein. The instruction set(s) 1140 may be embodied as a single software executable or multiple software executables.

[0129] Although the instruction set(s) 1140 are shown as residing on a single device, it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Moreover, Figure 11 is intended more as functional description of the various features which are present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. The actual number of instructions sets and how features are allocated among them may vary from one implementation toAttorney Docket No. 097425-01485(P68856WOl) another and may depend in part on the particular combination of hardware, software, and / or firmware chosen for a particular implementation.

[0130] Those of ordinary skill in the art will appreciate that well-known systems, methods, components, devices, and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the example implementations described herein. Moreover, other effective aspects and / or variants do not include all of the specific details described herein. Thus, several details are described in order to provide a thorough understanding of the example aspects as shown in the drawings. Moreover, the drawings merely show some example embodiments of the present disclosure and are therefore not to be considered limiting.

[0131] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0132] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.Attorney Docket No. 097425-01485(P68856WOl)

[0133] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0134] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0135] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an executionAttorney Docket No. 097425-01485(P68856WOl) environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a crossplatform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. Unless specifically stated otherwise, it is appreciated that throughout this specification discussions utilizing the terms such as “processing,” “computing,” “calculating,” “determining,” and “identifying” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

[0136] The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides a result conditioned on one or more inputs. Suitable computing devices include multipurpose microprocessor-based computer systems accessing stored software that programs or configures the computing system from a general-purpose computing apparatus to a specialized computing apparatus implementing one or more implementations of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device.

[0137] Implementations of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the examples above can be varied for example, blocks can be re-ordered, combined, and / or broken into sub-blocks. Certain blocks or processes can be performed in parallel. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0138] The use of “adapted to” or “configured to” herein is meant as open and inclusive language that does not foreclose devices adapted to or configured to performAttorney Docket No. 097425-01485(P68856WOl) additional tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or value beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting.

[0139] It will also be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first node could be termed a second node, and, similarly, a second node could be termed a first node, which changing the meaning of the description, so long as all occurrences of the “first node” are renamed consistently and all occurrences of the “second node” are renamed consistently. The first node and the second node are both nodes, but they are not the same node.

[0140] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0141] As used herein, the term “if’ may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordanceAttorney Docket No. 097425-01485(P68856WOl) with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.

Claims

Attorney Docket No. 097425-01485(P68856WOl)What is claimed is:

1. A method comprising: at a device having a processor: storing activity data comprising contextual information for a plurality of activities of a user in one or more physical environments, the contextual information based on sensor data obtained via one or more sensors of a wearable device worn by the user, the contextual information for each respective activity of the activities corresponding to a respective relationship between the user and the one or more physical environments during the respective activity; generating aggregated data based on aggregating activities of the activity data based on similarities in the contextual information; and providing the aggregated data to facilitate a user activity prediction associated with a current user activity based on contextual information about a current user state.

2. The method of claim 1 , wherein the contextual information for each respective activity of the activities corresponds to a relationship between the user and one or more objects in the one or more physical environments separate from the wearable device.

3. The method of claim 1, wherein the contextual information for each respective activity of the activities corresponds to one or more positions in the one or more physical environments.

4. The method of any of claims 1-3, wherein the contextual information for one or more of the plurality of activities comprises a relative location of the user with respect to one or more elements of the physical environment.

5. The method of any of claims 1-4, wherein the contextual information for one or more of the plurality of activities comprises an identification of whether the userAttorney Docket No. 097425-01485(P68856WOl) satisfies a positional classification or threshold relative to one or more elements of the physical environment.

6. The method of any of claims 1-5, wherein the contextual information for one or more of the plurality of activities comprises viewing direction of the user.

7. The method of any of claims 1 -6, wherein the contextual information for one or more of the plurality of activities comprises information obtained from another device in the physical environment, the another device separate from the wearable device.

8. The method of any of claims 1-7, wherein the contextual information for one or more of the plurality of activities comprises information identifying a movement of the user determined based on a motion sensor of the wearable device.

9. The method of any of claims 1-8, wherein the contextual information for one or more of the plurality of activities comprises information based on a scene understanding of a physical environment generated via computer vision using image data obtained via an image sensor of the wearable device.

10. The method of any of claims 1 -9, wherein the contextual information for one or more of the plurality of activities comprises information based on a semantic understanding of a physical environment generated via computer vision using image data obtained via an image sensor of the wearable device.

11. The method of any of claims 1-10, wherein the contextual information for one or more of the plurality of activities comprises a classification of a user state.

12. The method of claim 11, wherein the classification of the user state corresponds a user posture, gesture, whether the user is sitting or standing, whether the user is standing or walking, or whether the user is stationary or mobile.Attorney Docket No. 097425-01485(P68856WOl)13. The method of claim 11, wherein the classification of user state is based on sensor data from a biometric sensor of the wearable device.

14. The method of any of claims 1-13, wherein the contextual information comprises a fusion of information from at least two of the following sources: a source identifying the current time of day; a source identifying a current day or date; a calendar identifying a planned user activity; a messaging app identifying a user activity; and a source based on sensor data associated with the user or the physical environment.

15. The method of any of claims 1-14, wherein the aggregating is based on a taxonomy of user activities in which user activities are classified and related using multiple levels of granularity.

16. The method of claim 15, wherein the aggregating is based on selecting a level of granularity based on the contextual information about the current user state.

17. The method of any of claim 1-16, wherein the prediction comprises a likelihood that the current user activity corresponds to a user activity associated with a cluster of user activities identified via the aggregating.

18. The method of any of claim 1-16, wherein the prediction comprises likelihoods that the current user activity corresponds to each user activity associated with multiple clusters of user activities identified via the aggregating.

19. The method of any of claims 1-17, wherein a response to a user query is enabled based on the prediction associated with the current user activity.Attorney Docket No. 097425-01485(P68856WOl)20. The method of any of claims 1-19, wherein a user interface function is provided based on determining that the user interface function is likely to be used by the user based on the prediction associated with the current user activity.

21. The method of any of claims 1-20, wherein a point of view (PoV) correction is automatically initiated based on the prediction associated with the current user activity.

22. The method of any of claims 1-21, wherein a collision avoidance function is automatically initiated based on the prediction associated with the current user activity.

23. The method of any of claims 1-22, wherein the wearable device is a head-mounted device (HMD).

24. A device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the electronic device to perform operations comprising any one of the methods of claims 1- 23.

25. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors, of a head-mounted-device having a display, to perform operations comprising any one of the methods of claims 1-23.Attorney Docket No. 097425-01485(P68856WOl)26. A method comprising: at a wearable device having a processor and one or more sensors: obtaining current contextual information corresponding to a current user activity of a user with respect to a physical environment, the current contextual information based on sensor data obtained via the one or more sensors of the wearable device; generating a current user activity prediction based on the current contextual information and aggregated prior activity data for a plurality of prior activities of the user in one or more physical environments, the aggregated prior activity data having been aggregated based on prior contextual information, the prior contextual information for each respective prior activity having been generated based on sensor data obtained via one or more sensors of one or more devices worn by the user and corresponding to a respective relationship between the user and the one or more physical environments during the respective prior activity; and enhancing a user experience based on the current user activity prediction.

27. The method of claim 26, wherein the prior contextual information for each respective prior activity of the activities corresponds to a relationship between the user and one or more objects in the one or more physical environments separate from the one or more devices worn by the user during the respective prior activity.

28. The method of claim 26, wherein the prior contextual information for each respective prior activity of the activities corresponds to one or more positions in the one or more physical environments.

29. The method of any of claims 26-28, wherein the contextual information for one or more of the plurality of prior activities comprises a relative location of the user with respect to one or more elements of the physical environment.

30. The method of any of claims 26-29, wherein the contextual information for one or more of the plurality of prior activities comprises an identification of whetherAttorney Docket No. 097425-01485(P68856WOl) the user satisfies a positional classification or threshold with respect to one or more elements of the physical environment.

31. The method of any of claims 26-30, wherein the contextual information for one or more of the plurality of prior activities comprises viewing direction of the user.

32. The method of any of claims 26-31, wherein the contextual information for one or more of the plurality of prior activities comprises information obtained from another device in the physical environment, the another device separate from the one or more wearable devices.

33. The method of any of claims 26-32, wherein the contextual information for one or more of the plurality of prior activities comprises information identifying a movement of the user determined based on a motion sensor of the one or more wearable devices.

34. The method of any of claims 26-33, wherein the contextual information for one or more of the plurality of prior activities comprises information based on a scene understanding of a physical environment generated via computer vision using image data obtained via an image sensor of the one or more wearable devices.

35. The method of any of claims 26-34, wherein the contextual information for one or more of the plurality of prior activities comprises information based on a semantic understanding of a physical environment generated via computer vision using image data obtained via an image sensor of the one or more wearable devices.

36. The method of any of claims 26-35, wherein the contextual information for one or more of the plurality of prior activities comprises a classification of a user state.Attorney Docket No. 097425-01485(P68856WOl)37. The method of claim 36, wherein the classification of the user state corresponds a user posture, gesture, whether the user is sitting or standing, whether the user is standing or walking, or whether the user is stationary or mobile.

38. The method of claim 36, wherein the classification of user state is based on sensor data from a biometric sensor of the one or more wearable devices.

39. The method of any of claims 26-38, wherein the aggregating is based on a taxonomy of user activities in which user activities are classified and related using multiple levels of granularity.

40. The method of claim 39, wherein the aggregating is based on selecting a level of granularity based on the contextual information about the current user state.

41. The method of any of claims 26-40, wherein the prediction comprises a likelihood that the current user activity corresponds to a user activity associated with a cluster of user activities identified via the aggregating.

42. The method of any of claims 26-41, wherein the prediction comprises likelihoods that the current user activity corresponds to each user activity associated with multiple clusters of user activities identified via the aggregating.

43. The method of any of claims 26-42, wherein a response to a user query is enabled based on the prediction associated with the current user activity.

44. The method of any of claims 26-43, wherein a user interface function is provided based on determining that the user interface function is likely to be used by the user based on the prediction associated with the current user activity.Attorney Docket No. 097425-01485(P68856WOl)45. The method of any of claims 26-44, wherein a point of view (PoV) correction is automatically initiated based on the prediction associated with the current user activity.

46. The method of any of claims 26-45, wherein a collision avoidance function is automatically initiated based on the prediction associated with the current user activity.

47. The method of any of claims 26-46, wherein the wearable device is a head-mounted device (HMD).

48. A device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the electronic device to perform operations comprising any one of the methods of claims 26-47.

49. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors, of a head-mounted-device having a display, to perform operations comprising any one of the methods of claims 26-47.

Images