Create the best working, learning, and resting environment on the electronic device

Abstract

Some specific implementations disclosed herein present a computer-generated reality (CGR) environment in which a user participates in an activity, recognize the cognitive state of the user (e.g., working, studying, resting, etc.) based on data about the user's body (e.g., facial expressions, hand movements, physiological data, etc.), and update the environment with a surrounding environment that enhances the user's cognitive state of the activity.

Description

Technical Field

[0001] The present disclosure generally relates to displaying content on an electronic device and, more particularly, to systems, methods, and devices for enhancing a user's cognitive state for a particular user activity. Background Art

[0002] While viewing content on an electronic device, a user's cognitive state has a significant impact on the user's ability to work or study. For example, when using a head-mounted device (HMD) in a computer-generated reality (CGR) environment, it may be necessary to maintain focus and engagement to have a meaningful experience, such as learning new skills, watching instructional or entertainment content, or reading documents. Improved techniques for assessing and enhancing a user's cognitive state while viewing and interacting with content can enhance the user's productivity and ability to interact with content. Thus, based on being able to determine a user's cognitive state, content creators and display systems may be able to provide a better user experience that the user is more likely to enjoy, understand, and learn from. Summary of the Invention

[0003] Attributes of content can cause specific types of physiological responses in the body of a user viewing the content. For example, users may have different preferences for optimal work, study, and rest states. Some users may prefer to work in a quiet environment with low light, while other users may prefer to work with music and bright lights. Clutter in a room or on a desk may enhance the creativity of some people, but may be distracting to others. Similarly, the optimal rest environment is personalized and context-dependent. In some cases, a user may not even be aware that they need a break, or they may not discover the most restorative environment for themselves. Creating the best personalized environment for a user to work, study, and rest in a computer-generated reality (CGR) environment and presenting it to them at the right time can benefit the user by increasing their productivity and learning efficiency while also providing them with the necessary break opportunities when such an environment is not available in their surrounding environment.

[0004] The specific implementations herein recognize that a user's physiological responses over time or in response to changes in the CGR environment (e.g., heart rate, respiratory rate, brain activity, user satisfaction, facial expressions, body language, etc.) can indicate the user's cognitive state or preferences in the CGR environment. For example, a user's pupils can dilate or constrict at different rates in different cognitive states. By presenting the user with activities that involve multiple surrounding environments (e.g., living room, library, office, forest, coffee shop, ocean, waterfall, babbling brook, etc.), obtaining data about the user's body, and evaluating the data obtained for each surrounding environment, the specific implementations can identify the specific surrounding environments that promote a positive or otherwise desired cognitive state for the user with respect to that activity (e.g., identifying the best environment that places the user in the most relaxed or restorative state).

[0005] In addition, the specific implementations can identify the user's cognitive state and recommend changes to the user's activities. For example, the specific implementations can detect fatigue when the user is working, or can detect that the user has relaxed when the user's activity is relaxing or resting. Based on this detection, the specific implementations can then display the user's favorite calming forest when the user is fatigued, or display a motivating work / study environment when the user is resting.

[0006] In some specific implementations, the parameters of the surrounding environment, such as color, light level, decoration, sound, music, volume, etc., are changed to identify the specific parameters that enhance the user's cognitive state for the activity. In some specific implementations, the user's physiological data is obtained via a heart rate sensor, pulse oximeter, blood pressure sensor, temperature sensor, electrocardiogram (EKG) sensor, electroencephalogram (EEG) sensor, electromyogram (EMG) sensor, functional near-infrared spectroscopy signal (fNIRS) sensor, galvanic skin response (GSR) sensor, etc. In addition, an inward-facing camera can record eye features (e.g., eye tracking or pupil response) or a downward-facing camera can record facial expressions. Image-based or sensor-based hand / body tracking can be used to identify movements associated with stress, relaxation, etc.

[0007] In some embodiments, the context of content or activity is used to identify the user's current cognitive state or the user's desired cognitive state. For example, if the user's activity is work-related, the embodiment can identify that the user desires a productive, calm, or relaxed cognitive state. In some embodiments, information about the user's preferences and past activities, as well as the attributes of the content, can be used to identify or enhance the user's cognitive state. In some embodiments, a message is displayed to the user to make the user aware of his or her cognitive state, suggest taking a break, or provide options to change the environment, and in some embodiments, actions can occur automatically (e.g., automatically changing the environment based on the user's cognitive state). In some embodiments, content creators collect or receive data based on privacy settings in order to optimize the user's environment. For example, the content creator can save associated data, modify the content, or update the environments of other similar users performing similar activities.

[0008] In accordance with some embodiments, instructions are stored in a non-transitory computer-readable storage medium, the instructions being computer-executable to perform or cause to be performed any of the methods described herein. According to some embodiments, an apparatus includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to be performed any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Thus, the present disclosure can be understood by those of ordinary skill in the art, and a more detailed description can be referred to in aspects of some exemplary embodiments, some of which are shown in the drawings.

[0010] Figure 1 An apparatus for displaying content and obtaining physiological data from a user is shown in accordance with some embodiments.

[0011] Figure 2 Shown is Figure 1 the pupil of a user, where the diameter of the pupil changes.

[0012] Figure 3 is a flowchart showing the selection of an environment by evaluating different environments and changing parameters of the selected environment in accordance with some embodiments.

[0013] Figure 4A 、 4B and 4C show different environments in accordance with some embodiments.

[0014] Figure 5 is a flowchart showing the facilitation of user activities based on detecting the user's cognitive state in accordance with some embodiments.

[0015] Figure 6A , 6B Figures 6C illustrate productivity applications and different environments according to some specific implementations.

[0016] Figure 7 is a block diagram of device components showing an exemplary device according to some specific implementations.

[0017] Figure 8 is a block diagram of an exemplary head-mounted device (HMD) according to some specific implementations.

[0018] Figure 9 is a flowchart of an exemplary method showing an updated environment according to some specific implementations.

[0019] Figure 10 is a flowchart of an exemplary method of identifying a surrounding environment that enhances a user's cognitive state of an activity.

[0020] In accordance with common practice, the various feature portions shown in the drawings may not be drawn to scale. Accordingly, for clarity, the dimensions of the various feature portions may be arbitrarily expanded or reduced. Additionally, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, like reference numerals may be used to represent like feature portions. Detailed Description

[0021] Numerous details are described in order to provide a thorough understanding of the example specific implementations shown in the drawings. However, the drawings only show some example aspects of the present disclosure and should not be considered limiting. Those of ordinary skill in the art will know that other effective aspects or variations do not include all of the specific details described herein. Additionally, well-known systems, methods, components, devices, and circuits have not been described in detail so as not to obscure more relevant aspects of the exemplary specific implementations described herein.

[0022] Figure 1 shows a physical environment 5 that includes a device 10 (e.g., a handheld device) having a display 15. The device 10 may include an integrated controller or may communicate with a separate controller, where one or both controllers may be located within the physical environment 5. A physical environment refers to the physical world in which people can sense and / or interact without the assistance of an electronic system. Physical environments such as a physical park include physical items such as physical trees, physical buildings, physical locations, and physical people (e.g., user 25). People can directly sense and / or interact with the physical environment 5, such as through vision, touch, hearing, taste, and smell.

[0023] In some specific implementations, the device 10 is configured to manage, coordinate, and present a computer-generated reality (CGR) environment to the user 25. In some specific implementations, the device 10 includes a suitable combination of software, firmware, or hardware. The device 10 is described in more detail below with reference to FIGS. 6 and Figure 7 In some specific implementations, the controller of the device 10 is a computing device that is local or remote relative to the physical environment 5. In some specific implementations, the functions of the controller of the device 10 are provided by or in combination with the device 10, for example, in the case of a handheld device or a head-mounted device (HMD) used as a stand-alone unit.

[0024] In one example, the controller of the device 10 is a local server located within the physical environment 5. In another example, the controller of the device 10 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 5. In some specific implementations, the controller of the device 10 is communicatively coupled to the device 10 via one or more wired or wireless communication channels (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0025] According to some specific implementations, when the user 25 is present within the physical environment 5, the device 10 presents a CGR environment to the user 25. A CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements or their representations are tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that conforms to at least one physical law. For example, a CGR system can detect a person's head rotation, and in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in the physical environment. In some cases (e.g., for accessibility reasons), the adjustment of the characteristics of virtual objects in the CGR environment can be made in response to a representation of a physical movement (e.g., a voice command).

[0026] A person can use any of their senses to sense and / or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of point audio sources in a 3D space. Also, for example, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In certain CGR environments, a person can sense and / or only interact with audio objects.

[0027] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment is an environment designed to be a simulated environment that is completely computer-generated sensory input for one or more senses. A VR environment includes virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment by simulation of the person's presence within the computer-generated environment and / or by simulation of a subgroup of the person's physical movements within the computer-generated environment.

[0028] Compared with a VR environment that is designed to be completely computer-generated sensory input, a mixed reality (MR) environment is an environment designed to be a simulated environment that includes, in addition to computer-generated sensory input (e.g., virtual objects), sensory input from the physical environment or a representation thereof. On the virtual continuum, a mixed reality environment is any condition between a completely physical environment as one end and a virtual reality environment as the other end, but does not include these two ends.

[0029] In some MR environments, the computer-generated sensory input can respond to changes in the sensory input from the physical environment. Additionally, some electronic systems for presenting an MR environment can track the position and / or orientation relative to the physical environment so that virtual objects can interact with real objects (i.e., physical items from the physical environment or a representation thereof). For example, the system can cause movement such that a virtual tree appears stationary relative to the physical ground.

[0030] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or translucent display such that the person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system can have an opaque display and one or more imaging sensors that capture images or videos of the physical environment 5, which are representations of the physical environment 5. The system combines the images or videos with the virtual objects and presents the composition on the opaque display. The person uses the system to indirectly view the physical environment 5 via the images or videos of the physical environment 5 and to perceive the virtual objects superimposed on the physical environment 5. As used herein, a video of the physical environment 5 displayed on an opaque display is referred to as a "passthrough video," meaning that the system uses one or more image sensors to capture images of the physical environment 5 and uses those images when presenting the AR environment on the opaque display. Further alternatively, the system can have a projection system that projects virtual objects into the physical environment 5, such as as a hologram or on a physical surface, such that the person uses the system to perceive the virtual objects superimposed on the physical environment 5.

[0031] An augmented reality environment is also a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing a passthrough video, the system can transform one or more sensor images to impose an alternative perspective (e.g., viewpoint) different from the perspective captured by the imaging sensor. As another example, a representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions thereof such that the modified portions can be a representative but not a true version of the originally captured image. As yet another example, a representation of the physical environment can be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0032] An augmented virtuality (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but a person's face is a realistic reproduction from an image of a physical person. As another example, a virtual object can adopt the shape or color of a physical item imaged by one or more imaging sensors. As yet another example, a virtual object can adopt a shadow that conforms to the positioning of the sun in the physical environment.

[0033] There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed to be placed on a person's eye (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, the head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system can have a transparent or translucent display instead of an opaque display. The transparent or translucent display can have a medium through which light representing an image is directed to a person's eye. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one specific implementation, the transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphical image onto a person's retina. The projection system can also be configured to project virtual objects into the physical environment, such as as a hologram or on a physical surface.

[0034] As Figure 1 shown, in some specific implementations, device 10 displays content 20 to user 25. The content 20 can include video, presentations, other time-varying content, or content presented as part of a CGR environment. In some specific implementations, device 10 is configured to obtain image data or physiological data (e.g., pupil data, electrocardiogram (EKG) data, etc.) about user 25's body via one or more sensors. In some specific implementations, the input terminals used by device 10 to collect data about user 25's body include cameras (e.g., for detecting body language, performing hand tracking, recognizing facial expressions, etc.), microphones (e.g., for recognizing intonation), and physiological sensors (e.g., for measuring pupil size, gaze, electroencephalogram (EEG), EKG, electromyogram (EMG), functional near-infrared spectroscopy signals (fNIRS), galvanic skin response (GSR), pulse, respiratory rate, etc.). Additionally, in some specific implementations, the device can also determine the context of the content or content 20 (e.g., the user's location, language, intonation, pictures and videos, etc.).

[0035] In some specific implementations, device 10 can associate the captured image data or physiological data with the cognitive state of user 25. For example, device 10 can analyze various factors in real time to determine the cognitive state of the user. In some specific implementations, user body language is modeled using user location data in combination with hand tracking technology and inferred or measured body postures for user 25. In some specific implementations, device 10 utilizes a computational model for cognitive state assessment, including detecting body language associated with the cognitive state and any corresponding physiological markers of the cognitive state. For example, combining body language detection (such as yawning) with a reduced heart rate can provide an enhanced indicator that the user is tired or bored and not engaged in work activities.

[0036] The user will have the option to opt in or opt out of any feature regarding whether to obtain or use his or her user data or otherwise turn on and off the obtaining or using of user information. Additionally, each user will have the ability to access and otherwise find out anything the system has collected or determined about him or her. User data is securely stored on the user's device. For example, user data associated with the user's body and / or cognitive state can be stored in a secure enclave on the user device, thus restricting access to the user data and limiting the transmission of user data to other devices.

[0037] Although device 10 is shown as a handheld device, other specific implementations relate to devices with which the user interacts without holding and devices worn by the user. In some specific implementations, as Figure 1 shown, device 10 is a handheld electronic device (e.g., a smartphone or a tablet). In some specific implementations, device 10 is a laptop computer or a desktop computer. In some specific implementations, device 10 has a touchpad, and in some specific implementations, device 10 has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some specific implementations, device 10 is or communicates with a wearable device, such as a head-mounted display (HMD), a watch, or an armband.

[0038] Furthermore, although this example and other examples discussed herein show a single device 10 in a physical environment 5, the techniques disclosed herein apply to multiple devices as well as multiple real-world environments. For example, the functions of device 10 can be performed by multiple devices.

[0039] In some specific embodiments, device 10 includes an eye tracking system for detecting eye position and eye movement. For example, the eye tracking system may include one or more infrared (IR) light emitting diodes (LEDs), an eye tracking camera (e.g., a near-infrared (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) toward user 25's eyes. Additionally, the illumination source of device 10 may emit NIR light to illuminate user 25's eyes, and the NIR camera may capture images of user 25's eyes. In some specific embodiments, the images captured by the eye tracking system may be analyzed to detect the position and movement of user 25's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. Additionally, the gaze point estimated from the eye tracking images may enable gaze-based interaction with the content.

[0040] In some specific embodiments, device 10 has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, programs, or instruction sets stored in the memory for performing multiple functions. In some specific embodiments, user 25 interacts with the GUI through finger contacts and gestures on a touch-sensitive surface. In some specific embodiments, these functions include image editing, drawing, rendering, word processing, web page creation, disk editing, spreadsheet creation, playing games, making phone calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, and / or digital video playback. The executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0041] In some specific embodiments, device 10 employs various physiological sensors, detection, or measurement systems. The detected physiological data may include, but is not limited to, EEG, EKG, EMG, fNIRS, blood pressure, GSR, or pupil response. Additionally, device 10 may simultaneously detect multiple forms of physiological data in order to benefit from the synchronous acquisition of physiological data. Further, in some specific embodiments, the physiological data represents involuntary data, i.e., responses that are not under conscious control. For example, pupil response may represent involuntary movement.

[0042] In some specific embodiments, one or both of user 25's eyes 45 (including one or both of user 25's pupils 50) present physiological data in the form of a pupil response. User 25's pupil response causes a change in the size or diameter of pupil 50 via the optic nerve and the oculomotor cranial nerve. For example, the pupil response may include a constriction response (miosis), i.e., the pupil narrowing, or a dilation response (mydriasis), i.e., the pupil widening. In some specific embodiments, device 10 may detect a pattern of physiological data representing the time-varying pupil diameter.

[0043] Figure 2 shows Figure 1 the pupil 50 of user 25, where the diameter of the pupil 50 changes over time. As Figure 2 shown, the current physiological state (e.g., the current pupil diameter 55) may change compared to the past physiological state (e.g., the past pupil diameter 60). For example, the current physiological state may include the current pupil diameter and the past physiological state may include the past pupil diameter. This physiological data may represent a response pattern that changes dynamically over time.

[0044] Device 10 may use this physiological data to implement the techniques disclosed herein. For example, the response of the user's pupil to an environmental change in content 20 may be compared to the user's previous response to a similar environmental change event in the same or other content.

[0045] According to some embodiments, Figure 3 is a flowchart showing a method 300 for selecting an environment and changing parameters of the selected environment by evaluating different environments. In some embodiments, method 300 is performed by one or more devices (e.g., device 10). Method 300 may be performed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. Method 300 may be performed on an HMD having a screen for displaying 3D images or a screen for viewing stereoscopic images. In some embodiments, method 300 is performed by processing logic (including hardware, firmware, software, or combinations thereof). In some embodiments, method 300 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0046] At block 310, method 300 exposes the user to different environments by displaying on a display of the device content that includes each of the different environments (e.g., a rest environment). For example, as Figure 4A - 4C shown, the user may be exposed to a library environment 312, a living room control environment 314, and a forest environment 316.

[0047] At block 320, method 300 evaluates each of the environments based on the user's response to the environment. In some embodiments, physiological (e.g., eye tracking, pupil diameter, heart rate, respiratory rate, GSR, etc.), neurological (e.g., EEG, fNIRS, etc.), and behavioral signals (facial expressions, user ratings and performance, etc.) are combined by method 300 to evaluate each environment. For example, in a rest mode, heart rate and respiratory rate may slow down, heart rate variability (HRV) may increase, and pupil diameter and brain activity as measured by EEG or fNIRS are similar to default activity. Additionally, in learning and work modes, heart rate and respiratory rate can increase to normal levels, HRV can decrease, pupil diameter and brain activity can show maximum task engagement, and the user can demonstrate an enhanced level of focus and concentration.

[0048] At block 330, method 300 can change the parameters of the surrounding environment, e.g., from general characteristics to specific characteristics. In some embodiments, method 300 discovers personalized optimal work, learning, and rest states by iteratively updating the VR audio-visual environment. For example, some of the parameters to be adjusted can include location (living room, nature, coffee shop, etc.), light color, intensity, proximity and direction, audio loudness, music genre, noise type (white, pink, etc.), and type of natural sounds (forest, city, ocean, waterfall, babbling brook, etc.). Additionally, each parameter can be adjusted independently and / or jointly over time until the user's body data indicates that the environment is optimized.

[0049] In some embodiments, the environment includes the background of a scene (e.g., the background of a forest), the surrounding environment associated with the scene (e.g., the furniture in a library), or any number of other audio or visual features associated with the scene (e.g., the sound of turning pages, rain, white noise, etc.). As an example, if the user is working, method 300 can teleport the user to an environment such as a forest, beach, library, or coffee shop that is not easily accessible in the user's surrounding environment. In some embodiments, method 300 automatically adjusts the environment parameters to increase productivity without interrupting the experience. Additionally, method 300 can identify relaxation environments based on an individual's physiology, which may be different from the environments that the user previously thought were relaxing, thus enabling longer productivity sessions by preventing burnout and mental fatigue.

[0050] In some embodiments, alternative parameters (e.g., calm, high energy, etc.) are selected based on testing the actual effects (e.g., on the user's cognitive state) of using the surrounding environment with each or a combination of the parameters. For example, as Figure 3As shown in frame 330, the rest environment may include a forest 332, and then parameters such as "calm" 334 or "high energy" may be associated with the environment and presented to the user. Based on the impact on the user's cognitive state, method 300 may determine that the user is in a more relaxed or peaceful cognitive state when the forest 332 environment is associated with the calm 334 parameter, and method 300 may explore different audio 338 and visual 340 parameters to ultimately find the optimal rest environment 342.

[0051] According to some specific implementations, Figure 5 is a flowchart showing a method 500 for facilitating user activities based on detecting the user's cognitive state. For example, although an HMD is used in a CGR environment, the user may not know when he or she should be working or studying, and when they should relax or rest. In some specific implementations, method 500 is executed by one or more devices (e.g., device 10). Method 500 may be executed on a mobile device, an HMD, a desktop computer, a laptop computer, or a server device. Method 500 may be executed on an HMD having a screen for displaying 3D images or a screen for viewing stereoscopic images. In some specific implementations, method 500 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 500 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0052] At block 520, method 500 displays content to the user on the device's display, and the user executes a productivity application. For example, the user may execute a productivity application (e.g., Figure 6A - 6C the work application 610 shown in) for twenty minutes. In some specific implementations, the content displayed to the user includes a suitable surrounding environment for the user's activity (e.g., the work application 610 engaged in various environments including a library 312, a living room 314, a forest 316, etc.). For example, if the user is working or studying, the surrounding environment may include an optimized library scene, e.g., a library scene including a customized scene to optimize the user's productivity.

[0053] At block 540, method 500 detects a change in the user's cognitive state and recommends that the user take a break from the productivity application. In some specific implementations, method 500 combines physiological (e.g., eye tracking, pupil diameter, heart rate, respiratory rate, GSR, etc.), neural (e.g., EEG, fNIRS, etc.), and behavioral signals (e.g., facial expressions, user ratings, and performance, etc.) to identify the user's cognitive state. For example, the method may detect signs of mental fatigue and recommend that the user take a break.

[0054] In some specific implementations, examples of cognitive states include, but are not limited to: attentive, inattentive, sharp, focused, unfocused, distracted, interested, uninterested, curious, overly curious, skeptical, critical, certain, decisive, evaluating, evaluating logically, evaluating emotionally, accepting, non - accepting, daydreaming, thinking, observing, relaxed, processing information, related to past experience, recalling, being mindful, mind wandering, tired, resting, alert, attracted, repelled, guilty, and indignant. Evaluating a user's cognitive state may involve determining the level or degree of the type of cognitive state, for example, determining the level of attentiveness or inattentiveness on a numerical scale. As a specific example, depending on the pupil response amplitude, method 500 may determine that the user is attentive with a score of 0.6 on a normalized scale from 0 to 1 (0 being inattentive and 1 being fully focused).

[0055] In some specific implementations, statistical or machine - learning - based classification techniques are used to determine the cognitive state. For example, statistical or machine - learning - based techniques can be used to aggregate and classify pupil responses into different cognitive states. In one example, pupil responses are classified as attentive, inattentive, or neutral. In another example, pupil responses are classified as observing or processing information.

[0056] In some specific implementations, the user's physiological data is compared with the user's own previous responses to determine the user's current cognitive state. In some specific implementations, the user's physiological data is evaluated based on the physiological responses to various types of content, for example, comparing the user's current pupil response with the typical or average user response. In some specific implementations, observing the trend of physiological data over a duration (e.g., from seconds to minutes) can provide insights into the user's potential cognitive state at different time scales.

[0057] The user will have the option to opt-in or opt-out of any feature regarding whether to obtain or use his or her user data or otherwise turn on and off the obtaining or using of user information. Additionally, each user will have the ability to access and otherwise find out anything the system has collected or determined about him or her. User data is securely stored on the user's device. The user data used as input to the machine learning model is securely stored on the user's device, for example, to ensure user privacy. The user's device may have a secure storage area, such as a secure enclave, for protecting certain user information, such as data from image sensors and other sensors for face recognition, facial recognition, or biometrics. User data associated with the user's physical and / or cognitive state can be stored in such a secure enclave, thereby restricting access to the user data and restricting the transmission of the user data to other devices to ensure that the user data remains securely on the user's device. The user data may be prohibited from leaving the user device and may only be used in the machine learning model and other processes on the user device.

[0058] In some specific implementations, method 500 provides feedback to the user based on the user's cognitive state. For example, method 500 may detect inattentiveness and provide the user with a notification to take a break, re-watch a specific part of the content, or change the content presentation speed. In another example, method 500 may detect moments of high attentiveness in the content and suggest similar moments of the content to the user. In another example, a notification may be provided to the user to watch the content at a specific time or under specific conditions based on determining a cognitive state profile of the user. For example, such a profile may indicate that the user is more attentive in the morning than in the afternoon or evening, and a notification indicating that now is a good time to study or engage with the content may be provided to the user at 8:30 am.

[0059] At block 560, method 500 transports the user to a different environment based on a change in the user's activity. For example, if the user has changed his or her activity from work to rest, the user may be transported to his or her favorite peaceful forest environment, for example, the surrounding environment may change to a peaceful forest environment. In some specific implementations, the content is automatically adjusted based on the user's cognitive state according to automatically determined environmental parameters.

[0060] Thus, method 500 provides a flexible and ingenious way for the user to take a break without requiring a behavioral response and interruption (e.g., the user may not need to "think about" when or whether to take a break). Method 500 can be provided to users who wish to learn new skills, watch lectures, or perform tasks that require long-term attention and many other use cases, and can provide such users with information to facilitate the user's purpose. Additionally, providing timely interruptions or break periods to the user can allow for a more productive work and learning schedule.

[0061] At block 580, method 500 detects a change in the cognitive state (e.g., a relaxation state) and recommends that the user return to the productivity application. For example, method 500 can use biometric signals or algorithms to detect that the user is in a relaxation state. For example, the user may enter a relaxation state after a three-minute break, and method 500 can recommend that the user return to the learning or productivity application.

[0062] In some embodiments, method 500 provides feedback to content creators to facilitate improvement of the content or future / related content. The notification can confirm a portion of the content associated with the cognitive state of one or more users, e.g., confirm that users generally have less attention during a particular portion of the content than during another portion of the content. In some embodiments, the cognitive state data of multiple users who have viewed the content is aggregated to provide feedback on the content. The content creator can modify the content based on such feedback to make that portion shorter or more compelling.

[0063] Figure 7 FIG. is a block diagram of an example of device 10 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, device 10 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more displays 712, one or more internal and / or external-facing image sensor systems 714, memory 720, and one or more communication buses 704 for interconnecting these components and various other components.

[0064] In some specific implementations, the one or more communication buses 704 include circuitry that interconnects components of the system and controls communication between components. In some specific implementations, the one or more I / O devices and sensors 406 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a tactile engine, or one or more depth sensors (e.g., structured light, time of flight, etc.) and / or similar devices.

[0065] In some specific implementations, the one or more displays 712 are configured to present a user experience to the user 25. In some specific implementations, the one or more displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), retinal projection systems, and / or similar display types. In some specific implementations, the one or more displays 712 correspond to diffractive, reflective, polarization, holographic, etc. waveguide displays. For example, the device 10 includes a single display. As another example, the device 10 includes a display for each eye of the user 25, such as an HMD. In some specific implementations, the one or more displays 712 are capable of presenting CGR content.

[0066] In some specific implementations, the one or more image sensor systems 714 are configured to acquire image data corresponding to at least a portion of the face of the user 25 that includes the eyes of the user 25. For example, the one or more image sensor systems 714 include one or more RGB cameras (e.g., having a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, event-based cameras, etc. In various specific implementations, the one or more image sensor systems 714 further include an illumination source that emits light onto the portion of the face of the user 25, such as a flash or a flash source.

[0067] The memory 720 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific embodiments, the memory 720 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 720 optionally includes one or more storage devices that are remotely located from one or more processing units 702. The memory 720 includes non-transitory computer-readable storage media. In some specific embodiments, the memory 720 or the non-transitory computer-readable storage media of the memory 720 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 730 and a user experience module 740.

[0068] The operating system 730 includes procedures for handling various basic system services and for performing hardware-related tasks. In some specific embodiments, the user experience module 740 is configured to display content on an electronic device and to evaluate the cognitive state of a user viewing such content. To this end, in various specific embodiments, the user experience module 740 includes a content unit 742, a physiological tracking unit 744, an environmental assessment unit 746, and a cognitive state unit 748.

[0069] In some specific embodiments, the content unit 742 is configured to provide and / or track content for display on the device. The content unit 742 may be configured to monitor user activity, track the context of content over time, and / or identify user activity occurring within the content. In some specific embodiments, the content unit 742 may be configured to customize content for a particular user using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various specific embodiments, the unit includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0070] In some specific embodiments, the physiological tracking (e.g., body language, hand tracking, facial expressions, physiological data, etc.) unit 744 is configured to track a user's body movements or other physiological attributes using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various specific embodiments, the unit includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0071] In some specific embodiments, the environmental assessment unit 746 is configured to evaluate and debug the surrounding environment related to a user's activity using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various specific embodiments, the unit includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0072] In some embodiments, the cognitive state unit 748 is configured to evaluate a user's cognitive state based on physical (e.g., bodily) responses using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various embodiments, the unit includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0073] Although Figure 7 the units and modules of are shown as residing on a single device (e.g., device 10), it should be understood that in other embodiments, any combination of these units may be located in separate computing devices.

[0074] In addition, Figure 7 more serves as a functional description of the various features present in a particular embodiment, as opposed to a schematic diagram of the structure of the embodiments described herein. As will be appreciated by one of ordinary skill in the art, items shown separately may be combined and some items may be separated. For example, Figure 7 some of the functional modules shown separately in may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the specific partitioning of functions and how features are allocated therein will vary depending on the embodiment and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0075] Figure 8 FIG. shows a block diagram of an exemplary head-mounted device 800 according to some embodiments. The head-mounted device 800 includes a housing 801 (or enclosure) that houses the various components of the head-mounted device 800. The housing 801 includes (or is coupled to) an eye pad (not shown) disposed at the proximal (to user 25) end of the housing 801. In various embodiments, the eye pad is a plastic or rubber piece that comfortably and snugly holds the head-mounted device 800 in place on the face of user 25 (e.g., around the eyes of user 25).

[0076] The housing 801 houses a display 810 that displays images, emits light toward the eyes of user 25, or emits light into the eyes of the user. In various embodiments, the display 810 emits light through an eyepiece having one or more lenses 805 that refract the light emitted by the display 810 such that the display appears to user 25 to be at a virtual distance farther than the actual distance from the eyes to the display 810. For user 25 to be able to focus on the display 810, in various embodiments, the virtual distance is at least greater than the minimum focal length of the eyes (e.g., 7 cm). Additionally, to provide a better user experience, in various embodiments, the virtual distance is greater than 1 meter.

[0077] The housing 801 also houses a tracking system that includes one or more light sources 822, a camera 824, and a controller 880. The one or more light sources 822 emit light onto the eyes of the user 25, which is reflected as a light pattern (e.g., a flash ring) detectable by the camera 824. Based on the light pattern, the controller 880 can determine the eye movement tracking characteristics of the user 25. For example, the controller 880 can determine the gaze direction and / or blink state (open or closed eyes) of the user 25. As another example, the controller 880 can determine the pupil center, pupil size, or point of regard. Thus, in various embodiments, light is emitted by the one or more light sources 822, reflected from the eyes of the user 25, and detected by the camera 824. In various embodiments, the light from the eyes of the user 25 is reflected from a hot mirror or passes through an eyepiece before reaching the camera 824.

[0078] The display 810 emits light in a first wavelength range, and the one or more light sources 822 emit light in a second wavelength range. Similarly, the camera 824 detects light in the second wavelength range. In various embodiments, the first wavelength range is the visible wavelength range (e.g., a wavelength range of approximately 400 nm - 700 nm within the visible spectrum), and the second wavelength range is the near-infrared wavelength range (e.g., a wavelength range of approximately 700 nm - 1400 nm within the near-infrared spectrum).

[0079] Figure 9 is a flowchart showing an exemplary method of identifying and enhancing the user's awareness of the surrounding environment for a specific user activity according to some embodiments. In some embodiments, the method 900 is performed by a device (e.g., Figure 1 , Figure 7 and Figure 8 device 10). The method 900 can be performed on a mobile device, an HMD, a desktop computer, a laptop computer, a server device, or by multiple devices communicating with each other. In some embodiments, the method 900 is performed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 900 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0080] At block 910, the method 900 presents a CGR environment in which the user is engaged in an activity (e.g., working, studying, resting). In some embodiments, the CGR environment presents multiple surrounding environments (e.g., audio or visual) at different times during the user's engagement in the activity. For example, the method 900 can present a reference environment of a living room to the user, as well as one or more other surrounding environments, such as a forest or a coffee shop.

[0081] At block 920, when presenting multiple surrounding environments in a CGR environment, method 900 obtains data about the user's body via sensors. For example, physiological data can be obtained via a heart rate sensor, a pulse oximeter, a blood pressure sensor, a temperature sensor, an EKG sensor, an EEG sensor, an EMG sensor, a fNIRS sensor, a GSR sensor, etc. Additionally, an inward-facing camera can record eye features (e.g., eye tracking or pupil response) or a downward-facing camera can record facial expressions. In some specific implementations, image-based or sensor-based hand / body tracking can be used to identify movements associated with cognitive states such as stress, relaxation, etc.

[0082] At block 930, method 900 evaluates the multiple surrounding environments based on the data obtained about the user's body. For example, method 900 can evaluate the extent to which each of the surrounding environments enhances a cognitive state suitable for the user's activity. In some specific implementations, each surrounding environment is evaluated based on comparing the data obtained about the user's body in response to one or more other surrounding environments. In some specific implementations, the multiple surrounding environments can be evaluated based on an apparent subjective response from the user (e.g., voice input, gesture, touch input, keyboard input, etc.). For example, the user can specify a preference or aversion to one or more surrounding environments, and the user response can be combined with the data obtained about the user's body to evaluate one or more surrounding environments.

[0083] At block 940, based on the evaluation, method 900 identifies the surrounding environment(s) among the multiple surrounding environments that enhance the user's cognitive state for the activity. For example, method 900 can select the optimal environment that puts the user in the best resting state. In some specific implementations, method 900 changes one or more parameters of the surrounding environment (e.g., color, light level, decoration, sound or music, volume, etc.), and identifies the parameters that enhance the user's cognitive state for the activity.

[0084] Figure 10 is a flowchart showing an exemplary method for changing a CGR environment based on a detected cognitive state according to some specific implementations. In some specific implementations, method 1000 is executed by a device (e.g., Figure 1 , Figure 7 and Figure 8 device 10). Method 1000 can be executed at a mobile device, an HMD, a desktop computer, a laptop computer, a server device, or by multiple devices communicating with each other. In some specific implementations, method 1000 is executed by processing logic (including hardware, firmware, software, or a combination thereof). In some specific implementations, method 900 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0085] At block 1010, method 1000 presents a CGR environment configured based on the user's activity. For example, when the activity is work, the CGR environment can be a coffee shop; when the activity is study, the CGR environment can be a library; when the activity is relaxation, the CGR environment can be a forest, and so on.

[0086] At block 1020, when presenting multiple surrounding environments in the CGR environment, method 1000 obtains data about the user's body through sensors. For example, physiological data can be obtained through a heart rate sensor, a pulse oximeter, a blood pressure sensor, a temperature sensor, an EKG sensor, an EEG sensor, an EMG sensor, a fNIRS sensor, a GSR sensor, etc. In addition, an inward-facing camera can record eye features (e.g., eye tracking or pupil response), or a downward-facing camera can record facial expressions. In some specific implementations, image-based or sensor-based hand / body tracking can be used to identify movements associated with cognitive states such as stress, relaxation, etc.

[0087] At block 1030, method 1000 determines the user's cognitive state based on the data obtained about the user's body. For example, method 1000 can determine that the user is fatigued, energetic, etc. In some specific implementations, the cognitive state is classified by identifying the user's body movements or features and associating the identified body movements or features with cognitive states. In some specific implementations, multiple forms of body data are utilized to improve the accuracy of cognitive state classification. For example, machine learning algorithms can classify the cognitive state based on the user's eye features, facial expressions, body movements, and other physiological data (e.g., heart rate, temperature, electrocardiogram, etc.).

[0088] In some specific implementations, statistical or machine learning-based classification techniques are used to determine the cognitive state. In some specific implementations, statistical or machine learning-based techniques are used to aggregate and classify body data into different cognitive states. For example, a machine learning model can be trained to classify facial expressions into one or a fixed number of cognitive state categories.

[0089] At block 1040, based on the cognitive state, method 1000 presents a CGR environment configured based on an activity different from the first activity. For example, when a fatigued cognitive state is detected, method 1000 may switch from an environment associated with work (e.g., a coffee shop) to an environment associated with rest (e.g., a forest). In another example, when an energetic / relaxed / revitalized cognitive state is detected, method 1000 may switch from an environment associated with rest (e.g., a forest) to an environment associated with work (e.g., a coffee shop). In some embodiments, method 1000 may change the CGR environment automatically or based on providing a suggestion and receiving input from the user (e.g., voice input, gesture, touch input, keyboard input, etc.). For example, when method 1000 detects that the user's cognitive state has changed again, the method may change the environment back.

[0090] In some embodiments, method 1000 continues to obtain data about the user's body via sensors in the CGR and updates the CGR environment in response to determining another change in the user's cognitive state. For example, method 1000 may determine that the user's cognitive state has become peaceful during a break from a work activity and may change back from a rest environment (e.g., a forest) to a work environment (e.g., a coffee shop).

[0091] In some embodiments, method 1000 provides feedback to the user based on the user's cognitive state. For example, method 1000 may provide a notification to the user to take a break. In some embodiments, updating the CRG environment includes displaying a message to make the user aware of his or her cognitive state.

[0092] In some embodiments, method 1000 provides feedback to content creators to facilitate improvement of the content or future / related content. The notification may identify a portion of the content associated with the cognitive state of one or more users, e.g., identifying that a particular surrounding environment is preferred for content users performing the same activity. In some embodiments, the cognitive state data of multiple users who have viewed the surrounding environment is aggregated to provide feedback about the surrounding environment. Content creators may modify the surrounding environment based on such feedback to further enhance the cognitive state of users experiencing the CGR content.

[0093] It should be understood that the specific embodiments described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. Instead, the scope includes both combinations and sub - combinations of the various features described above, as well as variations and modifications of the various features that will occur to those skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0094] As described above, one aspect of the technology of the present invention is to collect and use data to improve the user experience of an electronic device in using electronic content. The present disclosure contemplates that, in some cases, the data collected may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or propensities of a particular person. For example, the device can learn the preferences for various different users (e.g., enabled by biometrics). Such personal information data may include physiological data, demographic data, location-based data, phone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0095] The present disclosure recognizes that the use of such personal information data in the technology of the present invention can be used to benefit the user. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control of the electronic device. In addition, the present disclosure also anticipates other uses of personal information data that are beneficial to the user.

[0096] The present disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other uses of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate purposes. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any necessary steps to safeguard and protect access to such personal information data and ensure that others who can access the personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove their compliance with widely accepted privacy policies and practices.

[0097] Regardless of the foregoing, the present disclosure also contemplates specific implementations where the user selectively blocks the use or access of personal information data. That is, the present disclosure anticipates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, in the case of a content delivery service customized for the user, the technology of the present invention can be configured to allow the user to select to "opt in" or "opt out" of participating in the collection of personal information data during the registration of the service. In another example, the user can choose not to provide personal information data for a targeted content delivery service. In yet another example, the user can choose not to provide personal information but allow the transmission of anonymous information for improving the function of the device.

[0098] Accordingly, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that the various embodiments may also be implemented without access to such personal information data. That is, the various embodiments of the present inventive technology will not fail to operate properly due to the lack of all or a portion of such personal information data. For example, preferences or settings may be inferred based on non-personal information data or minimal amounts of personal information such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information, and content may be selected and delivered to the user accordingly.

[0099] In some embodiments, a public key / private key system that allows only the owner of the data to decrypt the stored data is used to store the data. In some other specific implementations, the data may be stored anonymously (e.g., without identifying and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some specific implementations, a user may access their stored data from a user device different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.

[0100] Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, well-known methods, devices, or systems have not been described in detail so as not to obscure the claimed subject matter.

[0101] Unless otherwise specifically stated, it should be understood that throughout the specification, discussions using terms such as "processing," "computing," "computed," "determining," and "identifying" refer to actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0102] One or more of the systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditional on one or more inputs. Suitable computing devices include computer systems based on multipurpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device into a special-purpose computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages may be used to implement the teachings contained herein in the software used to program or configure the computing device.

[0103] Specific implementations of the methods disclosed herein may be carried out in the operation of such computing devices. The order of the blocks presented in the above examples may vary; for example, the blocks may be reordered, combined, or broken into sub-blocks. Certain blocks or processes may be performed in parallel.

[0104] The use of "configured to" or "adapted to" in this document means open and inclusive language that does not exclude devices that are adapted to or configured to perform additional tasks or steps. Additionally, the use of "based on" is open and inclusive because a process, step, calculation, or other action "based on" one or more of the stated conditions or values may, in practice, be based on additional conditions or values beyond those stated. The headings, lists, and numbers included in this document are for ease of explanation only and are not intended to be restrictive.

[0105] It will also be understood that although terms such as "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. The first node and the second node are both nodes, but they are not the same node.

[0106] The terms used herein are for the purpose of describing particular implementations only and are not intended to limit the claims. As used in the description of this implementation and the appended claims, the singular forms "a" and "the" are intended to also include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "comprises" or "comprising," when used in this specification, specify the presence of the stated features, integers, steps, operations, objects, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, objects, components, or groups thereof.

[0107] As used herein, the term "if" can be interpreted to mean "when the precondition is true" or "while the precondition is true" or "in response to determining" or "in accordance with determining" or "in response to detecting" the precondition is true, depending on the context. Similarly, the phrases "if it is determined [that the precondition is true]" or "if [the precondition is true]" or "when [the precondition is true]" are interpreted to mean "when it is determined that the precondition is true" or "in response to determining" or "in accordance with determining" the precondition is true or "when it is detected that the precondition is true" or "in response to detecting" the precondition is true, depending on the context.

[0108] The foregoing description and summary of the invention should be understood to be illustrative and exemplary in every respect and not restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the illustrative embodiments, but by the full breadth permitted by patent law. It should be understood that the specific embodiments shown and described herein are merely illustrative of the principles of the invention and that various modifications can be effected by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method for identifying a surrounding environment that enhances a user's cognitive state of a user activity, the method comprises: at a device including a processor: presenting a computer-generated reality (CGR) environment during an activity, wherein the CGR environment presents multiple surrounding environments at different times during the participation in the activity; when presenting the multiple surrounding environments in the CGR environment, obtaining data about the user's body via a sensor; determining a first cognitive state based on the obtained data about the user's body; providing feedback to the user based on the first cognitive state; detecting a change in the user's activity, wherein the activity is selected from the group including work, study, and rest; and based on the change in the user's activity, identifying and presenting a surrounding environment in the multiple surrounding environments that enhances a second cognitive state of the user for the changed activity, the second cognitive state being different from the first cognitive state.

2. The method according to claim 1, further comprising changing a parameter of the surrounding environment and identifying a parameter that enhances the second cognitive state of the user for the changed activity.

3. The method according to claim 2, wherein the parameter is selected from the group consisting of: color, light level, decoration, sound or music, and volume.

4. The method according to claim 1, wherein the multiple surrounding environments are evaluated based on comparing the obtained data about the user's body in response to at least a first surrounding environment and a second surrounding environment in the multiple surrounding environments.

5. The method according to claim 1, wherein the sensor is selected from the group consisting of: a heart rate sensor, a pulse oximeter, a blood pressure sensor, a temperature sensor, an electrocardiogram (EKG) sensor, an electroencephalogram (EEG) sensor, an electromyogram (EMG) sensor, a functional near-infrared spectroscopy signal (fNIRS) sensor, and a galvanic skin response (GSR) sensor.

6. The method according to claim 1, wherein the sensor is selected from the group consisting of an inward-facing camera and a downward-facing camera.

7. The method according to claim 6, wherein the data about the user's body includes hand / body tracking, and further comprising using the hand / body tracking to identify movements associated with the first cognitive state and the second cognitive state.

8. The method according to claim 1, wherein the multiple surrounding environments are evaluated by assessing the degree to which each of the surrounding environments enhances the second cognitive state.

9. A method for computer-generated reality (CGR) environment based on a detected change in a user's cognitive state, the method comprises: at a device including a processor: presenting a computer-generated reality (CGR) environment configured based on a first activity; obtaining data about the user's body via a sensor in the CGR environment; determining the user's cognitive state based on the obtained data about the user's body; and Update a CGR environment for a second activity configuration based on a recommendation different from the first activity, where the first activity and the second activity are selected from the group consisting of: work, study, and rest, based on the determined cognitive state.

10. The method according to claim 9, wherein the sensor is selected from the group consisting of: a heart rate sensor, a pulse oximeter, a blood pressure sensor, a temperature sensor, an electrocardiogram (EKG) sensor, an electroencephalogram (EEG) sensor, an electromyogram (EMG) sensor, a functional near-infrared spectroscopy signal (fNIRS) sensor, and a galvanic skin response (GSR) sensor.

11. The method according to claim 9, wherein the sensor is selected from the group consisting of an inward-facing camera and a downward-facing camera.

12. The method according to claim 11, wherein the data about the user's body includes hand / body tracking, and further includes using the hand / body tracking to identify movements associated with the cognitive state.

13. The method according to claim 9, wherein the CGR environment is automatically updated based on the second activity different from the first activity.

14. The method according to claim 9, further comprising providing a recommendation to the user and receiving an input from the user.

15. The method according to claim 9, further comprises: Obtaining second data about the user's body via the sensor in the CGR environment; Determining a second cognitive state of the user based on the obtained second data about the user's body; and Based on the second cognitive state, presenting the CGR environment based on a third activity configuration different from the second activity.

16. A non-transitory computer-readable storage medium storing computer-executable program instructions for performing operations including: Presenting a computer-generated reality (CGR) environment during an activity, where the CGR environment presents multiple surrounding environments at different times while the user is engaged in the activity; Obtaining data about the user's body via a sensor when presenting the multiple surrounding environments in the CGR environment; Determining a first cognitive state based on the obtained data about the user's body; Providing feedback to the user based on the first cognitive state; Detecting a change in the user's activity, where the activity is selected from the group including work, study, and rest; and Based on the change in the user's activity, identifying and presenting a surrounding environment in the multiple surrounding environments that enhances the user's second cognitive state of the changed activity, the second cognitive state being different from the first cognitive state.

17. The non-transitory computer-readable storage medium according to claim 16, wherein the operations further include changing parameters of the surrounding environment and identifying parameters that enhance the user's second cognitive state of the changed activity.

18. The non-transitory computer-readable storage medium according to claim 16, wherein the surrounding environment is evaluated based on the data obtained regarding the body of the user by comparing in response to at least a first surrounding environment and a second surrounding environment among the plurality of surrounding environments.

Images