User feedback based on retention prediction

Abstract

Various implementations disclosed herein include devices, systems, and methods to evaluate physiological data of a user and contextual understanding of a physical environment to determine retention status. For example, an example process may include obtaining physiological data in an environment during a first time period. The process may also include identifying one or more of the objects in the environment based on determining the scene understanding of the environment. The process may also include determining a feature associated with an interaction event for the one or more subjects based on the physiological data and the scene understanding. The process may also include determining a retention state during the first time period based on the feature, the retention state associated with a retention of a concept associated with an object of the one or more objects. The process may also include providing feedback based on identifying the retention state.

Description

Technical Field

[0001] The present disclosure relates generally to electronic devices, and in particular to systems, methods, and devices for determining a reservation state based on characteristics associated with an interaction event. Background Art

[0002] Electronic devices can be used to help users engage in various experiences where a specific user state is desired. Predicting subsequent factual memory while viewing content on an electronic device can facilitate a more meaningful experience. For example, an electronic device can be used to present content for an educational experience where a user desires to learn. Such content is typically not responsive or adaptable to the user's retention state, and therefore may not be as effective or efficient as desired. Content creators and systems may be able to provide a better and more customized user experience for concept retention based on predicting subsequent factual memory with greater accuracy as the user interacts with portions of the content. Summary of the invention

[0003] Various embodiments disclosed herein include devices, systems, and methods for presenting content and evaluating retention states associated with retention of concepts. Retention states may refer to whether an individual remembers a fact or concept or to a level of retention, such as how many of a set of facts or concepts are retained, how well a set of facts or concepts are retained, etc. Various embodiments disclosed herein include devices, systems, and methods that focus on obtaining physiological data (e.g., eye tracking data, head movement, and hand gestures) and scene understanding to identify specific interactions with virtual / real world objects that indicate memory / concept retention during a learning / productivity experience so as to provide feedback by modifying (e.g., enhancing) content based on retention states. For example, visual and / or audio notifications may be provided to a user to focus on a particular object or a portion of an object to improve concept retention (e.g., improve memory). Interactions over time are used to determine features that are input into a machine learning algorithm that is trained to predict subsequent fact memory. For example, a user study / experiment may include having a user focus on a particular phrase or a portion of an image to be remembered, and when the user is focusing his or her attention on the phrase (or a portion of an image), physiological data may be collected and determined as features of the interaction event. Some of these features (e.g., machine learning inputs) may include time spent fixating on an object, number of revisits to the same object, average change in pupil diameter across different visits, blinks during different visits, saccade amplitude / speed when exploring an object, etc.

[0004] Physiological data may include eye tracking data (e.g., gaze direction, gaze angle, pupil diameter, pupil location, etc. obtained via a video-based eye tracker, retinal / fundus imaging-based eye tracking, electrooculogram-based eye tracking, magnetic scleral search coil-based eye tracking, etc.), head representation / tracking data (e.g., head posture, rotation speed, etc. using image data, depth data, accelerometer data, magnetometer data, gyroscope data, etc.), hand tracking data (e.g., hand posture, movement speed, etc. using image data, depth data, accelerometer data, magnetometer data, gyroscope data, etc.) or other physiological data (e.g., heart rate, respiratory rate, body temperature, electrocardiogram, blood oxygen saturation, skin conductance, etc.).

[0005] The context of the environment may additionally be used to determine retention state. For example, scene analysis of an experience may determine scene understanding of visual and / or auditory attributes associated with content presented to the user (e.g., what is presented in video content) and / or attributes associated with the user's environment (e.g., where the user is, what the user is doing, what objects are nearby). These attributes of both the presented content and the user's environment may improve determination of user concept retention based on his or her eye movements and gaze behavior with respect to interaction events (e.g., focusing on an area of ​​an object).

[0006] Physiological data such as EEG amplitude / frequency, sensor data corresponding to pupil modulation, sensor data corresponding to eye gaze saccades, etc. may depend on the individual, the characteristics of the scene in front of him or her (e.g., video content), the properties of the physical environment surrounding the user (including the user's activities / movements). Physiological data can be obtained using a device with eye tracking technology (and other physiological sensors) while the task is being performed. In some specific implementations, other sensors (such as electroencephalogram (EEG) sensors or electrodermal activity (EDA) sensors) can be used to obtain physiological data. Observing repeated measurements of physiological data experienced can give insights into the user's intentions based on his or her eye movements and gaze behavior.

[0007] In some embodiments, determining a retention state associated with retention of a concept associated with an object may be based on characteristics of a user (e.g., physiological data) and the user's environment (e.g., a real-world physical environment, a virtual environment, or a combination of each). A device (e.g., a handheld device, a laptop computer, a desktop computer, or a head-mounted device (HMD)) provides an experience (e.g., a visual experience and / or an auditory experience) of a real-world physical environment or an extended reality (XR) environment. The device utilizes one or more sensors to obtain physiological data associated with the user (e.g., EEG amplitude, pupil modulation, eye gaze saccades, head movement measured by an inertial measurement unit (IMU), etc.). Based on the obtained physiological data, the technology described herein may determine an interaction event during the experience. Based on the physiological data and the associated physiological response (e.g., a user focusing on a particular area of ​​the content), the technology may provide a response to the user based on the interaction event and adjust the content corresponding to the experience.

[0008] Some implementations evaluate physiological data and other user information to help improve the user experience. In such processes, user preferences and privacy should be respected by, for example, ensuring that the user understands and consents to the use of user data, understanding what type of user data is used, controlling the collection and use of user data, and limiting the distribution of user data (e.g., by ensuring that user data is processed locally on the user's device). Users should have the option to opt-in or opt-out of whether their user data is obtained or used or otherwise turn on and off any features that obtain or use user information. In addition, each user should have the ability to access and otherwise find out anything that the system has collected or determined about him or her.

[0009] Generally speaking, an innovative aspect of the subject matter described in this specification may be embodied in a method comprising the following actions: at an electronic device having a processor and one or more sensors, obtaining physiological data in an environment during a first time period using the one or more sensors, the environment including one or more objects; identifying one or more of the objects in the environment based on determining scene understanding of the environment; determining features associated with interaction events with the one or more objects based on the physiological data and the scene understanding; determining a retention state during the first time period based on the features, the retention state being associated with retention of a concept associated with the object in the one or more objects; and providing feedback based on identifying the retention state.

[0010] These and other embodiments can optionally include one or more of the following features.

[0011] In some aspects, physiological data include position information and speed information for the left eye and the right eye. In some aspects, physiological data include the pupil distance (IPD) between the right eye and the left eye, and the diameter of the left eye and the diameter of the right eye. In some aspects, physiological data include gaze direction, gaze speed or pupil fixation. In some aspects, physiological data include images of the eyes or electrooculogram (EOG) data. In some aspects, physiological data include images of the head from one or more sensors. In some aspects, obtaining physiological data includes tracking the posture and movement of the head.

[0012] In some aspects, determining scene understanding of the physical environment includes determining the location, speed, or direction of motion of at least one of the one or more objects. In some aspects, determining scene understanding includes generating a three-dimensional (3D) representation of the physical environment. In some aspects, scene understanding is based on obtaining sensor data from one or more sensors corresponding to the physical environment.

[0013] In some aspects, the one or more objects include a plurality of objects, and wherein determining the retention status during the first time period is based on evaluating transitions between the plurality of objects.

[0014] In some aspects, the determined features include at least one of time spent fixated on one or more objects, number of revisits to one or more objects, average change in pupil diameter during a first time period, blinks during different subsequent time periods, or saccade amplitude / velocity when exploring one or more objects during a first time period. In some aspects, the features are identified based on user experiments for a plurality of different users, the user experiments including user instructions to focus on a particular object of the one or more objects during a study time period, wherein physiological data is obtained for the plurality of different users during the study time period.

[0015] In some aspects, determining features associated with interaction events for one or more objects is based on determining salient regions and non-salient regions associated with the one or more objects, and determining gaze intersections with the salient regions or the non-salient regions.

[0016] In some aspects, determining characteristics associated with an interaction event for one or more objects includes determining scene-induced pupil response change characteristics for a particular object among the one or more objects, and determining an interaction event based on the scene-induced pupil response change characteristics for the particular object.

[0017] In some aspects, providing feedback based on identifying the retention state includes updating a view of a display of the electronic device during a first time period. In some aspects, updating the view of the display is based on physiological data. In some aspects, providing feedback based on identifying the retention state includes adjusting content in response to determining an interaction event associated with retention of a concept associated with the object. In some aspects, providing feedback based on identifying the retention state includes providing a quality level of retention of the concept associated with the object, and providing a notification based on the concept associated with the object.

[0018] In some aspects, the method may also include: determining a plurality of retention states during a plurality of time periods based on the features; aggregating the determined features associated with the plurality of retention states associated with a plurality of objects in the one or more objects; and updating the feedback based on the aggregated features.

[0019] In some aspects, the retention state during the first time period is based on using a machine learning classifier model, wherein the physiological data is input into the machine learning classification model to identify retention of a concept associated with the object. In some aspects, the machine learning classification model is trained based on a plurality of eye characteristics, head posture characteristics, and hand gesture characteristics.

[0020] In some aspects, the physiological data is obtained from sensor data from one or more sensors, and wherein the sensor data includes depth data and light intensity image data obtained during a scanning process.

[0021] In some aspects, the environment is a physical environment. In some aspects, the environment is a virtual environment. In some aspects, the environment is a mixed reality (MR) experience. In some aspects, the electronic device is a head mounted device (HMD).

[0022] According to some specific implementations, a non-transitory computer-readable storage medium stores instructions that are computer executable to perform or cause the performance of any of the methods described herein. According to some specific implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order that the present disclosure may be understood by those skilled in the art, a more particular description will be given with reference to some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0024] Figure 1 Devices for obtaining sensor data from a user according to some implementations are illustrated.

[0025] Figure 2A and Figure 2B Obtaining physiological data of a user viewing content according to some implementations is illustrated.

[0026] Figure 3 A diagram illustrating transitions of eye gaze characteristics between multiple objects according to some implementations.

[0027] Figure 4 Illustrations of devices in a physical environment according to some implementations.

[0028] Figure 5 According to some specific implementations, Figure 4 The view of the gaze / attention tracker of the view.

[0029] FIG. 6A to FIG. 6C Illustrative views of an electronic device viewing an XR environment according to some implementations are illustrated.

[0030] Figure 7 A diagram of a system for evaluating a user's physiological data and scene understanding of a physical environment to determine a retention state is illustrated according to some implementations.

[0031] Figure 8 is a flowchart representation of a method for evaluating a user's physiological data and scene understanding of a physical environment to determine a retention state according to some specific implementations.

[0032] Fig. 9 is a block diagram illustrating device components of an exemplary device according to some implementations.

[0033] Fig.10 is a block diagram of an example head mounted device (HMD) according to some specific implementations.

[0034] As is common practice, the various features illustrated in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0035] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the example implementations described herein.

[0036] Figure 1 An example environment 100 of a real-world environment 105 (e.g., a room) is illustrated, which includes a device 10 having a display 15. In some implementations, the device 10 displays content 20 to a user 25. For example, the content 20 can be a button, a user interface icon, a text box, a graphic, an avatar of the user or another user, etc. In some implementations, the content 20 can occupy the entire display area of ​​the display 15.

[0037] The device 10 obtains image data, motion data, and / or physiological data (e.g., pupil data, facial feature data, etc.) from the user 25 via multiple sensors (e.g., sensors 35a, 35b, and 35c). For example, the device 10 obtains eye gaze characteristic data 40b via sensor 35b, upper facial feature characteristic data 40a via sensor 35a, and lower facial feature characteristic data 40c via sensor 35c. Additionally, each of the multiple sensors (e.g., sensors 35a, 35b, and 35c) may capture information about the position, location, motion, posture, etc. of the head 27 and / or body of the user 25.

[0038] Although this example and other examples discussed herein illustrate a single device 10 in a real-world environment 105, the techniques disclosed herein are applicable to multiple devices and other real-world environments. For example, the functionality of device 10 may be performed by multiple devices, with sensors 35a, 35b, and 35c located on each respective device, or distributed among them in any combination.

[0039] In some implementations, the plurality of sensors (e.g., sensors 35a, 35b, and 35c) may include any number of sensors that acquire data related to the appearance of user 25. For example, when a head mounted device (HMD) is worn, one sensor (e.g., a camera within the HMD) may acquire pupil data for eye tracking, and one sensor on a separate device (e.g., one camera, such as a wide angle view) may be able to capture all of the user's facial feature data. Alternatively, if device 10 is an HMD, a separate device may not be necessary. For example, if device 10 is an HMD, in one embodiment, sensor 35b may be located within the HMD to capture pupil data (e.g., eye gaze characteristic data 40b), and additional sensors (e.g., sensors 35a and 35c) may be located on the HMD but on an outer surface of the HMD facing the user's head / face to capture facial feature data (e.g., upper facial feature characteristic data 40a via sensor 35a, and lower facial feature characteristic data 40c via sensor 35c) and capture information about the position, location, movement, posture, etc. of the head 27 and / or body of the user 25.

[0040] In some specific implementations, such as Figure 1 As illustrated, device 10 is a handheld electronic device (e.g., a smartphone or tablet computer). In some implementations, device 10 is a laptop or desktop computer. In some implementations, device 10 has a touchpad, and in some implementations, device 10 has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some implementations, electronic device 10 can be worn by a user. For example, electronic device 10 can be a watch, HMD, head-mounted device (glasses), headphones, ear-hook devices, etc. In some implementations, the functions of device 10 are implemented via two or more devices, such as a mobile device and a base station or a head-mounted device and an ear-hook device. Various functions may be distributed in multiple devices, including but not limited to power functions, CPU functions, GPU functions, storage functions, memory functions, visual content display functions, audio content production functions, etc. Multiple devices that can be used to implement the functions of electronic device 10 can communicate with each other via wired communication or wireless communication and / or via an intermediate device such as a playback session server.

[0041] In some implementations, device 10 includes an eye tracking system for detecting eye position and eye movement via eye gaze characteristic data 40b. For example, the eye tracking system may include one or more infrared (IR) light emitting diodes (LEDs), an eye tracking camera (e.g., a near IR (NIR) camera), and an illumination source (e.g., a NIR light source) that emits light (e.g., NIR light) toward the eyes of user 25. In addition, the illumination source of device 10 may emit NIR light to illuminate the eyes of user 25, and the NIR camera may capture images of the eyes of user 25. In some implementations, the images captured by the eye tracking system may be analyzed to detect the position and movement of the eyes of user 25, or to detect other information about the eyes such as color, shape, state (e.g., wide open, squinting, etc.), pupil dilation, or pupil diameter. In addition, the gaze point estimated from the eye tracking image may enable gaze-based interaction with content shown on a near-eye display of device 10.

[0042] In some implementations, the device 10 has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing multiple functions. In some implementations, the user 25 interacts with the GUI through hand gestures, finger contacts, and gestures on the touch-sensitive surface. In some implementations, these functions include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, and / or digital video playback. Executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0043] In some implementations, the device 10 uses various physiological sensor detection or measurement systems. The detected physiological data may include, but is not limited to: electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG), functional near infrared spectroscopy signal (fNIRS), blood pressure, skin conductance, or pupil response. In addition, the device 10 can detect multiple forms of physiological data simultaneously in order to benefit from the synchronous acquisition of physiological data. In addition, in some implementations, the physiological data represents involuntary data, that is, a response that is not under conscious control. For example, a pupil response may represent an involuntary movement.

[0044] In some implementations, one or both eyes 45 of the user 25 (including one or both pupils 50 of the user 25) present physiological data (e.g., eye gaze characteristic data 40b) in the form of pupil responses. The pupil responses of the user 25 result in changes in the size or diameter of the pupil 50 via the optic nerve and the oculomotor cranial nerve. For example, the pupil response may include a constriction response (miosis) (e.g., a narrowing of the pupil) or an expansion response (mydriasis) (e.g., a widening of the pupil). In some implementations, the device 10 may detect a pattern of physiological data representing a time-varying pupil diameter. In some implementations, the device 10 may also determine the pupil distance (IPD) between the right eye and the left eye of the user.

[0045] User data (e.g., upper facial feature characteristic data 40a, lower facial feature characteristic data 40c, and eye gaze characteristic data 40b), including information about the position, location, movement, posture, etc. of the head 27 and / or body of the user 25, may change over time, and the device 10 may use the user data to generate and / or provide a representation of the user.

[0046] In some implementations, the user data (e.g., upper facial feature characteristic data 40a and lower facial feature characteristic data 40c) includes texture data of facial features, such as eyebrow movements, jaw movements, nose movements, cheek movements, etc. For example, when a person (e.g., user 25) smiles, the upper facial features and lower facial features (e.g., upper facial feature characteristic data 40a and lower facial feature characteristic data 40c) may include a number of muscle movements that can be repeated by a representation of the user (e.g., an avatar) based on captured data from sensor 35.

[0047] Physiological data (e.g., eye data, head / body data, etc.) may change over time, and device 10 may use the physiological data to measure one or both of a user's physiological response to visual characteristics 30 (e.g., viewing specific content on display 15, such as reading text) or a user's intention to interact with content 20. For example, when content 20 that may include interactive elements is presented by device 10, user 25 may select interactive elements based on the user's eye movements and gaze behavior associated with eye data, head / body data, etc., without requiring user 25 to complete physical button presses. In some implementations, the physiological data may include a physiological response to a visual stimulus or an auditory stimulus of a radius of pupil 50 after user 25 glances at content 20, measured via eye tracking technology (e.g., via an HMD). In some implementations, the physiological data includes EEG amplitude / frequency data measured via EEG technology or EMG data measured from an EMG sensor or motion sensor.

[0048] Figure 2A and Figure 2B Detecting eye movements and gaze behaviors of a user viewing content based on physiological data is illustrated. Figure 2A 204 to a user (e.g., Figure 1 The user 25) is presented with content 202, wherein the user has a physiological response to the content via the obtained physiological data (e.g., the user looks at a portion of the content as detected by the eye gaze characteristic data 212). For example, at content presentation moment 200A, the user is presented with content 202 including visual content (e.g., a video / image of an application including text), and the user's physiological data, such as pupil data 214, head data 216, and body data 218, is monitored. Figure 2B Illustrated with Figure 2A A similar example, except that the user focuses his or her gaze on interactive element 203 (e.g., the user wants to select text presented to him or her, such as an embedded interactable icon), as illustrated at selection notification 205. Thus, content 202 may be updated based on the user's interaction / focus on interactive element 203 (e.g., the user wants to select the embedded interactable icon represented by interactive element 250).

[0049] exist Figure 2A and Figure 2B In the specific example of , at the content presentation moment 200A, the user's eye gaze characteristics are less focused on the content 202. Then, at Figure 2B At the content presentation moment 200B (e.g., during the focus phase), the user's eye gaze characteristics 212 appear to be focused on the interactive element 203 of the content 202. For example, the system can determine based on the user's eye gaze characteristics 212, pupil data 214, head data 216, and / or body data 218 that the user's gaze may be focused on a particular area or text, and is no longer just looking at the content (e.g., reading the text), but rather wants to select a portion of the content (e.g., select an icon or interact with a portion of the text, such as placing an order on a menu).

[0050] Figure 3 A diagram 300 of transitions of eye gaze characteristics between multiple related objects and unrelated objects according to some implementations is illustrated. For example, learning can involve transitions between objects and linked ideas. Diagram 300 provides an example illustration of how a user can transition between objects in a learning experience. Specifically, diagram 300 provides an illustration of transitions between related objects 314, 318, 320, 324 and unrelated objects 310, 312, 316, 322, 326, such as transition 302 from related object 314 to unrelated object 310 and transition 304 from unrelated object 312 to related object 318. In some implementations, a content provider (e.g., a developer for an educational program) can determine which objects or learning objectives will be considered related objects and unrelated objects (e.g., 0 or 1 for tracking purposes). The diagram 300 can be simplified based on relevance tracking as illustrated in the simplified diagram 330 and transition table 340 to track key transitions (e.g., R→R and IR→R) from other related objects "R" or from irrelevant objects "IR". The relevance transition index formula 350 provides a score for gaze transitions to related objects. The relevance transition index formula 350 can be used as an input to a machine learning algorithm to predict a user's concept retention based on the user's transitions between objects in a learning segment. Figure 3 The illustrated use of tracking gaze transitions between relevant and irrelevant objects is unique for memory prediction because it requires custom annotations but only a single simple binary label from the content developer (e.g., determining whether an object is relevant to the learning experience), and uses this information to derive a measure of structure in gaze patterns over time. In some implementations, correlations can be used to further determine gaze features, such that gaze features relative to respective relevant objects can predict subsequent fact memory.

[0051] In some implementations, inference related objects can be determined based on feedback from content providers, which is based on what topics are considered important by experts in the subject area view of the content. For example, objects that subject matter experts spend time on (e.g., greater than a threshold in seconds) will be relevant, while objects they ignore (e.g., less than a threshold in seconds) will be irrelevant. Additionally, the relevance percentage can be determined based on averaging across different subject matter experts. Averaging across different subject matter experts can also help determine the degree of consistency of different experts. In some implementations, related objects can be determined from beginners who may have just learned a particular concept. However, the subsequent memory of beginners can be used to find out the percentage relevance. In other words, users (e.g., learners) who are really good in their evaluations can be considered experts, and the relevance percentage can be determined as described herein afterwards.

[0052] Figure 4 A view of a device (e.g., device 10) in a physical environment 400 according to some implementations is illustrated. Physical environment 400 is a city park with a city (buildings) in the background, which includes identifiable objects within the view of device 10. For example, objects include bystanders 420 and 422 and information sign 440 including content (e.g., a notice or an information board such as a historical marker). In some implementations, the view of physical environment 400 can be an image of the city park (e.g., a color photograph), so information sign 440 is a physical sign at the park (e.g., a historical board). Alternatively, the view of physical environment 400 can be a fully or partially immersive interactive experience with virtual content, such as an XR experience (e.g., when device 10 is worn as an HMD); thus, information sign 440 can be a physical sign at the park (e.g., a historical board) or can be virtual content that the user is trying to study and possibly remember the image or something written on it (e.g., a historical fact).

[0053] Figure 5 According to some specific implementations, Figure 4 4. For example, view 500 illustrates gaze locations on a representation of a view of physical environment 400 based on objects within the view. For example, gaze locations include gaze point 512 on representation 510 of the sun, gaze point 522 on representation 520 of bystander 420, gaze point 526 on representation 524 of bystander 422, gaze point 532 on representation 530 of a tree, and multiple gaze points (e.g., gaze points 542, 544, 546, 548) at or near representation 540 of information sign 440. Specifically, Figure 5How objects can be divided into salient areas and non-salient areas is illustrated, where salient areas within an object are considered to attract people's attention. In an exemplary embodiment, if the view of the physical environment 400 is an image and is classified as a single object, the content developer of the information within the image may want to attract the user's attention to the content displayed on the information sign 440. For example, according to the technology described herein, the area or area can be classified as a salient area (e.g., salient area 550). In particular, the fixation intersection on these salient areas (e.g., salient area 550) can lead to better predictive capabilities for memory classification performed by a machine learning classifier. In the exemplary technology described herein, a method may include determining features associated with interactive events for one or more objects based on determining salient areas and non-salient areas (e.g., salient areas 550) associated with one or more objects and determining fixation intersections with salient areas or non-salient areas (e.g., determining that the user focuses his or her gaze on the salient area 550).

[0054] FIG. 6A to FIG. 6C An exemplary view of an electronic device viewing an XR environment according to some specific implementations is illustrated. Specifically, FIG. 6A to FIG. 6C Various examples of displaying feedback content (e.g., virtual content) to a user viewing an XR environment are illustrated. For example, FIG. 6A to FIG. 6C An exemplary electronic device 10 is illustrated that provides a view 615A of a 3D environment 612A, a view 615B of a 3D environment 612B, and a view 615C of a 3D environment 612C, respectively, while operating in a physical environment 600 during viewing of content (e.g., a learning experience while walking in a city park). For example, FIG. 6A to FIG. 6C This may represent viewing of content at three different time periods while user 25 is viewing content on the display of device 10 and viewing physical environment 600. FIG. 6A to FIG. 6C In these examples, physical environment 600 is an urban park with a city (buildings) in the background including bystanders 620 and 622 and information sign 610 including content (e.g., notices or information boards such as historical markers). Specifically, FIG. 6A to FIG. 6C Each illustrates a user 25 (e.g., a viewer during the experience) viewing content on a device 10 during sunrise (e.g., in the morning) in a city park with a city view in the background. The informational sign 610 may be a physical sign at the park that the user sees in the pass-through video (e.g., a history board), or may be an informational sign 610 of virtual content that the user is trying to research and perhaps remember an image or something written on it (e.g., a historical fact).

[0055] The electronic device 10 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 600 and objects therein and information about the user 25 of the electronic device 10 (e.g., physiological data as discussed herein, location data of the user 25, etc.) and to evaluate the physical environment and objects therein. Information about the physical environment 600 and / or the user 25 can be used to provide feedback while the user is viewing visual content and / or listening to audio content during the experience. For example, a learning session can provide views (e.g., views 615A, 615B, and 615C) of a 3D environment (e.g., 3D environments 612A, 612B, and 612C) generated based on camera images and / or depth camera images of the physical environment 100, and optionally include virtual content as part of the meditation experience to simulate a nature walk in a city park, but supplement some of the physical content with virtual content (e.g., content enhancements, such as feedback data to assist the user in the learning experience).

[0056] exist Fig. 6A In the illustrated example, electronic device 10 provides view 615A that includes representation 630 of bystander 620 , representation 632 of bystander 622 , and representation 612 of information sign 610 . Fig. 6A A view 615A of the simulated environment 600 is provided, i.e., a view showing a representation including the sun in the same position (e.g., during sunrise). The view 615A also includes a retention indicator 650 that can be displayed to the user 25 during the experience, which indicates to the user 25 how beneficial the current view 615A may be for concept retention based on the physiological data of the user 25 (e.g., gaze characteristics, etc.). Alternatively, in some embodiments, the retention indicator 650 may be interactive (e.g., the user 25 may be able to slide a bar) to allow the user 25 to change one or more attributes associated with the view 615A to improve the retention level such as the immersion level (e.g., blocking potential distractions, changing weather, time of day (sunrise / sunset / moon), etc.) and / or provide feedback to improve the learning experience. Alternatively, the experience level indicator is not displayed in the view 615A, but rather stores data for the content developer.

[0057] Similarly, in Figure 6B In the illustrated example, electronic device 10 provides view 615B that includes representation 640 of bystander 620 , representation 642 of bystander 622 , representation 612 of information sign 610 , and reservation indicator 650 . Figure 6BA view 615B is provided that also includes feedback data (e.g., virtual content), such as a feedback content element 660 (e.g., a notification text bubble) and a feedback content element 662 (e.g., an arrow pointing to a representation 612 of a related object, the information placard 610). Additionally, in some implementations, the environment may be altered to provide a better learning experience. For example, Figure 6B A view 615B is provided in which the time of day has been changed (e.g., it is now sunset) and the appearance of the representations 640, 642 of the bystanders 620, 622, respectively, has been changed. For example, the user 25 may want to view the background of a city during sunset (e.g., while wearing the HMD) while attempting to learn (e.g., reading from a history board (i.e., information sign 610)). Additionally, the user may select any detected objects that may interfere with the user's learning session, or the system may automatically blur those objects or replace them with virtual objects. For example, if the distance between the person and the user 25 is greater than a certain threshold (e.g., 10 feet or more), the system may change the view of the representations 640, 642 of the bystanders 620, 622, respectively, as shown (e.g., opaque ellipses instead of people). Similarly, in Figure 6C In the illustrated example, electronic device 10 provides a view 615C that includes a representation 680 of the moon, a representation 612 of information sign 610 , and a reservation indicator 650 . Figure 6C A view 615C is provided that also includes feedback data (e.g., virtual content), such as feedback content element 670 and feedback content element 672 (e.g., a highlight of an associated object and / or a prominent area within representation 612 of an associated object—information sign 610). In addition, Figure 6C A view 615C is provided in which the time of day has been changed (e.g., now at night) and representations 640, 642 of bystanders 620, 622, respectively, have been removed. For example, when learning in a city park, the user 25 is viewing the background of the city during nighttime (e.g., while wearing the HMD) to obtain a more beneficial learning experience (e.g., focusing more on relevant objects than irrelevant objects in the background of the view).

[0058] Additionally, the user may select any detected objects that may interfere with the user's learning session, or the system may automatically blur or remove these objects, or replace them with virtual objects. For example, in the example of view 615C, if the distance between the person and the user 25 is greater than a certain threshold (e.g., 10 feet or more), the system may change view 615C and remove the representations 640, 642 of the bystanders 620, 622, respectively. Thus, if the bystanders 620 and / or 622 walk closer to the user (e.g., less than 20 feet), the representations 640, 642, respectively, may begin to fade in slowly as they walk closer to the user 25 until they reach another threshold (e.g., less than 10 feet), and then the representations 640, 642 may be fully shown, or the actual view of the bystanders 620, 622 may "break through" the view 615 so that the user 25 can clearly see that the object is approaching them.

[0059] Figure 7 is a system flow diagram of an example environment 700 according to some implementations, in which a retention evaluation and feedback system can evaluate a user's physiological data and scene understanding of a physical environment to determine a retention state associated with retention of a concept. In some implementations, the system flow of the example environment 700 is performed on a device (e.g., Figure 1 The content of the example environment 700 may be displayed on a device having a screen for displaying images (e.g., display 15) and / or a screen for viewing stereoscopic images (e.g., Figure 1 In some implementations, the system flow of the example environment 700 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the example environment 700 is executed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0060] The system stream of the example environment 700 acquires content (e.g., video content, a series of image data, a single image, etc.) and presents it to the user 25, analyzes the content and / or environment to obtain contextual data (e.g., scene understanding), obtains physiological data associated with the user during the presentation of the content (e.g., eye, head, hand gesture data, etc.), and identifies specific interactions with virtual / real world objects that indicate memory / concept retention during a learning / productivity experience (e.g., an XR experience). For example, if the user 25 focuses on an object within a specific time period, revisits the same object, average changes in pupil diameter across different visits, blinks during different visits, saccade amplitude / speed when exploring an object, transitions between objects (e.g., related objects 705 and related objects 706), etc., then the memory retention of those interaction events with one or more objects can be tracked and evaluated. For example, the retention assessment and feedback techniques described herein determine features associated with interaction events with one or more objects based on the acquired physiological data and scene understanding to identify the one or more objects, determine a retention state associated with retention of concepts associated with the one or more objects during an experience (e.g., a learning experience), and provide feedback (e.g., to a user and / or content developer) based on the identified retention state (e.g., notifications, auditory signals, warnings, etc. that alert the user to a current retention level, suggest where to focus, etc.).

[0061] The example environment 700 includes a content instruction set 710 configured to be executable by a processor to provide and / or track content to be displayed on a device (e.g., Figure 110). For example, when the user is in a physical environment 704 (e.g., a room, outdoors, etc.), the content instruction set 710 provides a content presentation moment 712 including the content 702 to the user 25. The content 702 includes an object 705 and an object 706 (e.g., a related object). For example, the content 702 may include background images and sound data (e.g., a video). The content presentation moment 712 may be an XR experience that includes some virtual content of the physical environment and some images or views. Alternatively, the user may wear the HMD and look at the real physical environment via a live camera view, or the HMD allows the user to view the display, such as wearing smart glasses through which the user can view, but still presents visual cues and / or audio cues. During the experience, while the user 25 is viewing the content 702, eye gaze characteristic data 715 and pupil data 714 of the user's eyes may be monitored and sent as physiological data 717. Other physiological data may be monitored and sent as physiological data 717, such as head movement data 716 and body movement data 718 (e.g., hand gesture data) obtained from an IMU and / or image data. For example, hand gesture data associated with detected hand movements may be obtained based on image analysis, or hand gesture data or other sensor data may be collected by an input device such as a watch. Additional physiological data may be monitored and sent as physiological data 717, such as heart rate, respiratory rate, body temperature, electrocardiogram, blood oxygen saturation, skin conductance, image data (face, body, etc.), EEG amplitude, pupil modulation, eye gaze saccades, etc.

[0062] The environment 700 also includes a physiological tracking instruction set 730 to track physiological attributes of a user as physiological tracking data 732 using one or more of the techniques discussed herein or other techniques as may be appropriate. For example, the physiological tracking instruction set 730 may obtain physiological data 717 (e.g., eye gaze characteristic data 715, pupil data 714) from a user 25 viewing the content 702. Additionally or alternatively, the user 25 may wear a sensor 720 (e.g., such as an EEG sensor, an EDA sensor, a heart rate sensor, etc.) that generates sensor data 722 (e.g., IMU or pose data for the head or body, EEG data, EDA data, heart rate data, etc.) as additional physiological data. Thus, when content 702 is presented to the user as content presentation moment 712, physiological data 717 (e.g., eye gaze characteristic data 715, pupil data 714) and / or sensor data 722 (e.g., head movement data 716 and body movement data 718) are sent to a physiological tracking instruction set 730 to track the user's physiological attributes as physiological tracking data 732 using one or more of the techniques discussed herein or other techniques that may be appropriate. In some implementations, information for the eyes may be obtained for both azimuth and elevation for both the position and velocity of each eye (e.g., eight data channels for a classification algorithm). In some implementations, the eye data may also include IPD information (e.g., one data channel for a classification algorithm) and pupil diameters for the left and right eyes (e.g., two data channels for a classification algorithm). Obtaining the eye data may involve obtaining EEG information, images of the eyes, and / or EOG data from which gaze directions, gaze angles, and / or eye movements may be determined. In some implementations, the eye data may also include gaze direction, gaze speed, or a combination thereof.

[0063] In an example implementation, the environment 700 also includes a scene understanding instruction set 740 configured with instructions that can be executed by a processor to obtain experience data (e.g., content 702) and other sensor data (e.g., image data of the environment 704, image data of the face and / or eyes of the user 25, etc.) presented to the user and generate scene understanding as context data 742 (e.g., identifying people, objects, etc. of the content 702 and the environment 704). For example, the scene understanding instruction set 740 obtains the content 702 and sensor data 722 (e.g., image data) from the sensor 720 (e.g., RGB camera, depth camera, etc.) and determines the context data 742 based on identifying one or more objects in the physical environment 704 and / or a view of the content 702 that may include a view of the physical environment 704 (e.g., identifying objects 705, 706). Alternatively, the scene understanding instruction set 740 selects context data associated with the content 702 from the context database 745 (e.g., if the content 702 has been previously analyzed by the scene understanding instruction set, e.g., a previously viewed / analyzed video). In some implementations, scene understanding instruction set 740 generates scene understanding associated with content 702 and / or environment 704 as context data 742. For example, scene understanding can be utilized to track what a user may be focused on during presentation of content 702, or where the user is, what the user is doing, and the overall context of what physical objects or people are near the user with respect to environment 704.

[0064] In an example implementation, the environment 700 also includes a retention evaluation and feedback instruction set 750 configured with instructions executable by a processor to evaluate physiological data and scene understanding of the physical environment to determine a retention state associated with retention of a concept and provide feedback based on the evaluation using one or more of the techniques discussed herein or other techniques that may be appropriate. Specifically, the retention evaluation and feedback instruction set 750 obtains context data 742 from the scene understanding instruction set 740 and physiological tracking data 732 from the physiological tracking instruction set 730 during the presentation of the content 702 while the user is viewing the content 702 (e.g., reading text or looking at a particular object). The context data 742 can provide scene analysis that can be used by the retention evaluation and feedback instruction set 750 to understand what a person is looking at, where they are, etc., and to improve the evaluation of the user's concept retention relative to related objects (e.g., objects 705, 706, etc.).

[0065] In some implementations, the retention evaluation and feedback instruction set 750 can then provide feedback data 752 to the user 25 via the content instruction set 710 based on the retention status evaluation. In some implementations, the feedback data 752 is provided to the content developer based on identifying the retention status of the user 25. The feedback data 752 may include physiological tracking data 732, contextual data 742, data associated with user interaction with the content 702 (e.g., user clicks on a particular icon, etc.). For example, providing performance feedback during a learning experience can enhance the user's experience, provide additional benefits from the session, and provide guidance and supportive teaching methods (e.g., scaffolding teaching methods) to the user to promote their retention (e.g., feedback content elements 660, notification text bubbles, or arrows pointing out relevant selections).

[0066] Additionally, for example, relevance may be determined for a certain event (e.g., eye movement toward a relevant object), and then the user interface (e.g., content instruction set 710) executes a response, such as a user "clicking" on a user interface element (e.g., some objects are distracting and can be removed, such as bystanders 620, 622 in FIG. 6). When the algorithm detects a higher likelihood of making a non-relevant transition, the content 702 may be updated based on the feedback data to reduce the salience of the non-relevant object (e.g., no longer relevant after a period of time).

[0067] In some implementations, feedback can include encouraging the user to revisit specific objects, such as all related (key) objects. Feedback can include encouraging more relevant transitions by optimally distributing objects in 3D space. In some implementations, feedback data 752 can allow content instruction set 710 or content creators to improve user-object interactions with related objects that will increase recall. For example, instead of creating a read-only text object, make it multimodal using voice. Additionally or alternatively, color can be added to the read-only text object to highlight specific relevant portions and / or categorize information (e.g., highlight specific areas of text, such as Figure 6C 672 in the feedback content element).

[0068] In some implementations, the content instruction set 710 can utilize the feedback data 752 to present an audio and / or visual feedback prompt or mechanism (e.g., a feedback content element 660, a notification text bubble, or an arrow pointing out a relevant selection) to the user 25. In an educational experience, based on an evaluation from the retention evaluation and the feedback instruction set 750 indicating that the user 25's mind is wandering because the user 25 is shifting more to irrelevant objects than to relevant objects, the feedback prompt to the user can be a gentle reminder (e.g., a soothing or calming visual alert and / or audio alert) to resume the learning task.

[0069] In some implementations, the experience database may be used to store information associated with the user 25. For example, historical data may be stored and analyzed at a later time. For example, after each learning session, physiological tracking data 732, contextual data 742, retention levels, etc. may be monitored and stored in the experience database before, during, and / or after the customized feedback content (e.g., feedback data 752) is presented to the user. In some implementations, retention levels may be determined based on analysis of historical data associated with the user 25 for prior learning experiences, thereby determining retention predictions based on specific interactions with virtual / real world objects that indicate memory / concept retention. Alternatively, in some implementations, concept retention levels for one or more related objects may be determined based on accessing a user profile stored in the experience database.

[0070] Figure 8 800 is a flowchart illustrating an exemplary method 800. In some implementations, according to some implementations, an electronic device (e.g., Figure 1 The method 800 is performed on a device 10 of the present invention to evaluate the user's physiological data and scene understanding of the physical environment to determine the retention state. In some implementations, the method 800 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some implementations, the method 800 is performed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 800 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0071] At block 810, method 800 obtains physiological data in an environment, including one or more objects, using one or more sensors during a first time period. For example, eye data, head data, and / or hand gesture data may be obtained during presentation of content in the environment, which may be a physical environment, a virtual environment, or a mixed reality environment. Sensor data (e.g., live data, such as video content including light intensity data (RGB) and depth data) may be associated with a point in time, such as from an inward / downward facing sensor (e.g., such as a camera) while the user is wearing the HMD. Figure 1 An image of the sensors 35a, 35b, 35c) is shown associated with the frame. In some implementations, the sensor data includes depth data (e.g., infrared, time of flight, etc.) and light intensity image data obtained during the scanning process.

[0072] In some implementations, the physiological data includes eye data associated with the gaze acquired during the first time period. In some implementations, information for the eyes may be acquired for both azimuth and elevation for both the position and velocity of each eye (e.g., eight data channels of the classification algorithm). In some implementations, the eye data may also include IPD information (e.g., one data channel for the classification algorithm) and pupil diameters of the left and right eyes (e.g., two data channels for the classification algorithm). Obtaining the eye data may involve obtaining EEG information, images of the eyes, and / or EOG data, from which gaze direction, gaze angle, and / or eye movement may be determined. In some implementations, the eye data may also include gaze direction, gaze velocity, or a combination thereof.

[0073] In some implementations, obtaining physiological data (e.g., pupil data 40) is associated with the user's gaze, which may involve obtaining images of the eye or electrooculogram signal (EOG) data from which gaze direction and / or movement may be determined. In some implementations, the physiological data includes at least one of skin temperature, respiration, photoplethysmography (PPG), electrodermal activity (EDA), eye gaze tracking, and pupil movement associated with the user. In some implementations, obtaining physiological data includes head movement of the user (e.g., obtained from an IMU or from image sensor data), such as head tracking data.

[0074] In some implementations, the physiological data includes head data and body motion data (e.g., hand gesture data) acquired during a first time period. The head / body data may include an image of the user's head / body from one or more sensors. In some implementations, obtaining the head data includes tracking the posture and movement (or hand gestures) of the user's head. For example, head / hand motion and posture may be acquired based on position information and velocity information acquired from one or more sensors. Information for the head / body may be acquired for both azimuth and elevation for both position and velocity (e.g., four data channels of a classification algorithm). Obtaining head / body data may involve obtaining an image of the head (e.g., RGB or IR), an IMU, depth sensing from an IFC, an IR flood, etc.

[0075] In some implementations, obtaining sensor data associated with the head data may include obtaining a second set of data corresponding to one or more partial views of the face from one or more image sensors while the user is using (e.g., wearing) the electronic device (e.g., HMD). For example, obtaining the sensor data may include obtaining live image data. In some implementations, the second set of data includes a partial image of the user's face and, therefore, may not represent all features of the face represented in the enrollment data. For example, the second set of images may include images from an upward-facing sensor (e.g., Figure 1 Additionally or alternatively, the second set of images may include images of some of the front face / eyes of the sensor 35a (e.g., facial feature characteristic data 40a). Additionally or alternatively, the second set of images may include images from the inward-facing sensor (e.g., Figure 1 Additionally or alternatively, the second set of images may include images of some of the eyes from sensors facing downward (e.g., Figure 1 In some implementations, the electronic device includes a first sensor (e.g., Figure 1 sensor 35a) and a second sensor (e.g., Figure 1 The method further comprises the step of: obtaining a second set of data from at least one partial image of the user's face seen from a first viewpoint from the first sensor (e.g., upper facial feature data 40a) and at least one partial image of the user's face seen from a second viewpoint different from the first viewpoint from the second sensor (e.g., lower facial feature data 40c) (e.g., multiple IFC cameras are used to capture different viewpoints of the user's face and body movements).

[0076] In some implementations, the movement and / or positioning and features of the user's 25 head 27 (e.g., the edges of the eyes, nose, or nostrils) are extracted by the device 10 and used to find the coarse positioning coordinates of the user's 25 eyes 45, thereby simplifying the determination of precise eye 45 features (e.g., position, gaze direction, etc.), and making the gaze characteristic measurement and corresponding reading characteristics more reliable and more robust. In addition, the device 10 can easily combine the 3D positioning of the portion of the head 27 with the gaze angle information obtained via the eye portion image analysis to identify a given screen object viewed by the user 25 at any given time. In some implementations, the use of 3D mapping in conjunction with gaze tracking allows the user 25 to freely move his or her head 27 and eyes 45 while reducing or eliminating the need to actively track the head 27 using sensors or transmitters on the head 27.

[0077] Some specific implementations obtain physiological data and other user information to help improve user experience. In such processes, as an example, user preferences and privacy should be respected by ensuring that the user understands and agrees to the use of user data, understands what type of user data is used, controls the collection and use of user data, and limits the distribution of user data (e.g., by ensuring that user data is processed locally on the user's device). The user should have the option of opting in or opting out of any feature part of whether to obtain or use its user data or otherwise turn on and off the acquisition or use of user information. In addition, each user will have the ability to access and otherwise find out anything about him or her that the system has collected or determined. User data is securely stored on the user's device. User data used as input for machine learning models is securely stored on the user's device, for example, to ensure the user's privacy. The user's device may have a secure storage area (e.g., a secure partition) for protecting certain user information, such as data from image sensors and other sensors for facial identification or biometric identification. User data associated with the user's body and / or attention state may be stored in such a secure partition, thereby limiting access to user data and limiting the transmission of user data to other devices to ensure that user data remains securely on the user's device. User data may be prohibited from leaving the user device and may be used only in machine learning models and other processes on the user device.

[0078] At box 820, method 800 identifies one or more of the objects in the environment based on determining scene understanding of the environment. For example, in a physical environment, the scene understanding instruction set identifies one or more of the objects in the environment based on obtaining sensor data from one or more sensors corresponding to the environment. For example, in a virtual environment, the scene understanding instruction set identifies one or more of the objects in the environment based on presented content. In some specific implementations, identifying one or more of the objects may include determining the location of each object based on an image of the user's current room, depth data, etc. Additionally, in some specific implementations, the positioning, speed, and direction of movement of objects in the scene may be determined as part of the scene understanding of the environment.

[0079] In some implementations, the physical environment includes one or more objects, and determining scene understanding of the physical environment includes determining the location, speed, or direction of motion of one or more objects. For example, determining scene understanding may include identifying one or more objects and their positions based on an image of the user's current room, depth data, etc. (including the location, speed, direction of motion of objects in the scene / environment). In some implementations, determining scene understanding includes generating a three-dimensional (3D) representation of the physical environment. For example, after the scanning process, a 3D model, such as a 3D point cloud or a 3D rendering, may be generated for the physical environment and / or objects within the physical environment.

[0080] In some aspects, method 800 determines the context of the experience based on sensor data of the environment. For example, determining the context may involve using computer vision to generate scene understanding of visual and / or auditory attributes of the environment—where the user is, what the user is doing, what objects are nearby. Additionally, scene understanding of content presented to the user may be generated, including visual and / or auditory attributes of the content the user is viewing.

[0081] In some aspects, different contexts of the presented content and environment are analyzed to determine where the user is, what the user is doing, what objects or people in the environment or within the content are nearby, and what the user did earlier. Additionally, the context analysis may include image analysis (semantic segmentation), audio analysis (vibration sound), positioning sensors (where the user is), motion sensors (fast-moving vehicles), and even access to other user data (e.g., the user's calendar). In an exemplary implementation, method 800 may also include: determining the context of the experience by generating a scene understanding of the environment based on sensor data of the environment, the scene understanding including visual or auditory attributes of the environment; and determining the context of the experience based on the scene understanding of the environment.

[0082] In some implementations, the sensor data includes image data, and generating scene understanding is based at least on performing semantic segmentation of the image data and detecting one or more objects within the environment based on the semantic segmentation. In some implementations, determining the context of the experience includes determining an activity of a user based on scene understanding of the environment. In some implementations, the sensor data includes location data of the user, and determining the context of the experience includes determining a location of the user within the environment based on the location data.

[0083] At block 830, method 800 determines features associated with interaction events for one or more objects based on physiological data and scene understanding. For example, determining features associated with interaction events for one or more objects may include dividing objects into salient / non-salient areas, text-related objects and non-text-related objects, etc. In some implementations, features associated with interaction events for one or more objects may include time spent fixating on an object, number of revisits to the same object, average change in pupil diameter across different visits, blinks during different visits, scan amplitude / speed when exploring an object, etc.

[0084] In some implementations, interactions over time are used to determine features that are input into a machine learning algorithm that is trained to predict subsequent factual memory. In some implementations, features are identified based on user experiments for multiple different users, which include user instructions to focus on a specific object in one or more objects during a study period, wherein physiological data is obtained for multiple different users during the study period. For example, a user study / experiment may include having a user focus on a specific phrase or part of an image to be remembered, and when the user is focusing his or her attention on the phrase (or part of an image), physiological data can be collected and determined as features of the interaction event. In some implementations, the determined features include at least one of the average change in pupil diameter during a first time period, blinking during different subsequent time periods, or scanning amplitude / speed when exploring one or more objects during a first time period. For example, some of these features (e.g., machine learning inputs) may include time spent fixating on an object, number of revisits to the same object, average change in pupil diameter across different visits, blinks during different visits, saccade amplitude / speed when exploring an object, etc.

[0085] In some embodiments, determining features may include determining features associated with an interaction event with one or more objects based on determining salient regions and non-salient regions associated with the one or more objects (e.g., salient region 550) and determining gaze intersections with the salient regions or the non-salient regions (e.g., determining that a user focuses his or her gaze on salient region 550).

[0086] In some implementations, determining features associated with an interaction event for one or more objects includes determining a scene-induced pupil response change characteristic for a particular object among the one or more objects, and determining the interaction event based on the scene-induced pupil response change characteristic for the particular object. For example, when a user's gaze intersects an object or user interface element, a machine learning model can be trained to predict a "click" or "no click" at each time point based on the presence of an attention-induced pupil response.

[0087] At block 840, method 800 determines a retention state during a first time period based on the features, the retention state associated with retention of a concept associated with an object in the one or more objects. For example, the determined features may be input into a machine learning algorithm trained to predict subsequent fact memory, such that one or more gaze, head pose, and hand gesture features may be determined, aggregated, and used to classify the user's eye movements and retention state using machine learning techniques to identify specific interactions with virtual / real world objects that facilitate a learning / productivity experience.

[0088] In some implementations, the retention state can be based on evaluating transitions between multiple objects. For example, Figure 3 Diagram 300 provides an example illustration of how a user may transition between objects (relevant and irrelevant) in a learning experience. The utilization of tracking gaze transitions between relevant and irrelevant objects is unique to retention prediction because it requires custom annotations but only a single simple binary label from the content developer (e.g., determining whether an object is relevant to the learning experience), and uses this information to derive structural measures in gaze patterns over time.

[0089] In some implementations, the retention state during the first time period is based on using a machine learning classifier model, and the physiological data is input into the machine learning classification model to identify the retention of concepts associated with the object. For example, the features are input into a machine learning algorithm (e.g., a trained neural network) to predict subsequent fact memory. In some implementations, the machine learning classification model is trained based on a plurality of eye characteristics, head posture characteristics, and hand gesture characteristics.

[0090] At box 850, method 800 provides feedback based on identifying the retention state. In some specific implementations, feedback can be provided directly to the user. Additionally or alternatively, in some specific implementations, feedback is provided to the content developer based on identifying the retention state of the user. The feedback data may include physiological data, contextual data, data associated with user interaction with the content (e.g., user clicks on a specific icon, etc.). For example, performance feedback is provided during the learning experience, which can enhance the user's experience, provide additional benefits from the session, and provide users with guidance and supportive teaching methods (e.g., scaffolding teaching methods) to promote their retention (e.g., feedback content elements 660, notification text bubbles, or arrows pointing out relevant selections). Additionally, for example, relevance can be determined for an event (e.g., eye movement toward a relevant object), and then the user interface (e.g., content instruction set 710) executes a response, such as a user "clicking" on a user interface element (e.g., some objects are distracting and can be removed, such as bystanders 620, 622 in Figure 6). When the algorithm detects a higher likelihood of making a non-relevant transition, the content may be updated based on the feedback data to reduce the salience of the non-relevant object (eg, no longer relevant after a period of time).

[0091] In some implementations, feedback can be provided directly to the user by updating a view of a display of the electronic device during a first time period. In some implementations, updating the view of the display is based on physiological data (e.g., based on head / eye movement, the user interface performs an action). In some implementations, providing feedback based on identifying the retention state includes providing a quality level of retention of concepts associated with the object, and providing notifications based on concepts associated with the object. For example, providing more explicit feedback about what the user may have missed, and redirecting the user to certain learning segments. Additionally, the system can suggest optimally revisiting certain objects, suggesting more engagement / exploration to have more interactions with specific (related) objects.

[0092] In some implementations, providing feedback based on identifying retention states includes adjusting content in response to determining interaction events associated with retention of concepts associated with an object. For example, content may be adapted to increase in difficulty (e.g., more information to view) as the learner progresses (e.g., higher retention makes the course more challenging, but lower retention makes the course easier / clearer). In some implementations, feedback may include encouraging users to revisit specific objects, such as all related (key) objects. Feedback may include encouraging more relevant transitions by optimally distributing objects in 3D space. In some implementations, feedback data 752 may allow the content instruction set 710 or content creator to improve user-object interactions with related objects that will increase recall. For example, instead of creating a read-only text object, make it multimodal using voice. Additionally or alternatively, color may be added to a read-only text object to highlight specific relevant portions and / or categorize information (e.g., highlight specific areas of text, such as Figure 6C Feedback content element 672 in .

[0093] In some implementations, method 800 further includes: determining multiple retention states during multiple time periods based on the features; aggregating the determined features associated with the multiple retention states associated with multiple objects in one or more objects; and updating feedback based on the aggregated features. For example, aggregating their attention and behavior measurements and letting the user know their most / least favorite topics, suggesting a break from the environment and moving to a more relaxing environment, aggregating measurements to inform the user of their most productive days of the week and the time of day (morning versus evening, etc.), suggesting different times of the day be used for different types of productive activities. For example, the user may be encouraged to do heavy memory tasks in the morning and problem solving in the afternoon.

[0094] Fig. 9is a block diagram of an example device 900. Device 900 illustrates an exemplary device configuration for device 10. While certain specific features are illustrated, those skilled in the art will appreciate from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 10 includes one or more processing units 902 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 906, one or more communication interfaces 908 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, one or more displays 912, one or more internal-facing and / or external-facing image sensor systems 914, memory 920, and one or more communication buses 904 for interconnecting these and various other components.

[0095] In some implementations, one or more communication buses 904 include circuits that interconnect system components and control communications between system components. In some implementations, one or more I / O devices and sensors 906 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0096] In some implementations, one or more displays 912 are configured to present a view of a physical or graphical environment to a user. In some implementations, one or more displays 912 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 912 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, device 10 includes a single display. In another example, device 10 includes a display for each eye of the user.

[0097] In some implementations, the one or more image sensor systems 914 are configured to obtain image data corresponding to at least a portion of the physical environment 105. For example, the one or more image sensor systems 914 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 914 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 914 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0098] The memory 920 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 920 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 920 optionally includes one or more storage devices remotely located from the one or more processing units 902. The memory 920 includes a non-transitory computer-readable storage medium.

[0099] In some implementations, the memory 920 or a non-transitory computer-readable storage medium of the memory 920 stores an optional operating system 930 and one or more instruction sets 940. The operating system 930 includes procedures for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 940 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 940 is software that can be executed by one or more processing units 902 to implement one or more of the techniques described herein.

[0100] The instruction set 940 includes a content instruction set 942, a physiological tracking instruction set 944, a scene understanding instruction set 946, and a retention evaluation and feedback instruction set 948. The instruction set 940 may be embodied as a single software executable file or as a plurality of software executable files.

[0101] In some implementations, the content instruction set 942 may be executed by the processing unit 902 to provide and / or track content for display on the device. The content instruction set 942 may be configured to monitor and track content over time (e.g., during an experience) and / or identify change events that occur within the content (e.g., based on identified / classified behavioral gaze events). In some implementations, the content instruction set 942 may be configured to add change events to the content (e.g., a feedback mechanism) using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0102] In some implementations, the physiological tracking instruction set 944 can be executed by the processing unit 902 to track physiological attributes of the user (e.g., EEG amplitude / frequency, pupil modulation, gaze, head posture / positioning data, heart rate, EDA data, etc.) using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0103] In some implementations, scene understanding instruction set 946 can be executed by processing unit 902 to determine the context of an experience and / or environment (e.g., creating scene understanding to determine objects or people in content or in an environment, where a user is, what a user is looking at, etc.) using one or more of the techniques discussed herein (e.g., object detection, facial recognition, etc.) or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0104] In some implementations, the retention evaluation and feedback instruction set 948 can be executed by the processing unit 902 to evaluate physiological data (e.g., eye data, head data, hand gesture data, etc.) to determine a retention state associated with the retention of a concept, and provide feedback based on the evaluation using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0105] Although the instruction set 940 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Fig. 9More as a functional description of the various features present in a particular implementation, as distinct from the structural schematics of the implementations described herein. As will be appreciated by one of ordinary skill in the art, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how the features are distributed among them will vary depending on the implementation, and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0106] Fig.10 A block diagram of an exemplary head mounted device 1000 according to some implementations is illustrated. The head mounted device 1000 includes a housing 1001 (or shell) that houses various components of the head mounted device 1000. The housing 1001 includes (or is coupled to) eye pads (not shown) disposed at the proximal (of the user 25) end of the housing 1001. In various implementations, the eye pads are plastic or rubber pieces that comfortably and snugly hold the head mounted device 1000 in place on the face of the user 25 (e.g., around the eyes of the user 25).

[0107] Housing 1001 houses display 1010, which displays images, emits light toward or onto the eyes of user 25. In various implementations, display 1010 emits light through an eyepiece having one or more optical elements 1005, which refract the light emitted by display 1010, so that the display appears to user 25 as a virtual distance farther than the actual distance from the eye to display 1010. For example, optical element 1005 may include one or more lenses, waveguides, other diffractive optical elements (DOEs), etc. To enable user 25 to focus on display 1010, in various implementations, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). In addition, to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

[0108] Housing 1001 also houses a tracking system that includes one or more light sources 1022, camera 1024, camera 1032, camera 1034, and controller 1080. One or more light sources 1022 emit light onto the eyes of user 25, which is reflected as a light pattern (e.g., a flashing circle) that can be detected by camera 1024. Based on the light pattern, controller 1080 can determine the eye tracking characteristics of user 25. For example, controller 1080 can determine the gaze direction and / or blink state (eyes open or closed) of user 25. As another example, controller 1080 can determine the pupil center, pupil size, or focus point. Therefore, in various specific implementations, light is emitted by one or more light sources 1022, reflected from the eyes of user 25, and detected by camera 1024. In various specific implementations, light from the eyes of user 25 reflects from a hot mirror or passes through an eyepiece before reaching camera 1024.

[0109] The display 1010 emits light within a first wavelength range, and the one or more light sources 1022 emit light within a second wavelength range. Similarly, the camera 1024 detects light within the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range of approximately 400 nm-700 nm within the visible spectrum), and the second wavelength range is a near infrared wavelength range (e.g., a wavelength range of approximately 700 nm-1400 nm within the near infrared spectrum).

[0110] In various implementations, eye tracking (or specifically, determined gaze direction) is used to enable user interaction (e.g., user 25 selecting an option on display 1010 by looking at it), provide foveated rendering (e.g., presenting a higher resolution in the area of ​​display 1010 that user 25 is looking at and a lower resolution elsewhere on display 1010), or correct distortion (e.g., for images to be provided on display 1010).

[0111] In various implementations, one or more light sources 1022 emit light toward the eyes of user 25, which is reflected in the form of multiple flashes.

[0112] In various implementations, camera 1024 is a frame / shutter based camera that generates images of the eyes of user 25 at a particular point in time or multiple points in time at a frame rate. Each image includes a matrix of pixel values ​​corresponding to pixels of the image, the pixels corresponding to the locations of the camera's light sensor matrix. In a specific implementation, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0113] In various implementations, camera 1024 is an event camera comprising multiple light sensors (e.g., a light sensor matrix) at multiple corresponding positions, which generates an event message indicating a specific position of a particular light sensor in response to the particular light sensor detecting a change in light intensity.

[0114] In various implementations, camera 1032 and camera 1034 are frame / shutter based cameras that can generate images of the face of user 25 at a specific point in time or multiple points in time at a frame rate. For example, camera 1032 captures an image of the user's face below the eyes, and camera 1034 captures an image of the user's face above the eyes. The images captured by camera 1032 and camera 1034 may include light intensity images (e.g., RGB) and / or depth image data (e.g., time of flight, infrared, etc.).

[0115] According to some specific implementations, an electronic device (e.g., device 10) may generate an extended reality (XR) environment during a communication session and present the XR environment to one or more users. An extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic device, compared to a physical environment that people can sense and / or interact with without the help of an electronic device. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, an XR system may detect head movement, and in response, adjust the graphics content and sound field presented to a person in a manner similar to how such views and sounds change in a physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, a tablet computer, a laptop computer, etc.) that presents an XR environment, and in response, adjust the graphics content and sound field presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (eg, for accessibility reasons), the XR system may adjust characteristics of graphical content in the XR environment in response to representations of physical movement (eg, voice commands).

[0116] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display may be configured to selectively become opaque. A projection-based system may employ retinal projection technology that projects a graphic image onto a person's retina. The projection system may also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0117] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0118] As described above, one aspect of the present technology is to collect and use physiological data to improve the user's electronic device experience in interacting with electronic content. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0119] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the interaction and control capabilities of electronic devices. Therefore, the use of such personal information data enables planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0120] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other uses of such personal information and / or physiological data will comply with established privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to the use of privacy policies and measures that are recognized as meeting or exceeding the industry or government requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of the entity, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after receiving the informed consent of the user. Additionally, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and measures.

[0121] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively prevent the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.

[0122] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0123] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identification and / or without personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access his or her stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access the data stored therein.

[0124] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, one skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0125] Unless otherwise specifically noted, it should be understood that throughout this specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0126] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0127] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0128] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. Additionally, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for ease of explanation only and are not intended to be limiting.

[0129] It will also be understood that, although the terms "first", "second", etc. may be used to describe various objects in this article, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0130] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, objects or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, parts or their groupings.

[0131] As used herein, the term "if" may be interpreted to mean "when the antecedent is true" or "when the antecedent is true" or "in response to determining" or "upon determining" or "in response to detecting" that the antecedent is true, depending on the context. Similarly, the phrase "if it is determined that [the antecedent is true]" or "if [the antecedent is true]" or "when [the antecedent is true]" is interpreted to mean "upon determining that the antecedent is true" or "in response to determining" or "upon determining" that the antecedent is true or "when detecting that the antecedent is true" or "in response to detecting" that the antecedent is true, depending on the context.

[0132] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method, comprising: At an electronic device having a processor and one or more sensors: obtaining physiological data in an environment during a first time period using the one or more sensors, the environment including one or more objects; identifying the one or more of the objects in the environment based on determining scene understanding of the environment; determining features associated with interaction events with the one or more objects based on the physiological data and the scene understanding; determining, based on the feature, a retention status during the first time period, the retention status associated with retention of a concept associated with an object of the one or more objects; as well as Feedback is provided based on identifying the reservation status. 2 . The method of claim 1 , wherein the physiological data comprises position information and velocity information for a left eye and a right eye. 3 . The method according to claim 1 , wherein the physiological data comprises an interpupillary distance (IPD) between a right eye and a left eye, and a diameter of the left eye and a diameter of the right eye.

4. The method according to any one of claims 1 to 3, wherein the physiological data comprises: Gaze direction; fixation speed; or The pupil is fixed.

5. The method according to any one of claims 1 to 4, wherein the physiological data comprises an image of the eye or electrooculogram (EOG) data.

6. The method of any one of claims 1 to 5, wherein the physiological data comprises images of the head from the one or more sensors.

7. The method according to any one of claims 1 to 6, wherein obtaining the physiological data comprises tracking the position and movement of the head.

8. The method of any one of claims 1 to 7, wherein determining the scene understanding of a physical environment comprises determining a position, a velocity, or a direction of motion of at least one of the one or more objects.

9. The method of any one of claims 1 to 8, wherein determining the scene understanding comprises generating a three-dimensional (3D) representation of the physical environment.

10. The method according to any one of claims 1 to 9, wherein the scene understanding is based on obtaining sensor data from the one or more sensors corresponding to a physical environment.

11. The method of any one of claims 1 to 10, wherein the one or more objects include a plurality of objects, and wherein determining the retention status during the first time period is based on evaluating transitions between the plurality of objects.

12. The method according to any one of claims 1 to 11, wherein the determined features include at least one of the following: the time spent fixated on the object or objects; the number of revisits for the subject or subjects; an average change in pupil diameter during said first time period; blinks during different follow-up time periods; or Amplitude / speed of glances while exploring the one or more objects during the first time period.

13. A method according to any one of claims 1 to 12, wherein the feature is identified based on a user experiment for multiple different users, the user experiment including user instructions to focus on a specific object among the one or more objects during a study time period, wherein physiological data is obtained for multiple different users during the study time period.

14. The method of any one of claims 1 to 13, wherein determining the features associated with the interaction event with the one or more objects is based on: determining salient regions and non-salient regions associated with the one or more objects; and Determine fixation intersections with salient or non-salient regions.

15. The method of any one of claims 1 to 14, wherein determining features associated with interaction events with the one or more objects comprises: determining a characteristic of a scene-induced pupillary response change for a particular object of the one or more objects; as well as An interaction event is determined based on a pupil response change characteristic caused by the scene for the specific object.

16. The method of any one of claims 1 to 15, wherein providing the feedback based on identifying the reservation status comprises: A view of a display of the electronic device is updated during the first time period. The method of claim 16 , wherein updating the view of the display is based on the physiological data.

18. The method of any one of claims 1 to 17, wherein providing the feedback based on identifying the reservation status comprises: Content is adjusted in response to determining an interaction event associated with the retention of the concept associated with the object.

19. The method according to any one of claims 1 to 18, providing the feedback based on identifying the reservation status comprises: providing a quality level of retention of said concepts associated with said object; as well as Notification is provided based on the concept associated with the object.

20. The method according to any one of claims 1 to 19, further comprising: determining a plurality of retention states during a plurality of time periods based on the characteristics; aggregating the determined features associated with the plurality of retention states associated with a plurality of objects among the one or more objects; as well as The feedback is updated based on the aggregated features.

21. A method according to any one of claims 1 to 20, wherein the retention status during the first time period is based on using a machine learning classifier model, wherein the physiological data is input into the machine learning classification model to identify the retention of the concept associated with the object.

22. The method of claim 21, wherein the machine learning classification model is trained based on a plurality of eye characteristics, head posture characteristics, and hand gesture characteristics.

23. The method of any one of claims 1 to 22, wherein the physiological data is obtained from sensor data from the one or more sensors, and wherein the sensor data includes depth data and light intensity image data obtained during a scanning process.

24. The method of any one of claims 1 to 23, wherein the environment is a physical environment.

25. The method of any one of claims 1 to 23, wherein the environment is a virtual environment.

26. The method of any one of claims 1 to 23, wherein the environment is a mixed reality (MR) experience.

27. The method of any one of claims 1 to 26, wherein the electronic device is a head mounted device (HMD).

28. A device, comprising: one or more sensors; a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining physiological data in an environment during a first time period using the one or more sensors, the environment including one or more objects; identifying the one or more of the objects in the environment based on determining scene understanding of the environment; determining features associated with interaction events with the one or more objects based on the physiological data and the scene understanding; determining, based on the feature, a retention status during the first time period, the retention status associated with retention of a concept associated with an object of the one or more objects; and Feedback is provided based on identifying the reservation status.

29. The apparatus of claim 28, wherein the physiological data includes position information and velocity information for a left eye and a right eye.

30. The apparatus according to claim 28 or 29, wherein the physiological data comprises an interpupillary distance (IPD) between a right eye and a left eye, and a diameter of the left eye and a diameter of the right eye.

31. The apparatus of any one of claims 28 to 30, wherein the physiological data comprises: Gaze direction; fixation speed; or The pupil is fixed.

32. An apparatus according to any one of claims 28 to 31, wherein the physiological data comprises an image of the eye or electrooculogram (EOG) data.

33. An apparatus according to any one of claims 28 to 32, wherein the physiological data comprises an image of the head from the one or more sensors.

34. Apparatus according to any one of claims 28 to 33, wherein obtaining the physiological data comprises tracking position and movement of the head.

35. The apparatus of any one of claims 28 to 34, wherein determining the scene understanding of the physical environment comprises determining a position, velocity, or direction of motion of at least one of the one or more objects.

36. The apparatus of any one of claims 28 to 35, wherein determining the scene understanding comprises generating a three-dimensional (3D) representation of the physical environment.

37. An apparatus according to any one of claims 28 to 36, wherein the scene understanding is based on obtaining sensor data from the one or more sensors corresponding to a physical environment.

38. The apparatus of any one of claims 28 to 37, wherein the one or more objects include a plurality of objects, and wherein determining the retention status during the first time period is based on evaluating transitions between the plurality of objects.

39. The apparatus of any one of claims 28 to 38, wherein the determined characteristics include at least one of: the time spent fixated on the object or objects; the number of revisits for the subject or subjects; an average change in pupil diameter during said first time period; blinks during different follow-up time periods; or Amplitude / speed of glances while exploring the one or more objects during the first time period.

40. An apparatus according to any one of claims 28 to 39, wherein the feature is identified based on user experiments for multiple different users, the user experiments including user instructions to focus on a specific object among the one or more objects during a study time period, wherein physiological data is obtained for multiple different users during the study time period.

41. The apparatus of any one of claims 28 to 40, wherein determining the characteristics associated with the interaction event with the one or more objects is based on: determining salient regions and non-salient regions associated with the one or more objects; and Determine fixation intersections with salient or non-salient regions.

42. The apparatus of any one of claims 28 to 41, wherein determining characteristics associated with the interaction event with the one or more objects comprises: determining a characteristic of a scene-induced pupillary response change for a particular object of the one or more objects; as well as An interaction event is determined based on a pupil response change characteristic caused by the scene for the specific object.

43. The apparatus of any one of claims 28 to 42, wherein providing the feedback based on identifying the reservation status comprises: A view of a display of the electronic device is updated during the first time period.

44. The apparatus of claim 43, wherein updating the view of the display is based on the physiological data.

45. The apparatus of any one of claims 28 to 44, wherein providing the feedback based on identifying the reservation status comprises: Content is adjusted in response to determining an interaction event associated with the retention of the concept associated with the object.

46. ​​The apparatus of any one of claims 28 to 45, providing the feedback based on identifying the reservation status comprises: providing a quality level of retention of said concepts associated with said object; as well as Notification is provided based on the concept associated with the object.

47. The apparatus of any one of claims 28 to 46, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, further cause the one or more processors to perform operations comprising: determining a plurality of retention states during a plurality of time periods based on the characteristics; aggregating the determined features associated with the plurality of retention states associated with a plurality of objects among the one or more objects; as well as The feedback is updated based on the aggregated features.

48. An apparatus according to any one of claims 28 to 47, wherein the retention state during the first time period is based on use of a machine learning classifier model, wherein the physiological data is input into the machine learning classification model to identify the retention of the concept associated with the object.

49. The apparatus of claim 48, wherein the machine learning classification model is trained based on a plurality of eye characteristics, head posture characteristics, and hand gesture characteristics.

50. An apparatus according to any one of claims 28 to 49, wherein the physiological data is obtained from sensor data from the one or more sensors, and wherein the sensor data includes depth data and light intensity image data obtained during a scanning process.

51. Apparatus according to any one of claims 28 to 50, wherein the environment is a physical environment.

52. An apparatus as claimed in any one of claims 28 to 50, wherein the environment is a virtual environment.

53. An apparatus as claimed in any one of claims 28 to 50, wherein the environment is a mixed reality (MR) experience.

54. The device of any one of claims 28 to 53, wherein the electronic device is a head mounted device (HMD).

55. A non-transitory computer-readable storage medium storing program instructions executable by one or more processors on a device to perform operations comprising: obtaining physiological data in an environment during a first time period using one or more sensors, the environment including one or more objects; identifying the one or more of the objects in the environment based on determining scene understanding of the environment; determining features associated with interaction events with the one or more objects based on the physiological data and the scene understanding; determining, based on the feature, a retention status during the first time period, the retention status associated with retention of a concept associated with an object of the one or more objects; as well as Feedback is provided based on identifying the reservation status.

56. The non-transitory computer-readable storage medium of claim 55, wherein the physiological data includes position information and velocity information for a left eye and a right eye.

57. The non-transitory computer-readable storage medium of claim 55 or 56, wherein the physiological data comprises an interpupillary distance (IPD) between a right eye and a left eye, and a diameter of the left eye and a diameter of the right eye.

58. The non-transitory computer-readable storage medium of any one of claims 55 to 57, wherein the physiological data comprises: Gaze direction; fixation speed; or The pupil is fixed.

59. The non-transitory computer-readable storage medium of any one of claims 55 to 58, wherein the physiological data comprises an image of an eye or electro-oculogram (EOG) data.

60. The non-transitory computer-readable storage medium of any one of claims 55 to 59, wherein the physiological data comprises an image of the head from the one or more sensors.

61. The non-transitory computer-readable storage medium of any one of claims 55 to 60, wherein obtaining the physiological data comprises tracking position and movement of the head.

62. The non-transitory computer-readable storage medium of any one of claims 55 to 61, wherein determining the scene understanding of the physical environment comprises determining a position, velocity, or direction of motion of at least one of the one or more objects.

63. The non-transitory computer-readable storage medium of any one of claims 55 to 62, wherein determining the scene understanding comprises generating a three-dimensional (3D) representation of the physical environment.

64. The non-transitory computer-readable storage medium of any one of claims 55 to 63, wherein the scene understanding is based on obtaining sensor data from the one or more sensors corresponding to a physical environment.

65. The non-transitory computer-readable storage medium of any one of claims 55 to 64, wherein the one or more objects include a plurality of objects, and wherein determining the retention status during the first time period is based on evaluating transitions between the plurality of objects.

66. The non-transitory computer-readable storage medium of any one of claims 55 to 65, wherein the determined characteristics include at least one of the following: the time spent fixated on the object or objects; the number of revisits for the subject or subjects; an average change in pupil diameter during said first time period; blinks during different follow-up time periods; or Amplitude / speed of glances while exploring the one or more objects during the first time period.

67. A non-transitory computer-readable storage medium according to any one of claims 55 to 66, wherein the feature is identified based on user experiments for multiple different users, the user experiments including user instructions to focus on a specific object among the one or more objects during a study time period, wherein physiological data is obtained for multiple different users during the study time period.

68. The non-transitory computer-readable storage medium of any one of claims 55 to 67, wherein determining the characteristics associated with the interaction event for the one or more objects is based on: determining salient regions and non-salient regions associated with the one or more objects; and Determine fixation intersections with salient or non-salient regions.

69. The non-transitory computer-readable storage medium of any one of claims 55 to 68, wherein determining characteristics associated with the interaction event with the one or more objects comprises: determining a characteristic of a scene-induced pupillary response change for a particular object of the one or more objects; as well as An interaction event is determined based on a pupil response change characteristic caused by the scene for the specific object.

70. The non-transitory computer-readable storage medium of any one of claims 55 to 69, wherein providing the feedback based on identifying the reservation status comprises: A view of a display of the electronic device is updated during the first time period.

71. The non-transitory computer-readable storage medium of claim 70, wherein updating the view of the display is based on the physiological data.

72. The non-transitory computer-readable storage medium of any one of claims 55 to 71, wherein providing the feedback based on identifying the reservation status comprises: Content is adjusted in response to determining an interaction event associated with the retention of the concept associated with the object.

73. The non-transitory computer-readable storage medium of any one of claims 55 to 72, providing the feedback based on identifying the reservation status comprises: providing a quality level of retention of said concepts associated with said object; as well as Notification is provided based on the concept associated with the object.

74. The non-transitory computer-readable storage medium of any one of claims 55 to 73, wherein the program instructions are executable by the one or more processors on the device and cause the one or more processors to perform operations comprising: determining a plurality of retention states during a plurality of time periods based on the characteristics; aggregating the determined features associated with the plurality of retention states associated with a plurality of objects among the one or more objects; as well as The feedback is updated based on the aggregated features.

75. A non-transitory computer-readable storage medium according to any one of claims 55 to 74, wherein the retention state during the first time period is based on use of a machine learning classifier model, wherein the physiological data is input into the machine learning classification model to identify the retention of the concept associated with the object.

76. The non-transitory computer-readable storage medium of claim 75, wherein the machine learning classification model is trained based on a plurality of eye characteristics, head posture characteristics, and hand gesture characteristics.

77. A non-transitory computer-readable storage medium according to any one of claims 55 to 76, wherein the physiological data is obtained from sensor data from the one or more sensors, and wherein the sensor data includes depth data and light intensity image data obtained during a scanning process.

78. The non-transitory computer-readable storage medium of any one of claims 55 to 77, wherein the environment is a physical environment.

79. The non-transitory computer-readable storage medium of any one of claims 55 to 77, wherein the environment is a virtual environment.

80. The non-transitory computer-readable storage medium of any one of claims 55 to 77, wherein the environment is a mixed reality (MR) experience.

81. The non-transitory computer-readable storage medium of any one of claims 55 to 80, wherein the electronic device is a head mounted device (HMD).

Images