Gaze behavior detection

Abstract

Various implementations disclosed herein include devices, systems, and methods of determining a gaze behavior state based on physiological data to identify a user's gaze transfer event, gaze hold event, and loss event. For example, an example process may include obtaining eye data (e.g., eye position and velocity, pupil distance, pupil diameter, etc.) associated with a gaze during a first time period. The process may also include obtaining head data (e.g., head position and velocity) associated with the gaze during the first time period. The process may also include determining a first gaze behavior state during the first time period to identify a gaze transfer event, a gaze hold event, and a loss event (e.g., one or more gaze and head pose characteristics may be determined, aggregated, and used to classify eye movement states of the user using machine learning techniques).

Description

Technical Field

[0001] The present disclosure relates generally to electronic devices, and in particular to systems, methods, and devices for determining a user's gaze behavior state based on eye tracking information and head tracking information. Background Art

[0002] Determining the user's intent while viewing content on an electronic device can facilitate a more meaningful experience. For example, a portion of the content (e.g., a selectable icon or button) can be automatically selected based on determining the user's intent to make such a selection, without the user having to perform a gesture, mouse click, or other input device-based action to initiate the selection. However, users can make different types of eye movements, and eye movement frequency and characteristics can depend on many factors, including tasks, mental state, and body posture. Therefore, improved techniques for evaluating target eye movement events relative to the user's intent to view and interact with the content can enhance the user's enjoyment, understanding, and learning of the content. Content creators and systems may be able to provide a better and more customized user experience based on determining the user's intent to interact with portions of the content with greater accuracy. Summary of the invention

[0003] Various embodiments disclosed herein include devices, systems, and methods for providing a real-time gaze classification algorithm to classify eye movements and gaze behavior types based on eye tracking data (e.g., gaze direction, gaze angle, pupil diameter, pupil location, etc. obtained via a video-based eye tracker, retinal / fundus imaging-based eye tracking, electrooculogram-based eye tracking, magnetic scleral search coil-based eye tracking, etc.) and head representation / tracking data (e.g., head posture, rotation speed, etc. using image data, depth data, accelerometer data, magnetometer data, gyroscope data, etc.). Classifying gaze behavior into discrete "events" can be used to improve gaze-based user interface interactions, user intent prediction, and selective filtering of raw eye tracking data so that latency is not increased due to filtering operations. In some embodiments, scene understanding information can be used to determine object representation data in the current environment (e.g., location, depth distance from the user, speed, direction of movement of objects in the scene).

[0004] In some implementations, the techniques described herein can classify gaze behavior states in real time. For example, eye data and head posture data can be acquired at the current time and some history (e.g., the last 10 samples) to make predictions about gaze events at the current time with minimal delay.

[0005] Some specific implementations focus on improving the accuracy of determining gaze behavior states in real time based on eye tracking data and head posture data to identify gaze shift events, gaze hold events and loss events and determine the sample-by-sample event level for eye movement types. Eye movements occur all the time, and even when the user thinks his or her gaze is fixed on an object, small eye movements occur. The human eye is foveated, which means that a small part of the retina has a high spatial resolution and the resolution drops rapidly away from the fovea, so a person needs to move his or her eyes to use the highest sensitivity part of the retina to view an object. There are many different types of eye movements, and they can be classified in various ways according to their functional or physiological origins. Eye gaze behavior events or eye movements are saccades (brief, rapid, fast, jerky and ballistic eye movements to search / analyze scenes and bring the high sensitivity portion of the retina to objects of interest), steady pursuit (tracking an object while it moves to keep the high sensitivity portion of the retina on the object of interest), vestibulo-ocular reflex (VOR) (which is a compensatory movement of the eye to keep the line of sight fixed on an object, regardless of head movement / rotation), gaze (the line of sight and the head are relatively still, with only drift), blink / wink (one or both eyes are completely or partially closed), data loss (wherein eye tracking or head tracking fails for various reasons), etc. For example, a more advanced classification may also be performed, wherein the gaze behavior is classified into a rapid behavior state, a stable behavior state, and a lost behavior state. A rapid behavior state may refer to a saccade. A stable behavior state may refer to a movement (e.g., a steady pursuit, VOR, or gaze event) in which a user may be trying to stabilize his or her gaze on an object. Lost behavior states may refer to eye blinks, eye blinks, and / or other data loss that may be determined as a lost behavior state due to other issues with eye / head tracking (e.g., dropped frames, segmentation issues, outside of the tracking range of a particular eye tracking technology, etc.).

[0006] Humans make different types of eye movements, but the frequency and characteristics of these eye movements depend on many factors, including tasks, mental states, body posture, etc. For example, lying down, sitting, standing, and walking produce significantly different eye / head movement behaviors, which is due to the main contributing factors of head static gaze characteristics vs. head free gaze characteristics and head straight gaze characteristics vs. head tilted gaze characteristics. Therefore, the general problem solved herein is to create a classification algorithm that interprets eye / head tracking data on a device in real time, independent of the user's body posture or task. In some specific implementations, a machine learning algorithm or another type of classification algorithm may be implemented to classify eye data and head tracking data into behaviorally and / or functionally related events. In some specific implementations, the algorithm and / or machine learning model may be configured / improved based on user-specific actions (e.g., learned from user-specific actions). The target event may be as low as gaze, steady pursuit, saccade, VOR, blink, blink, data loss, etc., or just a higher level such as stable gaze state, rapid gaze state, and loss state. Eye and head tracking data can be used to train machine learning classifiers when users perform various tasks in various body / head postures (such as sitting, standing, lying down, walking, etc.). Depending on the body posture and task requirements, gaze behavior and statistics can vary significantly. Therefore, the gaze classification algorithm presented in this article is posture-independent and task-independent, and is trained using eye data and head data from various scenes. In addition to body posture, different tasks can also be represented in the training data, such as reading, watching, playing games, searching for objects, talking to someone, interacting with user interface elements, etc. (e.g., scene understanding information).

[0007] Context can additionally be used to determine gaze behavior states to identify gaze shift events, gaze hold events, and / or loss events. For example, scene analysis of an experience can determine scene understanding of visual and / or auditory attributes associated with content presented to a user (e.g., what is presented in video content) and / or attributes associated with the user's environment (e.g., where the user is, what the user is doing, what objects are nearby). These attributes of both the presented content and the user's environment can improve the determination of the user's intent based on his or her eye movements and gaze behavior regarding interaction events (e.g., selecting an icon).

[0008] Physiological data such as EEG amplitude / frequency, sensor data corresponding to pupil modulation, sensor data corresponding to eye gaze glances, etc. may depend on the individual, the characteristics of the scene in front of him or her (e.g., video content), the properties of the physical environment surrounding the user (including the user's activities / movements). Physiological data can be obtained using a device with eye tracking technology (and other physiological sensors) while the user is performing a task. In some implementations, other sensors (such as electroencephalogram (EEG) sensors or electrodermal activity (EDA) sensors) can be used to obtain physiological data. Observing repeated measurements of physiological data for experience can give insights into the user's intentions based on his or her eye movements and gaze behavior.

[0009] In some embodiments, determining whether to initiate an interaction event (e.g., predicting whether a user intends to interact with a portion of the displayed content) may be based on characteristics of the user (e.g., physiological data) and the user's environment (e.g., a real-world physical environment, a virtual environment, or a combination of each). A device (e.g., a handheld device, a laptop computer, a desktop computer, or a head-mounted device (HMD)) provides an experience (e.g., a visual experience and / or an auditory experience) of a real-world physical environment or an extended reality (XR) environment. The device utilizes one or more sensors to obtain physiological data associated with the user (e.g., EEG amplitude, pupil modulation, eye gaze saccades, head movement measured by an inertial measurement unit (IMU), etc.). Based on the obtained physiological data, the technology described herein may determine an interaction event during the experience. Based on the physiological data and the associated physiological response (e.g., a user focusing on a specific area of ​​the content), the technology may provide a response to the user based on the interaction event and adjust the content corresponding to the experience.

[0010] Some implementations evaluate physiological data and other user information to help improve the user experience. In such processes, user preferences and privacy should be respected by, for example, ensuring that the user understands and consents to the use of user data, understanding what type of user data is used, controlling the collection and use of user data, and limiting the distribution of user data (e.g., by ensuring that user data is processed locally on the user's device). Users should have the option to opt-in or opt-out of whether their user data is obtained or used or otherwise turn on and off any features that obtain or use user information. In addition, each user should have the ability to access and otherwise find out anything that the system has collected or determined about him or her.

[0011] Generally speaking, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: at an electronic device having a processor and one or more sensors, obtaining eye data associated with gaze during a first time period, obtaining head data associated with gaze during the first time period, and determining a first gaze behavior state during the first time period based on the eye data and the head data to identify a gaze shift event, a gaze hold event, or a gaze loss event.

[0012] These and other embodiments can optionally include one or more of the following features.

[0013] In some aspects, the method further comprises an act of determining scene understanding of the physical environment based on obtaining sensor data from one or more sensors corresponding to the physical environment. In some aspects, determining the first gaze behavior state during the first time period is based on scene understanding of the physical environment.

[0014] In some aspects, the physical environment includes one or more objects, and wherein determining scene understanding of the physical environment includes determining a location, velocity, or direction of motion of the one or more objects. In some aspects, determining scene understanding includes generating a three-dimensional (3D) representation of the physical environment.

[0015] In some aspects, the eye data includes position information and speed information for the left eye and the right eye. In some aspects, the eye data includes the pupil distance (IPD) between the right eye and the left eye, and the diameter of the left eye and the diameter of the right eye. In some aspects, the eye data includes the gaze direction or the gaze speed. In some aspects, the eye data includes an image of the eye or electrooculogram (EOG) data.

[0016] In some aspects, the head data includes images of the head from one or more sensors. In some aspects, obtaining the head data includes tracking the pose and movement of the head.

[0017] In some aspects, determining a first gaze behavior state during the first time period is based on a set of data acquired before the first time period. In some aspects, the first gaze behavior state is a type of gaze behavior state in a plurality of gaze behavior states, wherein the plurality of gaze behavior states include a gaze hold state, a gaze shift state, and an eye tracking loss state.

[0018] In some aspects, the method further comprises an act of identifying a gaze shift event during the first time period based on determining that the first gaze behavior state includes a rapid gaze state during the first time period.

[0019] In some aspects, the method further comprises an act of identifying a gaze hold event during the first time period based on determining that the first gaze behavior state comprises a stable gaze state during the first time period.

[0020] In some aspects, the method further comprises an act of updating a view of a display of the electronic device during the first time period based on determining the first gaze behavior state during the first time period. In some aspects, updating the view of the display is based on eye data or head data.

[0021] In some respects, the first gaze behavior state during the first time period is based on using a machine learning classifier model, wherein eye data and head data are input into the machine learning classification model, to identify a gaze transfer event, a gaze retention event and a loss event. In some respects, the machine learning classification model is trained based on a plurality of body positions. In some respects, a plurality of body positions include lying position, standing position, sitting position, walking position or a combination thereof.

[0022] In some aspects, the eye data or head data is obtained from sensor data from one or more sensors, and wherein the sensor data includes depth data and light intensity image data obtained during a scanning process.

[0023] In some aspects, the electronic device is a head mounted device (HMD).

[0024] These and other embodiments can optionally include one or more of the following features.

[0025] According to some specific implementations, a non-transitory computer-readable storage medium stores instructions, which are computer-executable to perform or cause the performance of any method described herein. According to some specific implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0027] Figure 1 Devices for obtaining sensor data from a user according to some implementations are shown.

[0028] Figure 2A and Figure 2BDetecting eye movements and gaze behavior of a user viewing content based on physiological data according to some specific implementations is shown.

[0029] Figure 3 A classification framework for gaze behavior events according to some implementations is shown.

[0030] Figure 4 A diagram of a system for detecting gaze behavior events of a user viewing content based on physiological data according to some implementations is shown.

[0031] Figure 5 is a flowchart representation of a method for determining gaze behavior states based on physiological data to identify gaze shift events, gaze hold events, and gaze loss events of a user according to some specific implementations.

[0032] Figure 6 is a block diagram illustrating device components of an exemplary device according to some implementations.

[0033] Figure 7 is a block diagram of an example head mounted device (HMD) according to some specific implementations.

[0034] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0035] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the example implementations described herein.

[0036] Figure 1 An example environment 100 of a real-world environment 105 (e.g., a room) is shown that includes a device 10 having a display 15. In some implementations, the device 10 displays content 20 to a user 25. For example, the content 20 can be a button, a user interface icon, a text box, a graphic, an avatar of the user or another user, etc. In some implementations, the content 20 can occupy the entire display area of ​​the display 15.

[0037] The device 10 obtains image data, motion data, and / or physiological data (e.g., pupil data, facial feature data, etc.) from the user 25 via multiple sensors (e.g., sensors 35a, 35b, and 35c). For example, the device 10 obtains eye gaze characteristic data 40b via sensor 35b, upper facial feature characteristic data 40a via sensor 35a, and lower facial feature characteristic data 40c via sensor 35c. Additionally, each of the multiple sensors (e.g., sensors 35a, 35b, and 35c) may capture information about the position, location, motion, posture, etc. of the head 27 and / or body of the user 25.

[0038] Although this example and other examples discussed herein show a single device 10 in a real-world environment 105, the techniques disclosed herein are applicable to multiple devices and other real-world environments. For example, the functionality of device 10 may be performed by multiple devices, with sensors 35a, 35b, and 35c located on each respective device, or distributed among them in any combination.

[0039] In some implementations, the plurality of sensors (e.g., sensors 35a, 35b, and 35c) may include any number of sensors that acquire data related to the appearance of user 25. For example, when a head mounted device (HMD) is worn, one sensor (e.g., a camera within the HMD) may acquire pupil data for eye tracking, and one sensor on a separate device (e.g., one camera, such as a wide angle view) may be able to capture all of the user's facial feature data. Alternatively, if device 10 is an HMD, a separate device may not be necessary. For example, if device 10 is an HMD, in one embodiment, sensor 35b may be located within the HMD to capture pupil data (e.g., eye gaze characteristic data 40b), and additional sensors (e.g., sensors 35a and 35c) may be located on the HMD but on an outer surface of the HMD facing the user's head / face to capture facial feature data (e.g., upper facial feature characteristic data 40a via sensor 35a, and lower facial feature characteristic data 40c via sensor 35c) and capture information about the position, location, movement, posture, etc. of the head 27 and / or body of the user 25.

[0040] In some specific implementations, such as Figure 1As shown, device 10 is a handheld electronic device (e.g., a smart phone or a tablet computer). In some specific implementations, device 10 is a laptop or desktop computer. In some specific implementations, device 10 has a touchpad, and in some specific implementations, device 10 has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some specific implementations, electronic device 10 can be worn by a user. For example, electronic device 10 can be a watch, HMD, head-mounted device (glasses), headphones, ear-hook devices, etc. In some specific implementations, the functions of device 10 are implemented via two or more devices, such as a mobile device and a base station or a head-mounted device and an ear-hook device. Various functions can be distributed in multiple devices, including but not limited to power functions, CPU functions, GPU functions, storage functions, memory functions, visual content display functions, audio content production functions, etc. Multiple devices that can be used to implement the functions of electronic device 10 can communicate with each other via wired communication or wireless communication and / or via an intermediate device such as a playback session server.

[0041] In some implementations, device 10 includes an eye tracking system for detecting eye position and eye movement via eye gaze characteristic data 40b. For example, the eye tracking system may include one or more infrared (IR) light emitting diodes (LEDs), an eye tracking camera (e.g., a near IR (NIR) camera), and an illumination source (e.g., a NIR light source) that emits light (e.g., NIR light) toward the eyes of user 25. In addition, the illumination source of device 10 may emit NIR light to illuminate the eyes of user 25, and the NIR camera may capture images of the eyes of user 25. In some implementations, the images captured by the eye tracking system may be analyzed to detect the position and movement of the eyes of user 25, or to detect other information about the eyes such as color, shape, state (e.g., wide open, squinting, etc.), pupil dilation, or pupil diameter. In addition, the gaze point estimated from the eye tracking image may enable gaze-based interaction with content shown on a near-eye display of device 10.

[0042] In some implementations, the device 10 has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing multiple functions. In some implementations, the user 25 interacts with the GUI through hand gestures, finger contacts, and gestures on the touch-sensitive surface. In some implementations, these functions include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, and / or digital video playback. Executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0043] In some implementations, the device 10 employs various physiological sensors, detection or measurement systems. The detected physiological data may include, but is not limited to: electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG), functional near infrared spectroscopy signal (fNIRS), blood pressure, skin conductance, or pupil response. In addition, the device 10 may detect multiple forms of physiological data simultaneously in order to benefit from the simultaneous acquisition of physiological data. In addition, in some implementations, the physiological data represents involuntary data, that is, responses that are not under conscious control. For example, a pupil response may represent an involuntary movement.

[0044] In some implementations, one or both eyes 45 of the user 25 (including one or both pupils 50 of the user 25) present physiological data (e.g., eye gaze characteristic data 40b) in the form of pupil responses. The pupil responses of the user 25 result in changes in the size or diameter of the pupil 50 via the optic nerve and the oculomotor cranial nerve. For example, the pupil response may include a constriction response (miosis) (e.g., a narrowing of the pupil) or an expansion response (mydriasis) (e.g., a widening of the pupil). In some implementations, the device 10 may detect a pattern of physiological data representing a time-varying pupil diameter. In some implementations, the device 10 may also determine the pupil distance (IPD) between the right eye and the left eye of the user.

[0045] User data (e.g., upper facial feature characteristic data 40a, lower facial feature characteristic data 40c, and eye gaze characteristic data 40b), including information about the position, location, movement, posture, etc. of the head 27 and / or body of the user 25, may change over time, and the device 10 may use the user data to generate and / or provide a representation of the user.

[0046] In some implementations, the user data (e.g., upper facial feature characteristic data 40a and lower facial feature characteristic data 40c) includes texture data of facial features, such as eyebrow movements, jaw movements, nose movements, cheek movements, etc. For example, when a person (e.g., user 25) smiles, the upper facial features and lower facial features (e.g., upper facial feature characteristic data 40a and lower facial feature characteristic data 40c) may include a number of muscle movements that can be repeated by a representation of the user (e.g., an avatar) based on captured data from sensor 35.

[0047] Physiological data (e.g., eye data, head / body data, etc.) may change over time, and device 10 may use the physiological data to measure one or both of a user's physiological response to visual characteristics 30 (e.g., viewing specific content on display 15, such as reading text) or a user's intention to interact with content 20. For example, when content 20 that may include interactive elements is presented by device 10, user 25 may select interactive elements based on the user's eye movements and gaze behavior associated with eye data, head / body data, etc., without requiring user 25 to complete physical button presses. In some implementations, the physiological data may include a physiological response to a visual stimulus or an auditory stimulus of a radius of pupil 50 after user 25 glances at content 20, measured via eye tracking technology (e.g., via an HMD). In some implementations, the physiological data includes EEG amplitude / frequency data measured via EEG technology or EMG data measured from an EMG sensor or motion sensor.

[0048] Figure 2A and Figure 2B Detecting eye movements and gaze behavior of a user viewing content based on physiological data is shown. Figure 2A 204 to a user (e.g., Figure 1 The user 25) is presented with content 202, wherein the user has a physiological response to the content via the obtained physiological data (e.g., the user looks at a portion of the content as detected by the eye gaze characteristic data 212). For example, at content presentation moment 200A, the user is presented with content 202 including visual content (e.g., a video / image of an application including text), and the user's physiological data, such as pupil data 214, head data 216, and body data 218, is monitored. Figure 2B Shown with Figure 2A A similar example, except that the user focuses his or her gaze on interactive element 203 (e.g., the user wants to select text presented to him or her, such as an embedded interactable icon), as shown at selection notification 205. Thus, content 202 may be updated based on the user's interaction / focus on interactive element 203 (e.g., the user wants to select the embedded interactable icon represented by interactive element 250).

[0049] exist Figure 2A and Figure 2B In the specific example of , at the content presentation moment 200A, the user's eye gaze characteristics are less focused on the content 202. Then, at Figure 2B At the content presentation moment 200B (e.g., during the focus phase), the user's eye gaze characteristics 212 appear to be focused on the interactive element 203 of the content 202. For example, the system can determine based on the user's eye gaze characteristics 212, pupil data 214, head data 216, and / or body data 218 that the user's gaze may be focused on a particular area or text, and is no longer just looking at the content (e.g., reading the text), but rather wants to select a portion of the content (e.g., select an icon or interact with a portion of the text, such as placing an order on a menu).

[0050] In some embodiments, the body data 218 includes tracking data while the user performs various tasks in various body / head postures (such as sitting, standing, lying down, walking, etc.), which can be used to determine that the user has a specific gaze behavior state to identify gaze shift events, gaze hold events, and / or loss events (or a combination thereof). In an exemplary embodiment, the body data 218 is not used during real-time classification of gaze behavior events, but the body data 218 is used to train the classification algorithm because gaze behavior and associated statistics can vary significantly depending on body posture and task requirements. For example, a user lying down while watching content on the device 10 may have different gaze characteristics than a user sitting at his or her desk watching content on the device 10 (or walking, standing, etc.). Therefore, as discussed herein and Figure 3 The presented gaze classification algorithm is pose-independent and task-independent, and is trained using both eye and head data from various scenes.

[0051] Figure 3 An example environment 300 of a classification framework for detecting and classifying gaze behavior events according to some implementations is shown. In some implementations, the processing flow of the example environment 300 is performed on a device (e.g., Figure 1 The content of one or more applications 370 may be displayed on a device (e.g., a mobile device, a desktop computer, a laptop computer, or a server device) having a screen for displaying images (e.g., the display 15) and / or a screen for viewing stereoscopic images (e.g., an HMD). Figure 1In some implementations, the process flow of the example environment 300 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the process flow of the example environment 300 is executed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0052] The processing flow of the example environment 300 focuses on a gaze classification algorithm of an eye movement and gaze behavior instruction set 340, which is pose-independent and task-independent, and is trained using eye data and head data (e.g., one or more training data sets 350) from various scenes. Specifically, the processing flow of the example environment 300 is to obtain physiological data over a period of time at the eye movement and gaze behavior instruction set 340, and determine gaze behavior event tags 360 in real time for the user's eye movement based on the physiological data. The gaze behavior event tags 360 may include different classifications of gaze behavior events, such as category-1 371 (e.g., fast behavior state-saccade), category-2 372 (e.g., stable behavior state-steady pursuit, VOR or gaze event), or category-3 373 (e.g., loss behavior state-blink, blink or other data loss). The gaze behavior event tags 360 can then be sent to one or more applications 370, which can quickly utilize the information as an interaction event according to the techniques described herein (e.g., predicting whether the user intends to interact with a portion of the displayed content).

[0053] In an exemplary embodiment, the physiological data used by the eye movement and gaze behavior instruction set 340 to determine the gaze behavior event tag 360 for the user's eye movement in real time is based on the eye tracking data 310 and the head tracking data 320. The eye tracking data 310 may include video-based data, pupil and light-based data, retinal imaging-based data, scleral coil-based data, EOG-based data, etc., or any other eye movement data discussed herein. The eye tracking data can then be obtained and analyzed by the eye movement and gaze behavior instruction set 340 or another eye tracking analysis algorithm to determine the left eye representation 312, the right eye representation 314, and the binocular statistics 316. The left eye representation 312 and the right eye representation 314 information may include analysis of the specific gaze angle, pupil size, velocity, acceleration, eye openness, confidence, etc. for each specific eye. The binocular statistics 316 may include information such as eye vergence, IPD, etc. The head tracking data 320 may include scene camera data position (e.g., RGB or IR camera), IMU data, depth sensing data, IR flood data, etc., toward the user's head, or any other head movement data discussed herein. The head tracking data may then be obtained and analyzed by an eye movement and gaze behavior instruction set 340 or another head tracking analysis algorithm to determine a head representation 322.

[0054] In some implementations, the eye movement and gaze behavior instruction set 340 may also obtain additional data to analyze in real time before determining an event label for a particular gaze behavior. For example, visual data 330 may be utilized by the eye movement and gaze behavior instruction set 340. Visual data 330 may include scene camera information (e.g., image data of the physical environment), VR / AR or other content currently being displayed to the user, depth maps, saliency maps, etc. Additionally, the visual data may be analyzed by an object detection / representation algorithm to determine object representation data 332 and provide it to the eye movement and gaze behavior instruction set 340. Object representation data 332 may include the location, speed, direction of movement, etc. of any or all identified objects in the displayed content and / or physical environment. Object representation data 332 may be used by the eye movement and gaze behavior instruction set 340 to refine eye behavior event classification based on object-specific actions (e.g., learning from object-specific actions). For example, an object displayed in a scene, such as a dog, may be identified, and the dog's position, speed, direction of movement, etc. may be tracked so that if the user looks at the dog (e.g., interrupted), the classification analysis techniques described herein may utilize this information to further refine the eye behavior event classification.

[0055] Eye tracking data 310, head tracking data 320, and visual data 330 may be used to train eye movement and gaze behavior instruction set 340 as the user performs various tasks in various body / head postures (such as sitting, standing, lying down, walking, etc.). Depending on the body posture and task requirements, gaze behavior and statistics may vary significantly. Thus, the gaze classification algorithm presented herein for eye movement and gaze behavior instruction set 340 is posture-independent and task-independent, and is trained using eye data and head data from various scenes. In addition to body posture, different tasks may also be represented in training data set 350, such as reading, watching, playing games, searching for objects, talking to someone, interacting with user interface elements, etc. (e.g., scene understanding information). For example, the training data set 350 may include physiological data 352 (e.g., eye tracking data 310, head tracking data 320, etc.), body posture data 352 (e.g., information about the user's body posture (such as sitting, standing, lying down, etc.)), task data 354 (e.g., information about the user's specific task (such as reading, meditating, walking, etc.)), and mental state data 356 (e.g., the user's attention state, stress detection, distraction, etc.).

[0056] Figure 4 is a system flow diagram of an example environment 400 according to some implementations, in which a gaze behavior classification system can detect gaze behavior events of a user viewing content based on physiological data. In some implementations, the system flow of the example environment 400 is performed on a device (e.g., Figure 1 The content of the example environment 400 may be displayed on a device (e.g., a display 15) for displaying images and / or a screen for viewing stereoscopic images (e.g., an HMD) (e.g., a mobile device, a desktop computer, a laptop computer, or a server device). Figure 1 In some implementations, the system flow of the example environment 400 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the example environment 400 is executed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0057] System flow of example environment 400: obtain content (e.g., video content or a series of image data) and present it to user 25; analyze content and / or environment to obtain contextual data; obtain physiological data associated with the user during the presentation of content (e.g., eye data, head / body data, etc.); evaluate the user's intention to interact with a portion of the content (e.g., interactive element 405) based on determining a gaze behavior state according to the physiological data; and update the content based on an interaction event (e.g., if the user 25 focuses on the interactive element 405 for a certain period of time to activate or select the interactive element 405). For example, the gaze behavior classification technology described herein determines the user's intention to interact with a portion of the content (e.g., interactive element 405) during an experience (e.g., watching a video) based on the acquired physiological data and the determined gaze behavior (e.g., for identifying a gaze shift event, a gaze hold event, a loss event, or a combination thereof) by updating the content based on the user's interaction event (e.g., a notification, an auditory signal, a warning, etc. to alert the user that they have selected the interactive element 405 during the presentation of the content 402).

[0058] The example environment 400 includes a content instruction set 410 configured to be executable by a processor to provide and / or track content to be displayed on a device (e.g., Figure 1 402). For example, when the user is in a physical environment 404 (e.g., a room, outdoors, etc.), the content instruction set 410 provides a content presentation moment 412 including the content 402 to the user 25. For example, the content 402 may include background images and sound data (e.g., video). The content presentation moment 412 may be an XR experience that includes some virtual content of the physical environment and some images or views. Alternatively, the user may wear the HMD and look at the real physical environment via a live camera view, or the HMD allows the user to view the display, such as wearing smart glasses through which the user can view, but still presents visual cues and / or audio cues. During the experience, while the user 25 is viewing the content 402, eye gaze characteristic data 415 and pupil data 414 of the user's eyes may be monitored and sent as physiological data 417. Additionally, other physiological data may be monitored and sent as physiological data 417 (such as head movement data 416 and body movement data 418 obtained from an IMU, or image data).

[0059] The environment 400 also includes a physiological tracking instruction set 430 to track physiological attributes of a user as physiological tracking data 432 using one or more of the techniques discussed herein or other techniques as may be appropriate. For example, the physiological tracking instruction set 430 may obtain physiological data 417 (e.g., eye gaze characteristic data 415, pupil data 414) from a user 25 viewing the content 402. Additionally or alternatively, the user 25 may wear a sensor 420 (e.g., such as an EEG sensor, an EDA sensor, a heart rate sensor, etc.) that generates sensor data 422 (e.g., IMU or pose data for the head or body, EEG data, EDA data, heart rate data, etc.) as additional physiological data. Thus, when content 402 is presented to the user as content presentation moment 412, physiological data 417 (e.g., eye gaze characteristic data 415, pupil data 414) and / or sensor data 422 (e.g., head movement data 416 and body movement data 418) is sent to a physiological tracking instruction set 430 to track physiological attributes of the user as physiological tracking data 432 using one or more of the techniques discussed herein or other techniques that may be appropriate.

[0060] In an example implementation, the environment 400 also includes a scene understanding instruction set 440 configured with instructions executable by a processor to obtain experience data (e.g., content 402) and other sensor data (e.g., image data of the environment 404, image data of the face and / or eyes of the user 25, etc.) presented to the user and generate scene understanding as context data 442 (e.g., identifying people, objects, etc. of the content 402 and the environment 404). For example, the scene understanding instruction set 440 obtains the content 402 and the sensor data 422 (e.g., image data) from the sensor 420 (e.g., RGB camera, depth camera, etc.), and determines the context data 442 based on the identified area of ​​the content when the user is viewing a presentation of the content 402 (e.g., the content / video being viewed for the first time). Alternatively, the scene understanding instruction set 440 selects context data associated with the content 402 from the context database 445 (e.g., if the content 402 has been previously analyzed by the scene understanding instruction set, e.g., a previously viewed / analyzed video). In some implementations, scene understanding instruction set 440 generates scene understanding associated with content 402 and / or environment 404 as context data 442. For example, scene understanding can be utilized to track what a user may be focused on during presentation of content 402, or where the user is, what the user is doing, and the overall context of what physical objects or people are near the user with respect to environment 404.

[0061] In an example implementation, environment 400 also includes a gaze behavior classification instruction set 450 configured with instructions executable by a processor to evaluate the user 25's intent to interact with (e.g., select) an interactive element 405 (e.g., a portion of text) based on gaze behavior and eye movement evaluation using one or more of the techniques discussed herein or other techniques that may be appropriate. For example, the user 25's intent to interact with the interactive element 405 may be evaluated, such as determining that the user 25 is focused on a particular lighting area of ​​the interactive element 405 (e.g., such as a portion of text). Figure 2B Specifically, the gaze behavior classification instruction set 450 obtains the physiological tracking data 432 from the physiological tracking instruction set 430 and determines the user 25's intention to interact with (select) the interactive element 405 during the presentation of the content 402 while the user is viewing the content 402 (e.g., reading text with embedded selectable elements). In some specific implementations, the gaze behavior classification instruction set 450 can then provide the content instruction set 410 with gaze behavior event classification data 452 (e.g., signaling the user's selection of the interactive element 405 while viewing the content 402 and data of the reading characteristics associated with the user 25).

[0062] In some implementations, the gaze behavior classification instruction set 450 also obtains contextual data 442 (e.g., scene understanding data) from the scene understanding instruction set 440 with the physiological tracking data 432 to determine the user 25's intent to interact with (select) the interactable element 405 during the presentation of the content 402. For example, the contextual data 442 can provide a scene analysis that can be used by the gaze behavior classification instruction set 450 to understand what a person is looking at, where they are, etc., and improve the determination of the user's intent to select the interactable element 405.

[0063] Figure 5 5 is a flow chart illustrating an exemplary method 500. In some implementations, a device (e.g., Figure 1 The method 500 is performed by a device 10 of the present invention to determine the gaze behavior state based on the physiological data, thereby identifying the user's gaze shift event, gaze retention event, and loss event. In some specific implementations, the method 500 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some specific implementations, the method 500 is performed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 500 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0064] At block 502, method 500 obtains eye data associated with gaze during a first time period. In some implementations, information for the eye may be obtained for both azimuth and elevation for both position and velocity of each eye (e.g., eight data channels of a classification algorithm). In some implementations, the eye data may also include IPD information (e.g., one data channel for the classification algorithm) and pupil diameters for the left and right eyes (e.g., two data channels for the classification algorithm). Obtaining the eye data may involve obtaining EEG information, an image of the eye, and / or EOG data, from which a gaze direction, a gaze angle, and / or eye movement may be determined. In some implementations, the eye data may also include a gaze direction, a gaze velocity, or a combination thereof.

[0065] In some implementations, obtaining eye data is based on obtaining sensor data of the user. For example, sensor data (e.g., live data, such as video content including light intensity data (RGB) and depth data) is associated with a point in time, such as from an inward / downward facing sensor (e.g., when the user is wearing the HMD) Figure 1 An image of the sensor 35a, 35b, 35c) shown in FIG. 3 is associated with a frame. In some implementations, the sensor data includes depth data (e.g., infrared, time of flight, etc.) and light intensity image data obtained during the scanning process.

[0066] In some implementations, obtaining physiological data (e.g., pupil data 40) is associated with the user's gaze, which may involve obtaining images of the eye or electrooculogram signal (EOG) data from which gaze direction and / or movement may be determined. In some implementations, the physiological data includes at least one of skin temperature, respiration, photoplethysmography (PPG), electrodermal activity (EDA), eye gaze tracking, and pupil movement associated with the user. In some implementations, obtaining physiological data includes head movement of the user (e.g., obtained from an IMU or from image sensor data), such as head tracking data 320.

[0067] In some implementations, the movement and / or positioning and features of the user's 25 head 27 (e.g., the edges of the eyes, nose, or nostrils) are extracted by the device 10 and used to find the coarse positioning coordinates of the user's 25 eyes 45, thereby simplifying the determination of precise eye 45 features (e.g., position, gaze direction, etc.), and making the gaze characteristic measurement and corresponding reading characteristics more reliable and more robust. In addition, the device 10 can easily combine the 3D positioning of the portion of the head 27 with the gaze angle information obtained via the eye portion image analysis to identify a given screen object viewed by the user 25 at any given time. In some implementations, the use of 3D mapping in conjunction with gaze tracking allows the user 25 to freely move his or her head 27 and eyes 45 while reducing or eliminating the need to actively track the head 27 using sensors or transmitters on the head 27.

[0068] By tracking the eyes 45, some implementations reduce the need to recalibrate the user 25 after the user 25 moves his or her head 27. In some implementations, the device 10 uses depth information to track the movement of the pupil 50, thereby enabling a reliable rendered pupil diameter to be calculated based on a single calibration of the user 25. Using techniques such as pupil center corneal reflection (PCCR), pupil tracking, and pupil shape, the device 10 can calculate the pupil diameter and the gaze angle of the eyes 45 from a fixed point of the head 27, and use the positioning information of the head 27 to recalculate the gaze angle and other gaze characteristic measurements. In addition to reduced recalibration, further benefits of tracking the head 27 can include reducing the number of light projection sources and reducing the number of cameras used to track the eyes 45.

[0069] Some specific implementations obtain physiological data and other user information to help improve user experience. In such processes, as an example, user preferences and privacy should be respected by ensuring that the user understands and agrees to the use of user data, understands what type of user data is used, controls the collection and use of user data, and limits the distribution of user data (e.g., by ensuring that user data is processed locally on the user's device). The user should have the option of opting in or opting out of any feature part of whether to obtain or use its user data or otherwise turn on and off the acquisition or use of user information. In addition, each user will have the ability to access and otherwise find out anything about him or her that the system has collected or determined. User data is securely stored on the user's device. User data used as input for machine learning models is securely stored on the user's device, for example, to ensure the user's privacy. The user's device may have a secure storage area (e.g., a secure partition) for protecting certain user information, such as data from image sensors and other sensors for facial identification or biometric identification. User data associated with the user's body and / or attention state may be stored in such a secure partition, thereby limiting access to user data and limiting the transmission of user data to other devices to ensure that user data remains securely on the user's device. User data may be prohibited from leaving the user device and may be used only in machine learning models and other processes on the user device.

[0070] At box 504, method 500 obtains head data associated with the gaze during the first time period. The head data may include an image of the user's head from one or more sensors. In some specific implementations, obtaining the head data includes tracking the pose and movement of the user's head. For example, head motion and pose may be obtained based on position information and velocity information obtained from one or more sensors. Information for the head (e.g., four data channels of a classification algorithm) may be obtained for both azimuth and elevation for both position and velocity. Obtaining head data may involve obtaining an image of the head (e.g., RGB or IR), an IMU, depth sensing from an IFC, an IR flood, etc.

[0071] In some implementations, obtaining sensor data associated with the head data may include obtaining a second set of data corresponding to one or more partial views of the face from one or more image sensors while the user is using (e.g., wearing) the electronic device (e.g., HMD). For example, obtaining the sensor data may include obtaining live image data. In some implementations, the second set of data includes a partial image of the user's face and, therefore, may not represent all features of the face represented in the enrollment data. For example, the second set of images may include images from an upward-facing sensor (e.g., Figure 1Additionally or alternatively, the second set of images may include images of some of the front face / eyes of the sensor 35a (e.g., facial feature characteristic data 40a). Additionally or alternatively, the second set of images may include images from the inward-facing sensor (e.g., Figure 1 Additionally or alternatively, the second set of images may include images of some of the eyes from sensors facing downward (e.g., Figure 1 In some implementations, the electronic device includes a first sensor (e.g., Figure 1 sensor 35a) and a second sensor (e.g., Figure 1 The method further comprises the step of: obtaining a second set of data from at least one partial image of the user's face seen from a first viewpoint from the first sensor (e.g., upper facial feature data 40a) and at least one partial image of the user's face seen from a second viewpoint different from the first viewpoint from the second sensor (e.g., lower facial feature data 40c) (e.g., multiple IFC cameras are used to capture different viewpoints of the user's face and body movements).

[0072] At box 506, method 500 determines a first gaze behavior state during a first time period based on eye data and head data (e.g., collected at boxes 502 and 504, respectively) to identify a gaze transfer event, a gaze maintenance event, or a loss event (or a combination thereof). For example, the technology described herein can obtain eye and head data associated with the user's gaze, and the eye and head posture characteristics can be determined, aggregated, and used to classify the user's eye movement state using machine learning techniques. Categories may include main categories, such as gaze maintenance states, gaze transfer states, and loss behavior states. In some specific implementations, these three main categories may include additional subcategories that can be classified. For example, a gaze maintenance state (e.g., a stable behavior state) may include gaze, VOR, tracking, slow convergence, etc. A gaze transfer state (e.g., a fast behavior state) may include one or more glance levels, fast convergence, etc. In some specific implementations, a main category may include a combination of a gaze maintenance state and a gaze transfer state (e.g., optokinetic nystagmus, eye following response (OFR), vestibular nystagmus, etc.). Lost behavior states (e.g., eye tracking loss) may include eye blinks, eye blinks, and other data loss events that may be determined as lost behavior states due to other issues with eye / head tracking (e.g., dropped frames, segmentation issues, etc.).

[0073] In some implementations, determining that the user has a first gaze behavior state during a first time period is based on a set of data acquired before the first time period. For example, in order to determine the user's gaze behavior and eye movement in real time, a previous time period (e.g., the most recent N samples, where N is a positive integer, such as the most recent 10 data samples) for obtaining eye data and / or head data can be used in conjunction with the current data set.

[0074] In some specific implementations, the first gaze behavior state is a type of gaze behavior state in a plurality of gaze behavior states, and the plurality of gaze behavior states include a stable gaze state, a fast gaze state, and a lost gaze state. In some specific implementations, the method 500 includes identifying a gaze transfer event associated with the user during the first time period based on determining that the first gaze behavior state includes a fast gaze state during the first time period. In some specific implementations, the method 500 includes identifying a gaze retention event associated with the user during the first time period based on determining that the first gaze behavior state includes a stable gaze state during the first time period. For example, a higher level of classification may also be performed, wherein the gaze behavior is classified into a fast behavior state, a stable behavior state, and a lost behavior state. The fast behavior state may refer to a glance. The stable behavior state may refer to a user who may be trying to stabilize his or her gaze on an object (e.g., a steady pursuit, a vestibular ocular reflex (VOR), or a gaze event). The lost behavior state may refer to blinking, blinking, and / or other data loss that may be determined as a lost behavior state due to other problems (e.g., frame loss, segmentation problems, etc.) of eye / head tracking.

[0075] In some specific implementations, determining that a user has a first gaze behavior state during a first time period is based on using a machine learning classifier model, wherein eye data and head data are input into the machine learning classification model to identify gaze transfer events, gaze retention events, and / or loss events (or a combination thereof). In some specific implementations, the machine learning classification model is trained based on multiple body postures of a group of users. For example, a group of users of the training data may be informed to perform multiple body postures / actions, such as lying down, standing, sitting, walking, etc., and eye data and head data may be obtained for the machine learning classification model to identify gaze transfer events, gaze retention events, and loss events that can adapt to the range of eye and head movement based on different body postures / actions. Depending on body posture and task requirements, gaze behavior and statistical values ​​may vary significantly. Therefore, the gaze classification algorithm presented herein is posture-independent and task-independent, and is trained using eye data and head data from various scenes.

[0076] In some implementations, method 500 includes determining scene understanding of a physical environment based on obtaining sensor data from one or more sensors corresponding to the physical environment. In some implementations, determining that a user has a first gaze behavior state during a first time period is based on scene understanding of the physical environment. In some implementations, the physical environment includes one or more objects, and determining scene understanding of the physical environment includes determining the location, speed, or direction of motion of one or more objects. For example, determining scene understanding may include identifying one or more objects in the object and their location based on an image of the user's current room, depth data, etc. (including the location, speed, direction of motion of objects in the scene / environment). In some implementations, determining scene understanding includes generating a three-dimensional (3D) representation of the physical environment. For example, after the scanning process, a 3D model, such as a 3D point cloud or a 3D rendering, may be generated for the physical environment and / or objects within the physical environment.

[0077] In some aspects, method 500 determines the context of the experience based on sensor data of the environment. For example, determining the context may involve using computer vision to generate scene understanding of visual and / or auditory attributes of the environment—where the user is, what the user is doing, what objects are nearby. Additionally, scene understanding of content presented to the user may be generated, including visual and / or auditory attributes of the content the user is viewing.

[0078] In some aspects, different contexts of the presented content and environment are analyzed to determine where the user is, what the user is doing, what objects or people in the environment or within the content are nearby, what the user did earlier (e.g., meditating in the morning). Additionally, the context analysis may include image analysis (semantic segmentation), audio analysis (vibration sound), positioning sensors (where the user is), motion sensors (fast-moving vehicles), and even access to other user data (e.g., the user's calendar). In an exemplary implementation, method 500 may also include determining the context of the experience by generating a scene understanding of the environment based on sensor data of the environment, the scene understanding including visual or auditory attributes of the environment, and determining the context of the experience based on the scene understanding of the environment.

[0079] In some implementations, the sensor data includes image data, and generating scene understanding is based at least on performing semantic segmentation of the image data and detecting one or more objects within the environment based on the semantic segmentation. In some implementations, determining the context of the experience includes determining an activity of a user based on scene understanding of the environment. In some implementations, the sensor data includes location data of the user, and determining the context of the experience includes determining a location of the user within the environment based on the location data.

[0080] In some implementations, method 500 includes updating a view of a display of an electronic device during a first time period based on determining that a user has a first gaze behavior state during a first time period. For example, in response to a correlation between an event (e.g., eye movement), an application on the display performs a certain type of response (e.g., the user interacts with a selectable element on a GUI). In some implementations, updating the view of the display is based on eye data and / or head data. For example, based on a gaze behavior associated with a head or eye movement (e.g., a particular type of classification), the application performs a particular action.

[0081] Figure 6 6 is a block diagram of an example device 600. Device 600 illustrates an exemplary device configuration for device 10. Although certain specific features are illustrated, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 10 includes one or more processing units 602 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 606, one or more communication interfaces 608 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 610, one or more displays 612, one or more internal-facing and / or external-facing image sensor systems 614, memory 620, and one or more communication buses 604 for interconnecting these and various other components.

[0082] In some implementations, the one or more communication buses 604 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 606 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0083] In some implementations, one or more displays 612 are configured to present a view of a physical or graphical environment to a user. In some implementations, one or more displays 612 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 612 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, device 10 includes a single display. In another example, device 10 includes a display for each eye of the user.

[0084] In some implementations, the one or more image sensor systems 614 are configured to obtain image data corresponding to at least a portion of the physical environment 105. For example, the one or more image sensor systems 614 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 614 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 614 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0085] The memory 620 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, the memory 620 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 620 optionally includes one or more storage devices remotely located from the one or more processing units 602. The memory 620 includes a non-transitory computer-readable storage medium.

[0086] In some implementations, the memory 620 or a non-transitory computer-readable storage medium of the memory 620 stores an optional operating system 630 and one or more instruction sets 640. The operating system 630 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 640 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 640 is software that can be executed by one or more processing units 602 to implement one or more of the techniques described herein.

[0087] The instruction set 640 includes a content instruction set 642, a physiological tracking instruction set 644, a scene understanding instruction set 646, and a gaze behavior instruction set 648. The instruction set 640 may be embodied as a single software executable file or as a plurality of software executable files.

[0088] In some implementations, content instruction set 642 may be executed by processing unit 602 to provide and / or track content for display on a device. Content instruction set 642 may be configured to monitor and track content over time (e.g., during an experience) and / or identify change events that occur within the content (e.g., based on identified / classified behavioral gaze events). In some implementations, content instruction set 642 may be configured to add change events to content (e.g., a feedback mechanism) using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0089] In some implementations, the physiological tracking instruction set 644 may be executed by the processing unit 602 to track physiological attributes of the user (e.g., EEG amplitude / frequency, pupil modulation, gaze, head pose / positioning data, heart rate, EDA data, etc.) using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0090] In some implementations, scene understanding instruction set 646 may be executed by processing unit 602 to determine the context of an experience and / or environment (e.g., creating scene understanding to determine objects or people in content or in an environment, where a user is, what a user is viewing, etc.) using one or more of the techniques discussed herein (e.g., object detection, facial recognition, etc.) or other techniques that may be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0091] In some implementations, the gaze behavior instruction set 648 may be executed by the processing unit 602 to evaluate physiological data (e.g., eye data and head data) using one or more of the techniques discussed herein or other techniques that may be appropriate and determine that the user has a first gaze behavior state during a period of time to identify gaze shift events, gaze hold events, and loss events. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0092] Although the instruction set 640 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Figure 6 More so as to serve as functional descriptions of various features present in a particular implementation that are different from the structural schematics of the implementations described herein. As will be appreciated by one of ordinary skill in the art, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how the features are distributed among them will vary depending on the implementation and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0093] Figure 7 A block diagram of an exemplary head mounted device 700 according to some implementations is shown. The head mounted device 700 includes a housing 701 (or enclosure) that houses various components of the head mounted device 700. The housing 701 includes (or is coupled to) eye pads (not shown) disposed at the proximal (of the user 25) end of the housing 701. In various implementations, the eye pads are plastic or rubber pieces that comfortably and snugly hold the head mounted device 700 in place on the face of the user 25 (e.g., around the eyes of the user 25).

[0094] Housing 701 houses display 710, which displays images, emits light toward or onto the eyes of user 25. In various implementations, display 710 emits light through an eyepiece having one or more optical elements 705, which refract the light emitted by display 710, so that the display appears to be at a virtual distance farther from the user 25 than the actual distance from the eye to display 710. For example, optical element 705 may include one or more lenses, waveguides, other diffractive optical elements (DOEs), etc. In order for user 25 to be able to focus on display 710, in various implementations, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). In addition, in order to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

[0095] The housing 701 also houses a tracking system that includes one or more light sources 722, a camera 724, a camera 732, a camera 734, and a controller 780. The one or more light sources 722 emit light onto the eyes of the user 25, which is reflected as a light pattern (e.g., a flashing circle) that can be detected by the camera 724. Based on the light pattern, the controller 780 can determine the eye tracking characteristics of the user 25. For example, the controller 780 can determine the gaze direction and / or blink state (eyes open or closed) of the user 25. As another example, the controller 780 can determine the pupil center, pupil size, or focus point. Therefore, in various specific implementations, light is emitted by the one or more light sources 722, reflected from the eyes of the user 25, and detected by the camera 724. In various specific implementations, the light from the eyes of the user 25 is reflected from the hot mirror or passes through the eyepiece before reaching the camera 724.

[0096] The display 710 emits light within a first wavelength range, and the one or more light sources 722 emit light within a second wavelength range. Similarly, the camera 724 detects light within the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range of approximately 400 nm-700 nm within the visible spectrum), and the second wavelength range is a near infrared wavelength range (e.g., a wavelength range of approximately 700 nm-1400 nm within the near infrared spectrum).

[0097] In various implementations, eye tracking (or specifically, determined gaze direction) is used to enable user interaction (e.g., user 25 selects an option on display 710 by looking at it), provide perforated rendering (e.g., presenting a higher resolution in the area of ​​display 710 that user 25 is looking at and a lower resolution elsewhere on display 710), or correct distortion (e.g., for an image to be provided on display 710).

[0098] In various implementations, the one or more light sources 722 emit light toward the eyes of the user 25, which is reflected in the form of a plurality of flashes.

[0099] In various implementations, camera 724 is a frame / shutter based camera that generates images of the user's 25 eyes at a particular point in time or multiple points in time at a frame rate. Each image includes a matrix of pixel values ​​corresponding to pixels of the image, the pixels corresponding to the locations of the camera's light sensor matrix. In a specific implementation, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0100] In various implementations, camera 724 is an event camera that includes multiple light sensors (e.g., a light sensor matrix) at multiple corresponding locations, which generates an event message indicating a specific location of a specific light sensor in response to the specific light sensor detecting a change in light intensity.

[0101] In various implementations, camera 732 and camera 734 are frame / shutter based cameras that can generate images of the face of user 25 at a specific point in time or multiple points in time at a frame rate. For example, camera 732 captures an image of the user's face below the eyes, and camera 734 captures an image of the user's face above the eyes. The images captured by camera 732 and camera 734 may include light intensity images (e.g., RGB) and / or depth image data (e.g., time of flight, infrared, etc.).

[0102] According to some specific implementations, an electronic device (e.g., device 10, device 600, device 700) may generate an extended reality (XR) environment during a communication session and present the XR environment to one or more users. An extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic device, compared to a physical environment that people can sense and / or interact with without the help of an electronic device. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system may detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. As another example, the XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting the XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), the XR system may adjust characteristics of the graphical content in the XR environment in response to representations of physical movement (e.g., voice commands).

[0103] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display may be configured to selectively become opaque. A projection-based system may employ retinal projection technology that projects a graphic image onto a person's retina. The projection system may also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0104] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0105] As described above, one aspect of the present technology is to collect and use physiological data to improve the user's electronic device experience in interacting with electronic content. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0106] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the interaction and control capabilities of electronic devices. Therefore, the use of such personal information data enables planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0107] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and measures that are recognized as meeting or exceeding the industry or government requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. Additionally, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0108] Regardless of the foregoing, the present disclosure also contemplates implementations in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.

[0109] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0110] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identification and / or without personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access his or her stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access the data stored therein.

[0111] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, one skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0112] Unless otherwise specifically noted, it should be understood that throughout this specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0113] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0114] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0115] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. Additionally, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for ease of explanation only and are not intended to be limiting.

[0116] It will also be understood that, although the terms "first", "second", etc. may be used to describe various objects in this article, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0117] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, objects or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, parts or their groupings.

[0118] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0119] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method comprising: At an electronic device having a processor and one or more sensors: obtaining eye data associated with gaze during a first time period; obtaining head data associated with the gaze during the first time period; as well as A first gaze behavior state during the first time period is determined based on the eye data and the head data to identify a gaze shift event, a gaze hold event, or a gaze loss event.

2. The method according to claim 1, further comprising: Scene understanding of the physical environment is determined based on obtaining sensor data corresponding to the physical environment from the one or more sensors. The method of claim 2 , wherein determining the first gaze behavior state during the first time period is based on the scene understanding of the physical environment.

4. The method of claim 2 or 3, wherein the physical environment comprises one or more objects, and wherein determining the scene understanding of the physical environment comprises determining a position, velocity, or direction of motion of the one or more objects.

5. The method of any one of claims 2 to 4, wherein determining scene understanding comprises generating a three-dimensional (3D) representation of the physical environment.

6. The method according to any one of claims 1 to 5, wherein the eye data comprises position information and velocity information for a left eye and a right eye.

7. The method according to any one of claims 1 to 6, wherein the eye data comprises an interpupillary distance (IPD) between a right eye and a left eye, and a diameter of a left eye and a diameter of a right eye.

8. The method according to any one of claims 1 to 7, wherein the eye data comprises: Gaze direction; or Look at the speed.

9. The method according to any one of claims 1 to 8, wherein the eye data comprises an image of the eye or electrooculogram (EOG) data.

10. The method of any one of claims 1 to 9, wherein the head data comprises an image of the head from the one or more sensors.

11. The method according to any one of claims 1 to 10, wherein obtaining the head data comprises tracking the position and movement of the head. 12 . The method according to claim 1 , wherein determining the first gaze behavior state during the first time period is based on a set of data acquired before the first time period.

13. The method according to any one of claims 1 to 12, wherein the first gaze behavior state is a type of gaze behavior state among multiple gaze behavior states, wherein the multiple gaze behavior states include a gaze hold state, a gaze shift state, and an eye tracking loss state.

14. The method according to claim 13, further comprising: A gaze shift event during the first time period is identified based on determining that the first gaze behavior state includes a rapid gaze state during the first time period.

15. The method according to claim 13, further comprising: A gaze hold event during the first time period is identified based on determining that the first gaze behavior state includes a stable gaze state during the first time period.

16. The method according to any one of claims 1 to 15, further comprising: A view of a display of the electronic device during the first time period is updated based on determining the first gaze behavior state during the first time period. The method of claim 16 , wherein updating the view of the display is based on the eye data or the head data.

18. A method according to any one of claims 1 to 17, wherein the first gaze behavior state during the first time period is based on using a machine learning classifier model, wherein the eye data and the head data are input into the machine learning classification model to identify gaze shift events, gaze hold events and loss events.

19. The method of claim 18, wherein the machine learning classification model is trained based on multiple body poses.

20. The method of claim 19, wherein the plurality of body positions comprises lying, standing, sitting, walking, or a combination thereof.

21. A method according to any one of claims 1 to 20, wherein the eye data or the head data is obtained from sensor data from the one or more sensors, and wherein the sensor data includes depth data and light intensity image data obtained during a scanning process.

22. The method according to any one of claims 1 to 21, wherein the electronic device is a head mounted device (HMD).

23. A device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: obtaining eye data associated with gaze during a first time period; obtaining head data associated with the gaze during the first time period; and A first gaze behavior state during the first time period is determined based on the eye data and the head data to identify a gaze shift event, a gaze hold event, or a gaze loss event.

24. The apparatus of claim 23, wherein the program instructions further cause the system to perform operations comprising: Scene understanding of the physical environment is determined based on obtaining sensor data corresponding to the physical environment from one or more sensors.

25. The apparatus of claim 24, wherein determining the first gaze behavior state during the first time period is based on the scene understanding of the physical environment.

26. The apparatus of claim 24 or 25, wherein the physical environment comprises one or more objects, and wherein determining the scene understanding of the physical environment comprises determining a position, velocity, or direction of motion of the one or more objects.

27. The apparatus of any one of claims 24 to 26, wherein determining scene understanding comprises generating a three-dimensional (3D) representation of the physical environment.

28. An apparatus according to any one of claims 23 to 27, wherein the eye data comprises position information and velocity information for a left eye and a right eye.

29. The apparatus of any one of claims 23 to 28, wherein the eye data comprises an interpupillary distance (IPD) between a right eye and a left eye, and a diameter of a left eye and a diameter of a right eye.

30. Apparatus according to any one of claims 23 to 29, wherein the eye data comprises: Gaze direction; or Look at the speed.

31. An apparatus according to any one of claims 23 to 30, wherein the eye data comprises an image of the eye or electrooculogram (EOG) data.

32. An apparatus as claimed in any one of claims 23 to 31, wherein the head data comprises an image of the head from one or more sensors.

33. An apparatus according to any one of claims 23 to 32, wherein obtaining the head data comprises tracking the position and movement of the head.

34. The apparatus according to any one of claims 23 to 33, wherein determining the first gaze behavior state during the first time period is based on a set of data acquired before the first time period.

35. A device according to any one of claims 23 to 34, wherein the first gaze behavior state is a type of gaze behavior state among multiple gaze behavior states, wherein the multiple gaze behavior states include a gaze hold state, a gaze shift state, and an eye tracking loss state.

36. The apparatus of claim 35, wherein the program instructions further cause the system to perform operations comprising: A gaze shift event during the first time period is identified based on determining that the first gaze behavior state includes a rapid gaze state during the first time period.

37. The apparatus of claim 35, wherein the program instructions further cause the system to perform operations comprising: A gaze hold event during the first time period is identified based on determining that the first gaze behavior state includes a stable gaze state during the first time period.

38. The apparatus of any one of claims 23 to 37, wherein the program instructions further cause the system to perform operations comprising: A view of a display of the electronic device during the first time period is updated based on determining the first gaze behavior state during the first time period.

39. The apparatus of claim 38, wherein updating the view of the display is based on the eye data or the head data.

40. An apparatus according to any one of claims 23 to 39, wherein the first gaze behavior state during the first time period is based on using a machine learning classifier model, wherein the eye data and the head data are input into the machine learning classification model to identify gaze shift events, gaze hold events and loss events.

41. The apparatus of claim 40, wherein the machine learning classification model is trained based on multiple body poses.

42. The apparatus of claim 41, wherein the plurality of body positions comprises lying, standing, sitting, walking, or a combination thereof.

43. A device according to any one of claims 23 to 42, wherein the eye data or the head data is obtained from sensor data from one or more sensors, and wherein the sensor data includes depth data and light intensity image data obtained during a scanning process.

44. The device of any one of claims 23 to 43, wherein the electronic device is a head mounted device (HMD).

45. A non-transitory computer-readable storage medium storing program instructions executable on a device to perform operations comprising: obtaining eye data associated with gaze during a first time period; obtaining head data associated with the gaze during the first time period; as well as A first gaze behavior state during the first time period is determined based on the eye data and the head data to identify a gaze shift event, a gaze hold event, or a gaze loss event.

46. ​​The non-transitory computer-readable storage medium of claim 45, wherein the program instructions further cause the device to perform operations comprising: Scene understanding of the physical environment is determined based on obtaining sensor data corresponding to the physical environment from one or more sensors.

47. The non-transitory computer-readable storage medium of claim 46, wherein determining the first gaze behavior state during the first time period is based on the scene understanding of the physical environment.

48. A non-transitory computer-readable storage medium according to claim 46 or 47, wherein the physical environment includes one or more objects, and wherein determining the scene understanding of the physical environment includes determining the position, speed, or direction of movement of the one or more objects.

49. The non-transitory computer-readable storage medium of any one of claims 46 to 48, wherein determining scene understanding comprises generating a three-dimensional (3D) representation of the physical environment.

50. The non-transitory computer-readable storage medium of any one of claims 45 to 49, wherein the eye data comprises position information and velocity information for a left eye and a right eye.

51. The non-transitory computer-readable storage medium of any one of claims 45 to 50, wherein the eye data comprises an interpupillary distance (IPD) between a right eye and a left eye, and a diameter of a left eye and a diameter of a right eye.

52. The non-transitory computer-readable storage medium of any one of claims 45 to 51, wherein the eye data comprises: Gaze direction; or Look at the speed.

53. The non-transitory computer-readable storage medium of any one of claims 45 to 52, wherein the eye data comprises an image of the eye or electrooculogram (EOG) data.

54. The non-transitory computer-readable storage medium of any one of claims 45 to 53, wherein the head data comprises an image of the head from one or more sensors.

55. The non-transitory computer-readable storage medium of any one of claims 45 to 54, wherein obtaining the head data comprises tracking a pose and movement of the head.

56. The non-transitory computer-readable storage medium of any one of claims 45 to 55, wherein determining the first gaze behavior state during the first time period is based on a set of data acquired prior to the first time period.

57. A non-transitory computer-readable storage medium according to any one of claims 45 to 56, wherein the first gaze behavior state is a type of gaze behavior state among a plurality of gaze behavior states, wherein the plurality of gaze behavior states include a gaze hold state, a gaze shift state, and an eye tracking loss state.

58. The non-transitory computer-readable storage medium of claim 57, wherein the program instructions further cause the device to perform operations comprising: A gaze shift event during the first time period is identified based on determining that the first gaze behavior state includes a rapid gaze state during the first time period.

59. The non-transitory computer-readable storage medium of claim 57, wherein the program instructions further cause the device to perform operations comprising: A gaze hold event during the first time period is identified based on determining that the first gaze behavior state includes a stable gaze state during the first time period.

60. The non-transitory computer readable storage medium of any one of claims 45 to 59, The program instructions further cause the device to perform operations including: A view of a display of the electronic device during the first time period is updated based on determining the first gaze behavior state during the first time period.

61. The non-transitory computer readable storage medium of claim 60, wherein updating the view of the display is based on the eye data or the head data.

62. A non-transitory computer-readable storage medium according to any one of claims 45 to 61, wherein the first gaze behavior state during the first time period is based on using a machine learning classifier model, wherein the eye data and the head data are input into the machine learning classification model to identify gaze shift events, gaze hold events, and loss events.

63. The non-transitory computer-readable storage medium of claim 62, wherein the machine learning classification model is trained based on multiple body poses.

64. The non-transitory computer-readable storage medium of claim 63, wherein the plurality of body postures comprises lying, standing, sitting, walking, or a combination thereof.

65. A non-transitory computer-readable storage medium according to any one of claims 45 to 64, wherein the eye data or the head data is obtained from sensor data from one or more sensors, and wherein the sensor data includes depth data and light intensity image data obtained during a scanning process.

66. The non-transitory computer-readable storage medium of any one of claims 45 to 65, wherein the electronic device is a head mounted device (HMD).

Images