Adjusting content depth in an electronic device

Abstract

The invention relates to adjusting content depth in an electronic device. In a head-mounted device, a gaze tracking sensor may measure a divergence angle associated with an eye of a user. The divergence angle may be used to determine the eye depth of focus of the user. Virtual content may be presented at a depth equal to the depth of focus of the eye. When the depth of the virtual content is not equal to the depth of focus of the eye, the virtual content may be shifted toward the depth of focus of the eye. Once the depth of the virtual content is approximately equal to the depth of eye focus, the depth of the virtual content may be locked. The depth of the virtual content may remain locked until one or more unlocking criteria are met. The unlocking criteria may be based on a divergence angle of the user's eyes, a head rotation, and / or a head position.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 970,341, filed on December 5, 2024, and U.S. Provisional Patent Application No. 63 / 625,378, filed on January 26, 2024, the entireties of which are hereby incorporated by reference. Background Art

[0002] The present disclosure generally relates to head-mounted devices, and more particularly to head-mounted devices having a display.

[0003] Some electronic devices, such as head-mounted devices, include a display that presents virtual content at a certain depth. Without attention, the depth of the virtual content may not match the depth at which the user's eyes are focused, resulting in the virtual content appearing blurry to the user.

[0004] It is in this context that the embodiments herein are created. Summary of the Invention

[0005] An electronic device may include one or more sensors, one or more displays, one or more processors, and a memory that stores instructions configured to be executed by the one or more processors, the instructions for: displaying a virtual object at a first depth via the one or more displays; using a first subgroup of the one or more sensors to determine that the eye focus depth is different from the first depth; while displaying the virtual object, causing the virtual object to shift from the first depth towards the eye focus depth; and locking the virtual object at the current depth based on determining that a first criterion associated with the current depth of the virtual object relative to the eye focus depth is met.

[0006] An electronic device may include one or more sensors, one or more displays, one or more processors, and a memory that stores instructions configured to be executed by the one or more processors, the instructions for: displaying a virtual object at a first depth via the one or more displays; using a first subgroup of the one or more sensors to determine one or more parameters including the vergence angle; continuing to display the virtual object at the first depth based on determining that a first change in any one of the one or more parameters does not meet a first criterion; and causing the virtual object to shift from the first depth towards the current eye focus depth based on determining that a second change in the vergence angle does meet the first criterion.

[0007] An electronic device may include one or more displays, one or more lenses, one or more processors, and a memory that stores instructions configured to be executed by the one or more processors. The instructions are for: displaying a virtual object at a first depth via the one or more displays while the one or more lenses have a focal length at the first depth; adjusting the focal length of the one or more lenses from the first depth to a second depth different from the first depth at a first rate; and shifting the virtual object from the first depth toward the second depth at a second rate based on the first rate while displaying the virtual object. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of an exemplary head-mounted device according to some embodiments.

[0009] Figures 2A to 2C are illustrations of an exemplary user of a head-mounted device according to some embodiments, which show how the head pose of the user can be defined by yaw, roll, and pitch, respectively.

[0010] Figure 3 is a top view of an exemplary head-mounted device having a lens module according to some embodiments.

[0011] Figure 4A and Figure 4B are illustrations showing how the focal length of the lens module in a head-mounted device can vary in accordance with the depth of virtual content presented by the head-mounted device according to some embodiments.

[0012] Figure 5A is a graph showing an exemplary profile of the focal length of the lens module as a function of time according to some embodiments.

[0013] Figure 5B is a graph showing an exemplary profile of the depth of virtual content as a function of time according to some embodiments.

[0014] Figure 6A and Figure 6B are illustrations of exemplary left and right displays according to some embodiments, which show how the depth of virtual content can be updated by laterally shifting a first image and a second image on the left and right displays and changing the sizes of the first image and the second image.

[0015] Figure 7 is a flowchart of an exemplary method of operating a head-mounted device that updates the depth of virtual content at a rate based on the rate of change of the focal length in the lens module.

[0016] Figure 8A andFigure 8B FIG. illustrates how the vergence angle of a user's eyes can indicate the depth at which the eyes are focused, according to some embodiments.

[0017] Figure 9 FIG. illustrates an exemplary cyclic vergence estimation error caused by an error in the vergence angle measurement made by a gaze tracking sensor, according to some embodiments.

[0018] Figures 10A to 10D FIG. illustrates how the depth of virtual content can be locked to reduce cyclic vergence estimation error, according to some embodiments.

[0019] Figure 11A and Figure 11B FIG. illustrates how head rotation can be used to unlock the depth of virtual content, according to some embodiments.

[0020] Figure 12A and Figure 12B FIG. illustrates how head position can be used to unlock the depth of virtual content, according to some embodiments.

[0021] Figure 13 FIG. is an illustration of an exemplary head-mounted device having a filtering circuit that filters data from a gaze tracking sensor, according to some embodiments.

[0022] Figure 14A and Figure 14B FIG. is a flowchart of an exemplary method for locking and unlocking the depth of virtual content, according to some embodiments. DETAILED DESCRIPTION

[0023] The head-mounted device can display different types of extended reality content for the user. The head-mounted device can display virtual objects perceived at an apparent depth within the user's physical environment. The virtual objects can sometimes be displayed at a fixed location relative to the user's physical environment. For example, consider an example where the user's physical environment includes a table. The virtual object can be displayed to the user such that the virtual object appears to be resting on the table. When the user moves their head and otherwise interacts with the XR environment, the virtual object remains at the same fixed position on the table (e.g., as if the virtual object were another physical object in the XR environment). This type of content can be referred to as world-locked content (because the position of the virtual object is fixed relative to the user's physical environment).

[0024] Other virtual objects can be displayed at locations defined relative to the head-mounted device or the user of the head-mounted device. First, consider an example of a virtual object displayed at a location defined relative to the head-mounted device. When the head-mounted device moves (e.g., as the user's head rotates), the virtual object remains in a fixed position relative to the head-mounted device. For example, the virtual object can be displayed at a specific distance in front of and centered on the head-mounted device (e.g., in the center of the device's field of view or the user's field of view). When the user moves their head left and right, their field of view of their physical environment changes accordingly. However, when the user moves their head, the virtual object can remain fixed at a specific distance in the center of the device's field of view or the user's field of view (assuming the gaze direction remains constant). This type of content can be referred to as head-locked content. The head-locked content is fixed in a given position relative to the head-mounted device (and thus relative to the user's head that supports the head-mounted device). The head-locked content may not be adjusted based on the user's gaze direction. In other words, if the user's head position remains constant and their gaze is directed away from the head-locked content, the head-locked content will remain in the same apparent position.

[0025] Second, consider an example of a virtual object displayed at a location defined relative to a part of the user of the head-mounted device (e.g., relative to the user's torso). This type of content can be referred to as body-locked content. For example, the virtual object can be displayed in front of and to the left of the user's body (e.g., at a position defined by a distance and an angular offset from the forward direction of the user's torso), regardless of which direction the user's head is facing. If the user's body is facing a first direction, the virtual object will be displayed in front of and to the left of the user's body. When facing the first direction, regardless of whether the user rotates their head left and right (to look towards and away from the virtual object), the virtual object can remain in the same fixed position relative to the user's body in the XR environment. However, the virtual object can move within the device's field of view or the user's field of view in response to the user rotating their head. If the user turns around and their body is facing a second direction opposite to the first direction, the virtual object will be repositioned within the XR environment such that the virtual object is still displayed in front of and to the left of the user's body. When facing the second direction, regardless of whether the user rotates their head left and right (to look towards and away from the virtual object), the virtual object can remain in the same fixed position relative to the user's body in the XR environment.

[0026] In the examples mentioned above, even when the user's body rotates, the body-locked content is displayed at a fixed position / orientation relative to the user's body. For example, a virtual object can be displayed at a fixed distance in front of the user's body. If the user is facing north, the virtual object is at a fixed distance in front of (northward of) the user's body. If the user turns around and faces south, the virtual object is at a fixed distance in front of (southward of) the user's body.

[0027] Alternatively, the distance offset between the body-locked content and the user can be fixed relative to the user, while the orientation of the body-locked content can remain fixed relative to the physical environment. For example, when the user is facing north, a virtual object can be displayed at a fixed distance from the user in front of the user's body. If the user turns around and faces south, the virtual object remains at a fixed distance from the user north of the user's body.

[0028] The body-locked content is also configured to always maintain gravity or horizontal alignment, such that rolling orientation head and / or body changes will not cause the body-locked content to move within the XR environment. Translational movement can cause the body-locked content to be repositioned within the XR environment to maintain a fixed distance from the user. Subsequent descriptions of the body-locked content may include both of the above types of body-locked content.

[0029] In Figure 1 a schematic diagram of an exemplary head-mounted device is shown. As Figure 1 shown, the head-mounted device 10 (sometimes referred to as electronic device 10, system 10, head-mounted display 10, etc.) can have a control circuit 14. The control circuit 14 can be configured to perform operations in the head-mounted device 10 using hardware (e.g., dedicated hardware or circuitry), firmware, and / or software. Software code and other data for performing operations in the head-mounted device 10 are stored on a non-transitory computer-readable storage medium (e.g., a tangible computer-readable storage medium) in the control circuit 14. Software code is sometimes referred to as software, data, program instructions, instructions, or code. The non-transitory computer-readable storage medium (sometimes generally referred to as memory) can include non-volatile memory such as non-volatile random access memory (NVRAM), one or more hard disk drives (e.g., disk drives or solid-state drives), one or more removable flash drives, or other removable media, etc. The software stored on the non-transitory computer-readable storage medium can be executed on the processing circuitry of the control circuit 14. The processing circuitry can include an application-specific integrated circuit having processing circuitry, one or more microprocessors, digital signal processors, graphics processing units, central processing unit (CPU), or other processing circuitry.

[0030] The head-mounted device 10 may include an input-output circuit 20. The input-output circuit 20 may be used to allow the head-mounted device 10 to receive data from external equipment (such as a tethered computer, a portable device (such as a handheld device or a laptop computer), or other electrical equipment), and to allow a user to provide user input to the head-mounted device 10. The input-output circuit 20 may also be used to collect information about the environment in which the head-mounted device 10 operates. The output components in the circuit 20 may allow the head-mounted device 10 to provide output to the user and may be used to communicate with external electrical equipment.

[0031] As Figure 1 shown, the input-output circuit 20 may include a display, such as display 32. The display 32 may be used to display images for a user of the head-mounted device 10. The display 32 may be a transparent display such that the user may view physical objects through the display while overlaying computer-generated content over the physical objects by presenting computer-generated images on the display. The transparent display may be formed of an array of transparent pixels (e.g., a transparent organic light-emitting diode display panel) or may be formed of a display device that provides an image to the user through a beam splitter, a holographic coupler, or other optical coupler (e.g., a display device such as a liquid crystal on silicon display). Alternatively, the display 32 may be an opaque display that blocks light from physical objects when the user operates the head-mounted device 10. In this type of arrangement, a perspective camera may be used to display physical objects to the user. The perspective camera may capture images of the physical environment, and the physical environment images may be displayed on the display for the user to view. Additional computer-generated content (e.g., text, game content, other visual content, etc.) may optionally be superimposed over the physical environment images to provide an extended reality environment for the user. When the display 32 is opaque, the display may also optionally display the entire computer-generated content (e.g., without displaying an image of the physical environment).

[0032] The display 32 may include one or more optical systems (e.g., lenses) that allow a viewer to view the images on the display 16. A single display 32 may generate images for both eyes, or a pair of displays 16 may be used to display images. In a configuration having multiple displays (e.g., a left-eye display and a right-eye display), the focal length and position of the lenses may be selected such that any gap present between the displays will be invisible to the user (e.g., so that the images of the left and right displays overlap or merge seamlessly). A display module that generates different images for a user's left and right eyes may be referred to as a stereoscopic display. The stereoscopic display may be capable of presenting two-dimensional content (e.g., a user notification with text) and three-dimensional content (e.g., a simulation of a physical object such as a cube).

[0033] The input-output circuit 20 may include various other input-output devices for collecting data and user input and for providing output to the user. For example, the input-output circuit 20 may include one or more speakers 34 configured to play audio.

[0034] The input-output circuit 20 may include one or more cameras 36. The cameras 36 may include one or more outward-facing cameras (as an example, when the electronic device is mounted on the user's head, the camera faces the physical environment around the user). The cameras 36 may capture visible light images, infrared images, or any other desired type of images. If desired, the camera may be a stereo camera. The outward-facing camera may capture perspective video for the device 10. The camera 22 may also include an inward-facing camera (e.g., for gaze detection).

[0035] The input-output circuit 20 may include a gaze tracker 40 (sometimes referred to as a gaze tracking system or a gaze tracking camera). The gaze tracker 40 may be used to obtain gaze input from the user during operation of the head-mounted device 10.

[0036] The gaze tracker 40 may include a camera and / or other gaze tracking system components (e.g., a light source that emits a beam of light such that the reflection of the beam from the user's eye can be detected) to monitor the user's eyes. The gaze tracker 40 may face the user's eyes and may track the user's gaze. The camera in the gaze tracking system may determine the position of the user's eyes (e.g., the center of the user's pupil), may determine the direction in which the user's eyes are oriented (the direction of the user's gaze), may determine the vergence angle associated with the user's eyes, may determine the user's pupil size (e.g., so as to adjust the light modulation and / or other optical parameters based on the pupil size and / or adjust the sequential amount by which one or more of these parameters are adjusted spatially and / or the region in which one or more of these optical parameters are adjusted), may be used to monitor the current focus of the lens in the user's eye and / or other gaze information. The camera in the gaze tracking system may sometimes be referred to as an inward-facing camera, a gaze detection camera, an eye tracking camera, a gaze tracking camera, or an eye monitoring camera. If desired, other types of image sensors (e.g., infrared and / or visible light light-emitting diodes and light detectors, etc.) may also be used to monitor the user's gaze. The use of the gaze detection camera in the gaze tracker 40 is merely illustrative.

[0037] As Figure 1As shown, the input-output circuit 20 may include a position and motion sensor 38 (e.g., a compass, gyroscope, accelerometer, and / or other devices for monitoring the position, orientation, and movement of the electronic device 10, a satellite navigation system circuit such as a global positioning system circuit for monitoring the user's position, etc.). Using the sensor 38, for example, the control circuit 14 may monitor the current direction (e.g., the user's head pose) in which the user's head is oriented relative to the surrounding environment, the current position of the user's head relative to the surrounding environment, etc. The camera in the camera 36 may also be considered part of the position and motion sensor 38. The camera can be used for face tracking (e.g., by capturing images of the user's chin, mouth, etc. when the device is worn on the user's head), body tracking (e.g., by capturing images of the user's torso, arms, hands, legs, etc. when the device is worn on the user's head), and / or for positioning (e.g., using visual ranging, visual-inertial ranging, or other simultaneous localization and mapping (SLAM) techniques).

[0038] The input-output circuit 20 may include one or more depth sensors 42. Each depth sensor can be a pixelated depth sensor (e.g., configured to measure multiple depths across a physical environment) or a point sensor (configured to measure a single depth in a physical environment). Camera images (e.g., from one of the cameras 36) can also be used for monocular and / or stereo depth estimation. Each depth sensor (whether a pixelated depth sensor or a point sensor) can use phase detection (e.g., phase detection autofocus pixels) or light detection and ranging (LIDAR) to measure depth. Any combination of depth sensors can be used to determine the depth of physical objects in a physical environment.

[0039] The input-output circuit 20 may include a haptic output device 44. The haptic output device 44 may include actuators such as electromagnetic actuators, motors, piezoelectric actuators, electroactive polymer actuators, vibrators, linear actuators (e.g., linear resonant actuators), rotary actuators, actuators that bend a flexible member, etc. The haptic output device 44 can be controlled to provide any desired vibration pattern.

[0040] If desired, the input-output circuit 20 may further include other sensors and input-output components (e.g., an ambient light sensor, a force sensor, a temperature sensor, a touch sensor, a button, a capacitive proximity sensor, a light-based proximity sensor, other proximity sensors, a strain gauge, a gas sensor, a pressure sensor, a humidity sensor, a magnetic sensor, a microphone, a light-emitting diode, other light sources, wired and / or wireless communication circuits, etc.).

[0041] Users can sometimes use the position and motion sensors 38 to provide user input to the head-mounted device 10. Specifically, the position and motion sensors 38 can detect changes in head pose (sometimes referred to as head movement) during operation of the head-mounted device 10.

[0042] If desired, changes in yaw, roll, and / or pitch of the user's head (and accordingly, the head-mounted device) can all be interpreted as user input. Figures 2A to 2C Illustrates how yaw, roll, and pitch can be defined for the user's head. Figures 2A to 2C Illustrates the user 24. In Figures 2A to 2C In each figure, the user is facing the Z direction and the Y axis is aligned with the user's height. The X axis can be considered the left-right axis of the user's head, the Z axis can be considered the front-back axis of the user's head, and the Y axis can be considered the vertical axis of the user's head. The X axis can be referred to as extending from the user's left ear to the user's right ear, from the left side of the user's head to the right side of the user's head, etc. The Z axis can be referred to as extending from the back of the user's head to the front of the user's head (e.g., to the user's face). The Y axis can be referred to as extending from the bottom of the user's head to the top of the user's head.

[0043] As Figure 2A shown, yaw can be defined as rotation about the vertical axis (e.g., Figures 2A to 2C the Y axis in

[0044] As Figure 2B shown, roll can be defined as rotation about the front-back axis (e.g., Figures 2A to 2C the Z axis in

[0045] As Figure 2C shown, pitch can be defined as rotation about the left-right axis (e.g., Figures 2A to 2C the X axis in Figure 2CThe direction 30 in (following the right arrow) can be referred to as a downward head movement. Rotating upward about the left - right axis (e.g., pitch increasing, along Figure 2C The direction 30 in (following the left arrow) can be referred to as an upward head movement.

[0046] It should be understood that the position and motion sensor 38 can directly determine the pose, movement, yaw, pitch, roll, etc. of the head - mounted device 10. The position and motion sensor 38 can assume that the head - mounted device is mounted on the user's head. Thus, in this document, references to head pose, head movement, yaw of the user's head, pitch of the user's head, roll of the user's head, etc. can be considered interchangeable with references to device pose, device movement, yaw of the device, pitch of the device, roll of the device, etc.

[0047] At any given time, the position and motion sensor 38 (and / or the control circuit 14) can determine the yaw, roll, and pitch of the user's head. The yaw, roll, and pitch of the user's head can jointly define the orientation of the user's head pose.

[0048] Figure 3 is a top view of an exemplary arrangement of the head - mounted device 10. As Figure 3 shown, the head - mounted device 10 can include a support structure (e.g., see Figure 1 the support structure 62), which is used to house the components of the device 10 and mount the device 10 on the user's head. These support structures can include, for example, structures forming the outer shell walls and other structures for the main unit 62 - 2 (e.g., outer shell walls, lens module structures, etc.) and temple arms or other supplementary support structures, such as structures 62 - 1 that help keep the main unit 62 - 2 on the user's face.

[0049] The electronic device can include an optical module such as the optical module 70. The electronic device can include a left optical module and a right optical module corresponding to the user's left eye and right eye respectively. Figure 3 The optical module corresponding to the user's left eye is shown in.

[0050] Each optical module 70 includes a corresponding lens module 72 (sometimes referred to as a lens stack 72, lens 72, or adjustable lens 72) and a display 32. The lens 72 can include one or more lens elements arranged along a common axis. Each lens element can have any desired shape and can be formed of any desired material (e.g., having any desired refractive index). The individual lens elements can have unique shapes and refractive indices that, when combined, focus light (e.g., from the display or from the physical environment) in the desired manner. Each lens element of the lens module 72 can be formed of any desired material (e.g., glass, polymer materials such as polycarbonate or acrylic resin, crystals such as sapphire, etc.).

[0051] A positioning circuit such as locator 58 can be optionally used to position module 70 relative to the user's eyes and relative to some of the housing wall structures in the housing wall structure of main unit 26-2 independently. Locator 58 can include a stepper motor, a piezoelectric actuator, a motor, a linear electromagnetic actuator, and / or other electronic components for adjusting the position of the display, optical module 70, and / or lens module 72. During operation of device 10, locator 58 can be controlled by control circuit 14. For example, locator 58 can be used to adjust the spacing between modules 70 (and thus the lens-to-lens spacing between the left and right lenses of module 70) to match the interpupillary distance IPD of the user's eyes. In another example, the lens module can include an adjustable lens element. The curvature of the adjustable lens element can be adjusted in real time by locator 58 to compensate for the user's vision and / or viewing conditions.

[0052] During operation of the head-mounted device 10, the lens module 72 can be adjusted to change the focal length of the lens module. Consider Figure 4A and Figure 4B as examples. At a first time, as Figure 4A shown, the lens module 72 in the head-mounted device 10 can have an associated focal length F1. Virtual content can be presented by the display 32 at a depth D1 corresponding to the focal length F1 (e.g., D1 can be equal to F1 or within a threshold distance of F1). When the focal length F1 is equal to the depth D1, the virtual content is in focus for the viewer.

[0053] Subsequent changes in the user's gaze, changes in environmental conditions, changes in the distance to the nearest physical object, and / or other expected triggering factors can cause the depth of the presented virtual content to change. For example, it may be desirable to display virtual content at a depth D2 in Figure 4B . The focal length of the lens module 72 can be changed to a focal length F2 corresponding to the depth D2 (e.g., D2 can be equal to F2 or within a threshold distance of F2) such that the virtual content is in focus at the new depth.

[0054] During Figure 4A and Figure 4B , both the depth of the virtual content and the focal length of the lens module change. To maintain the focus of the virtual content throughout this change, the adjustment of the depth of the virtual content can be synchronized with the adjustment of the focal length of the lens module. Due to the physical limitations associated with lens module 72, the focal length of the lens module does not change immediately. Instead, the focal length of the lens module gradually changes over time at a given rate.

[0055] Figure 5A Illustrates an exemplary graph of the focal length over time for lens module 72. Figure 5ATwo profiles are shown: a linear profile 82 and a non-linear profile 84. Both profiles show how the focal length can vary from F1 to F2 over the duration between t1 and t2. This focal length change can follow a linear rate (as in profile 82) or a non-linear rate (as in profile 84).

[0056] To synchronize the change in the depth of the virtual content with the change in the focal length of the lens module, the depth of the virtual content can be changed based on the rate of change of the focal length. Figure 5B An exemplary graph of the depth of the virtual content over time is shown. Figure 5B Two profiles are shown: a linear profile 86 and a non-linear profile 88. Both profiles show how the depth of the virtual content can vary from D1 to D2 over the duration between t1 and t2. This change in the depth of the virtual content can follow a linear rate (as in profile 86) or a non-linear rate (as in profile 88).

[0057] The control circuit 14 can select a profile for the change in the depth of the virtual content that matches or approximately matches the profile used for the change in the focal length. The profile used for the change in the focal length can be determined using the calibration of the lens module 72 and stored in the control circuit 14. Alternatively or additionally, the profile used for the change in the focal length can be estimated in real time based on the current operating conditions of the head-mounted device 10. Once the control circuit 14 has the predicted profile for the change in the focal length, the control circuit 14 can select a profile for the change in the depth of the virtual content based on the profile for the change in the focal length.

[0058] Consider an example in which the control circuit 14 predicts that the change in the focal length will follow the linear profile 82. In this case, the control circuit 14 can select the linear profile 86 for the change in the depth of the virtual content. The focal length and the depth of the virtual content can be updated according to the linear profile 82 and the linear profile 86 respectively between t1 and t2. Since the profiles 82 and 86 are the same, the depth of the virtual content can remain in focus throughout the change from D1 to D2.

[0059] Consider an example in which the control circuit 14 predicts that the change in the focal length will follow the non-linear profile 84. In this case, the control circuit 14 can select the non-linear profile 88 for the change in the depth of the virtual content. The focal length and the depth of the virtual content can be updated according to the non-linear profile 84 and the non-linear profile 88 respectively between t1 and t2. Since the profiles 84 and 88 are the same, the depth of the virtual content can remain in focus throughout the change from D1 to D2.

[0060] Note that the virtual content depth profile may not be the same as the focal length profile. The control circuit 14 may always use a linear profile for the virtual content depth change (regardless of whether the focal length follows a linear profile or a non-linear profile). For example, the linear profile 86 may be used for the virtual content depth change, while the lens module 72 follows the non-linear profile 84. This type of scheme can eliminate the complications associated with determining the exact profile of the focal length change. As an alternative to determining the exact focal length profile, the control circuit 14 may determine the total duration of the change in the focal length (e.g., t2 - t1) and change the virtual content depth at a constant (linear) rate within the total duration (between t1 and t2).

[0061] Figure 6A and Figure 6B illustrates an example of changing the virtual content depth on the display 32. As Figure 6A shown, there may be a first display 32L that presents an image to the user's left eye and a second display 32R that presents an image to the user's right eye. To present the virtual object 90, a first image 90L is presented on the display 32L, and a second image 90R is presented on the display 32R. The user's eyes will view the images 90L and 90R and perceive a single virtual object 90 at the depth D1.

[0062] To change the virtual content depth from D1 to D2, these images can be moved closer together on the displays 32L / 32R. As Figure 6A shown, the image 90L is laterally shifted in the direction 92 (e.g., towards the display 32R), while the image 90R is laterally shifted in the direction 94 (e.g., towards the display 32L). Laterally shifting the images closer together (as Figure 6A in) will decrease the depth of the virtual content, while laterally shifting the images further apart (e.g., Figure 6A the opposite direction) will increase the depth of the virtual content.

[0063] Figure 6B illustrates the new positions of the images 90L and 90R on the displays 32L and 32R after the depth of the virtual object has been changed to D2. Figure 6B Also illustrated is how the sizes of the images 90L and 90R can be changed during the change in the depth of the virtual object. If the images 90R and 90L remain the same size on the displays 32L / 32R, the perceived size of the virtual object 90 will decrease as the depth of the virtual object decreases and increase as the depth of the virtual object increases. To mitigate this effect, the sizes of the images 90L / 90R on the displays 32L / 32R can be increased when the depth of the virtual object decreases and decreased when the depth of the virtual object increases. Figure 6B illustrates that the size of the image 90L (when the depth is D2) is larger thanFigure 6A the size of the image 90L in Figure 6A (when the depth is D1) and the size of the image 90R in Figure 6A (when the depth is D2) are greater than Figure 6A An example of the size of the image 90R in Figure 6A (when the depth is D1).

[0064] Figure 7 is a flowchart showing an exemplary method for operating a head-mounted device. Figure 7 The method of Figure 7 can be executed by the control circuit 14 of the head-mounted device 10.

[0065] During the operation of block 102, the control circuit 14 can use the display 32 to display a virtual object at a first depth while one or more lenses (e.g., the lens module 72) have a focal length equal to the first depth.

[0066] For example, during the operation of block 102, the head-mounted device 10 can display a virtual object at D1 while the lens module 72 has a focal length F1 equal to D1 (as Figure 4A shown).

[0067] During the operation of block 104, the control circuit 14 can adjust the focal length of the one or more lenses from the first depth to a second depth different from the first depth at a first rate. The adjustment rate of the one or more lenses can be linear (as Figure 5A shown by the profile 82 in Figure 5A ) or non-linear (as Figure 5B shown by the profile 84 in Figure 5B ).

[0068] Continuing the foregoing example, the focal length of the lens module 72 can be adjusted from F1 to F2 during the operation of block 104. The adjustment rate from F1 to F2 can be linear (as Figure 5A shown by the profile 82 in Figure 5A ) or non-linear (as Figure 5B shown by the profile 84 in Figure 5B ).

[0069] During the operation of block 106, the control circuit 14 can shift the virtual object from the first depth toward the second depth at a second rate based on the first rate from block 104 while displaying the virtual object. The adjustment rate of the depth of the virtual object can be linear (as Figure 5B shown by the profile 86 in Figure 5B ) or non-linear (as Figure 5B shown by the profile 88 in Figure 5B ). As an example, the second rate can be selected to be equal to the first rate. As another example, the second rate can be selected such that the total duration of the depth change is equal to the linear rate of the total duration of the focal length change.

[0070] As Figure 7As shown, the operation of block 106 may include shifting the first image and the second image horizontally on the respective first display and second display (during the operation of block 108) and / or changing the size of the first image and the size of the second image on the respective first display and second display (during the operation of block 110). Shifting the first image and the second image horizontally on the respective first display and second display may cause a change in the perceived depth of the image. Shifting the first image and the second image closer together may reduce the depth, while shifting the first image and the second image further apart may increase the depth. Changing the size of the first image and the second image may mitigate the change in the perceived size of the virtual object when adjusting the depth of the virtual object.

[0071] Continuing the foregoing example, during the operation of block 106, the depth of the displayed virtual object may shift from a first depth (D1) from block 102 towards a second depth (D2) equal to the second focal length F2 (see Figure 4B ). The depth of the virtual object may be adjusted from D1 to D2 between t1 and t2, as Figure 5B shown. The adjustment rate between D1 and D2 may be linear (as shown by profile 86 in Figure 5B ) or non-linear (as shown by profile 88 in Figure 5B ). Figure 6A and Figure 6B show how the images 90L / 90R of the virtual object may be shifted closer together such that the depth of the virtual object changes from D1 to D2. Figure 6B shows how the images 90L / 90R may have a larger size on the displays 32L / 32R when presented at the depth D2 compared to when presented at D1 in Figure 6A .

[0072] For certain types of content such as head-locked content, it may be desirable to present the content at a depth that matches the focal length of the user's eyes. For example, if the user is focusing at a near depth (e.g., 30 centimeters), the head-locked content may be presented at that near depth, while if the user is focusing at a far depth (e.g., 2 meters), the head-locked content may be presented at that far depth. Selecting the depth of the head-locked content based on the focal length of the user's eyes may be beneficial for providing a comfortable user experience. However, determining the focal length of the user's eyes in real time may be challenging.

[0073] One way to determine the focal length of the user's eyes in real time is to use the vergence angle. The vergence angle is the angle between the gaze directions of the user's left and right eyes. Figure 8A and Figure 8B show two different examples of the vergence angle. As Figure 8A and Figure 8BAs shown, the left eye 112L has a corresponding gaze direction 114L, and the right eye 112R has a corresponding gaze direction 114R. In Figure 8A , the gaze direction defines a vergence angle A1 associated with a first depth D1. In Figure 8B , the gaze direction defines a vergence angle A2 associated with a second depth D2. The focusing depth of the user's eyes is negatively correlated with the vergence angle. In other words, a smaller vergence angle has a larger corresponding depth, while a larger vergence angle has a smaller corresponding depth. This is shown in Figure 8A and Figure 8B , where A1 < A2 and D1 > D2.

[0074] The gaze tracking sensor 40 can measure the vergence angle of the user's eyes in real time during the operation of the head-mounted device 10. The measured vergence angle can be used to calculate the depth at which the user is focusing their eyes. The calculated depth at which the user is focusing their eyes can be used to select the depth of virtual content or for other desired purposes within the head-mounted device 10.

[0075] In an exemplary arrangement, the vergence angle of the user's eyes can be continuously measured, and the depth of the virtual content can be continuously updated to match the depth indicated by the measured vergence angle. In other words, the depth of the virtual content is continuously updated to "chase" the depth at which the user's eyes are focused (as indicated by the vergence angle). One problem that can arise with this type of scheme is a cyclic vergence estimation error caused by errors in the vergence angle measurements made by the gaze tracking sensor 40. This phenomenon is illustrated in Figure 9 .

[0076] Figure 9 shows an example of presenting virtual content at a depth D1 at a first time (t1). At the first time, the actual vergence angle of the user's eyes can be equal to V 1_A . The vergence angle V 1_A has a corresponding depth equal to D1. In other words, the user is focusing on the virtual content at depth D1 at t1, such that the virtual content appears clear. Ideally, since the actual focal length of the user's eyes matches the depth of the virtual content, the virtual content should remain at D1. However, errors in the measurements of the gaze tracking sensor 40 can cause a cyclic shift in the depth of the virtual content.

[0077] Figure 9 shows that the measurement from the gaze tracking sensor 40 results in a value less than V 1_AAn example of the measured vergence V1. In other words, the gaze tracking sensor indicates that the user is focusing at a depth further away than the depth at which he / she is actually focusing. Thus, the vergence angle measurement at t1 indicates a mismatch between the focusing depth of the user's eyes and the depth of the virtual content. The depth of the virtual content is updated from D1 to D2 to mitigate the measured mismatch and match the measured focusing depth associated with the vergence angle V1. However, this causes the user to change the focus of his / her eyes to a further depth (associated with the increased depth of the virtual content). Thus, at t2, when the virtual content is at depth D2, the gaze tracking sensor measures a change in vergence from V1 to V2 (less than V1). The vergence angle measurement at t2 indicates a mismatch between the focusing depth of the user's eyes and the depth of the virtual content. Thus, the depth of the virtual content is updated from D2 to D3 to match the measured focusing depth associated with the vergence angle V2. However, this causes the user to change the focus of his / her eyes to a further distance (associated with the increased depth of the virtual content). Thus, at t3, when the virtual content is at depth D3, the gaze tracking sensor measures a change in vergence from V2 to V3 (less than V2). Thus, the root cause of this cyclic error is that a change in the depth of the virtual content can motivate a change in the vergence angle of the user's eyes, a change in the vergence angle of the user's eyes can motivate a change in the depth of the virtual content, a change in the depth of the virtual content can motivate a change in the vergence angle of the user's eyes, and so on.

[0078] To prevent this type of cyclic error, once the depth of the virtual content approximately matches the vergence angle of the user's eyes, the depth of the virtual content can be locked. Once the depth of the virtual content is locked, the depth of the virtual content can be kept fixed until an unlocking criterion is met. A small change in the vergence angle may not meet the unlocking criterion. Thus, even when the vergence angle measurement indicates a small change in the vergence angle, the depth of the virtual content remains fixed. A large change in the vergence angle may meet the unlocking criterion. Once the unlocking criterion is met, the depth of the virtual content is adjusted until the depth of the virtual content approximately matches the vergence angle of the user's eyes, at which point the depth of the virtual content will be locked again.

[0079] Figures 10A to 10D An example of this type of locking and unlocking scheme for the depth of virtual content is shown. Figure 10A An example is shown where the depth of the virtual object (D1) is approximately equal to the depth corresponding to the measured vergence angle V1. Thus, the depth of the virtual object is locked at Figure 10A . Thus, the depth of the virtual object remains fixed at D1 until the unlocking criterion is met. An example of the unlocking criterion (used in Figures 10A to 10D ) is to compare the change in the measured vergence angle with a threshold magnitude. In other words, as long as the vergence angle stays within the threshold range V 范围1 , the virtual object can remain locked at D1. As shown in Figure 10AAs shown, the vergence angle range V 范围1 can be equal to the vergence angle associated with D1 + / - a given vergence angle magnitude.

[0080] Figure 10B An example is shown where the vergence angle has changed to V2 which is greater than V1 (indicating that the user is focusing at a depth closer than D1). However, V2 is still within the vergence range V 范围1 . Therefore, the unlock criteria have not been met and the virtual content remains fixed at D1.

[0081] Figure 10C An example is shown where the vergence angle has changed to V3 which is greater than V2 (indicating that the user is focusing at a depth closer than D1). V3 is outside the vergence range V 范围1 . Therefore, in Figure 10C , the unlock criteria have been met and the virtual content is unlocked. Once the virtual content is unlocked, the virtual content can start to shift in direction 116 towards the depth corresponding to V3.

[0082] Figure 10D An example is shown where the depth of the virtual content reaches a depth D3 that is close to (but not equal to) the depth corresponding to the real-time vergence angle V3. Even though D3 is not equal to the depth corresponding to the vergence angle V3, the two depths can be close enough such that in Figure 10D the depth of the virtual content is locked. Once the virtual content is locked, a new vergence range V 范围3 centered on V3 is established. The new unlock criteria can involve comparing the real-time vergence angle with V 范围3 .

[0083] As a specific example mapped to Figures 10A to 10D , consider an example where the vergence angle V1 is equal to 3.0 degrees and the depth D1 is equal to 1.00 meter. 1.00 meter can be equal to the focusing depth associated with a 3.0 degree vergence angle. The threshold for the vergence angle change used to define V 范围1 can be equal to 0.5 degrees. Therefore, V Figure 10A in 范围1 is equal to between 2.5 degrees and 3.5 degrees.

[0084] In Figure 10B , the vergence angle V2 is equal to 3.3 degrees. Since 3.3 degrees is within V 范围1 (between 2.5 degrees and 3.5 degrees), the depth of the content remains fixed at 1.00 meter (D1).

[0085] In Figure 10C , the vergence angle V3 is equal to 3.7 degrees. Since 3.7 degrees is outside V 范围1Outside (between 2.5 degrees and 3.5 degrees), the virtual content is unlocked and the depth of the content is reduced. The vergence angle V3 can have an associated depth of 0.85 meters. Thus, in Figure 10C the depth of the content begins to shift from 1.00 meters towards 0.85 meters.

[0086] In Figure 10D the depth of the content can reach a depth D3 equal to 0.9 meters. Although D3 is not exactly equal to the depth 0.85 meters associated with V3, these two depths can be close enough to lock the virtual content at D3. Figure 10D The new V in 范围3 is equal to V3 + / - 0.5 degrees (e.g., between 3.2 degrees and 4.2 degrees).

[0087] Thus, Figures 10A to 10D shows an example where the measured change in vergence angle is used as an unlocking criterion. This example is for illustrative purposes only. Alternatively or additionally, other factors such as head rotation and / or head position can be used as unlocking criteria.

[0088] Figure 11A and Figure 11B show how head rotation can be used as an unlocking criterion. Head rotation can be an indicator that the user has turned to face a new physical object (which may thus be at a different depth from the initial physical object being viewed). For example, the user may be viewing a laptop monitor and rotates their head to view another physical object (e.g., a clock on the wall) at a different depth from the laptop monitor. Considering these types of situations, it may be desirable for head rotation to be used as an unlocking criterion.

[0089] Figure 11A shows an example where the user's head 122 faces direction 118A. Then, the user rotates their head towards direction 120. Figure 11B shows that after rotating towards direction 120, the user's head faces direction 118B. The head rotation causes a change in the yaw of the user's head by an angle A1, as Figure 11B shown.

[0090] A rotational change exceeding a certain threshold can be recognized as an unlocking criterion for virtual content. Alternatively or additionally, the angular acceleration of the user's head can be recognized as an unlocking criterion for virtual content. The threshold for the rotational change used as an unlocking criterion can be equal to 3 degrees or more, equal to 5 degrees or more, equal to 10 degrees or more, equal to 20 degrees or more, etc. The threshold for the angular acceleration used as an unlocking criterion can be equal to 5 degrees / second or more, equal to 10 degrees / second or more, equal to 15 degrees / second or more, equal to 20 degrees / second or more, etc.

[0091] The above-described head rotation unlocking criteria can be applied to yaw, roll, and / or pitch. In other words, one or more of yaw, roll, and pitch can be analyzed separately to identify changes or accelerations that meet the unlocking criteria.

[0092] Figure 12A and Figure 12B illustrates how head position can be used as an unlocking criterion. In contrast to the above-described head rotation (characterized by rotation about Figures 2A to 2C the X, Y, and Z axes), head position can refer to the position of the head in three-dimensional space along the X, Y, and Z axes (independent of rotation). FIG. 12 shows an example where the user's head 122 is separated from the physical object 124 by a distance 126A. The user then moves their head in the direction 128 (without any head rotation). Figure 12B illustrates the user's head position after moving in the direction 128. This head movement causes a displacement of the total displacement 130 in the position of the user's head. The user's head is now separated from the physical object 124 by a distance 126B, which is less than Figure 12A the distance 126A in

[0093] A change in head position that exceeds a certain threshold can be recognized as an unlocking criterion for virtual content. Alternatively or in addition, the acceleration of the user's head can be recognized as an unlocking criterion for virtual content. The threshold for the change in head position used as an unlocking criterion can be equal to 10 cm or greater, equal to 30 cm or greater, equal to 50 cm or greater, equal to 100 cm or greater, and so on. The threshold for the acceleration used as an unlocking criterion can be equal to 10 cm / sec or greater, equal to 30 cm / sec or greater, equal to 50 cm / sec or greater, equal to 100 cm / sec or greater, and so on. The above-described head position unlocking criterion can be applied to position changes in any direction (e.g., position changes along the X axis, position changes along the Y axis, position changes along the Z axis, position changes along any vector between the axes, etc.).

[0094] The vergence angle measurements obtained by the gaze tracking sensor 40 may be relatively noisy. To reduce the noise in the vergence angle data, the vergence angle data from the gaze tracking sensor 40 can be filtered. Figure 13 is a diagram illustrating how the control circuit 14 can include a filtering circuit 132 in addition to the depth selection circuit 134. The gaze tracking sensor 40 can provide raw data to the filtering circuit 132. The gaze tracking sensor can output data at a frequency greater than 4 Hz, greater than 10 Hz, greater than 20 Hz, less than 30 Hz, less than 20 Hz, a frequency between 2 Hz and 20 Hz, a frequency between 3 Hz and 12 Hz, and so on.

[0095] The filter circuit 132 can filter the raw data from the gaze tracking sensor 40 and provide the filtered data to the depth selection circuit 134. The filter circuit 132 can filter the data from the gaze tracking sensor 10 in a variety of ways. As an example, the filter circuit 132 can perform a time average on the data over a certain duration. This duration can have any desired magnitude (e.g., at least 0.4 seconds, at least 0.7 seconds, at least 1.0 seconds, at least 2.0 seconds, etc.). One or more outliers can also be removed from the time average window before performing the averaging. At least one maximum vergence angle measurement from the time window can be discarded and / or at least one minimum vergence angle measurement from the time window can be discarded. In a specific example, the largest 25% of the vergence angle measurements from the time window can be discarded, and the smallest 25% of the vergence angle measurements from the time window can be discarded. In other words, only the middle 50% of the vergence angle measurements from the time window are averaged.

[0096] Consider an example where the gaze tracking sensor 40 outputs raw data at 8 Hz and the filter circuit 132 uses a 1 - second time average window. The filter circuit 132 can receive raw data indicating that the vergence angle measurements in the previous second were: 4.1 degrees, 3.2 degrees, 3.6 degrees, 3.0 degrees, 3.1 degrees, 4.0 degrees, 3.7 degrees, and 2.9 degrees. The highest 25% of these values (e.g., 4.1 degrees and 4.0 degrees) can be discarded by the filter circuit 132 before performing the time average. The lowest 25% of these values (e.g., 2.9 degrees and 3.0 degrees) can be discarded by the filter circuit 132 before performing the time average. The remaining data (e.g., the middle 50% of these values) can be averaged to obtain a filtered vergence measurement of 3.4 degrees ((3.2 + 3.6 + 3.1 + 3.7) / 4 = 3.4).

[0097] This process can be repeated after the filtering circuit 132 receives each new piece of raw data. Continuing with the above example, the filtering circuit 132 may receive a new vergence angle measurement of 3.0 degrees from the gaze tracking sensor 40. The new measurement (3.0 degrees) replaces the oldest measurement (4.1 degrees) from the previous data set. Thus, the new vergence angle data set for the previous second is: 3.2 degrees, 3.6 degrees, 3.0 degrees, 3.1 degrees, 4.0 degrees, 3.7 degrees, 2.9 degrees, 3.0 degrees. The highest 25% of these values (e.g., 4.0 degrees and 3.7 degrees) may be discarded by the filtering circuit 132 before performing the temporal averaging. The lowest 25% of these values (e.g., 2.9 degrees and 3.0 degrees) may be discarded by the filtering circuit 132 before performing the temporal averaging. The remaining data (e.g., the middle 50% of these values) may be averaged to obtain a filtered vergence measurement of 3.225 degrees ((3.2 + 3.6 + 3.1 + 3.0) / 4 = 3.225).

[0098] Thus, the filtering circuit 132 may output the filtered vergence data to the depth selection circuit 134 at the same frequency as the frequency at which the raw vergence data is received. In addition to the filtered vergence data, the depth selection circuit 134 may also use depth data from the depth sensor 42 and position and motion data from the sensor 38 to select the depth of the virtual content presented on the display 32.

[0099] When the filtered vergence data is stable (e.g., when the difference between the maximum filtered vergence measurement and the minimum filtered vergence measurement within a given duration is less than a threshold) and when the depth of the virtual content is close to the distance associated with the most recent filtered vergence angle measurement (e.g., within its threshold), the depth selection circuit 134 may lock the depth of the virtual content. The specific values of the duration and threshold mentioned above can be adjusted to optimize the user experience.

[0100] As a specific example, the duration during which the filtered vergence data is examined to evaluate whether the vergence measurement is stable can be at least 0.1 second, at least 0.3 second, at least 0.5 second, at least 1.0 second, at least 2.0 seconds, etc. The threshold for the difference between the maximum filtered vergence measurement and the minimum filtered vergence measurement within this duration can be less than 2 degrees, less than 1.5 degrees, less than 1.0 degree, less than 0.8 degree, less than 0.6 degree, less than 0.4 degree, less than 0.2 degree, etc. The threshold for comparing the depth of the virtual content with the distance associated with the most recent filtered vergence measurement can be less than 20 centimeters, less than 10 centimeters, less than 5 centimeters, less than 3 centimeters, less than 2 centimeters, etc.

[0101] The example of converting the vergence angle measurement result into depth and using a threshold in units of distance to evaluate the "closeness" of the depth of the virtual content to the filtered vergence angle measurement result is merely illustrative. In another possible arrangement, the depth of the virtual content can be converted into a corresponding vergence angle and a threshold in degrees can be used to evaluate the "closeness" of the depth of the virtual content to the filtered vergence angle measurement result. In this case, the threshold can be less than 3 degrees, less than 2 degrees, less than 1 degree, less than 0.8 degree, less than 0.6 degree, less than 0.4 degree, less than 0.2 degree, less than 0.1 degree, etc.

[0102] The depth selection circuit 134 can unlock the depth of the virtual content based on the change in the filtered vergence data, based on the depth data from the depth sensor 42, based on the head position data from the sensor 38, and / or based on the head rotation data from the sensor 38.

[0103] Possible unlocking criteria include that the change in the filtered vergence angle (e.g., in degrees) is greater than a threshold, the rate of change of the filtered vergence angle (e.g., in degrees / second) is greater than a threshold, the change in head rotation (e.g., in degrees) is greater than a threshold, the rate of change of head rotation (e.g., in degrees / second) is greater than a threshold, the change in head position (e.g., in centimeters) is greater than a threshold, and / or the rate of change of head position (e.g., in centimeters / second) is greater than a threshold.

[0104] After the depth of the virtual content is unlocked, the depth selection circuit 134 can shift the depth of the virtual content towards the depth associated with the most recent filtered vergence angle measurement result until a locking criterion is met. The rate at which the depth of the virtual content is updated can be capped at a maximum value to avoid the appearance of content that "jumps" between depths. Alternatively or in addition, the depth of the virtual content can be updated at a rate based on the rate at which one or more lenses can be adjusted as discussed above in Figures 3 to 7 The depth data from the depth sensor 42 can indicate the depth of one or more physical objects in the user's physical environment. The depth of the physical object can be used to determine the target depth of the virtual content.

[0105] The depth data from the depth sensor 42 can indicate the depth of one or more physical objects in the user's physical environment. The depth of the physical object can be used to determine the target depth of the virtual content.

[0106] Figure 14A and Figure 14B are flowcharts showing an exemplary method for operating the head-mounted device 10. During the operation of block 202, the control circuit 14 can display a virtual object at a first depth on one or more displays 32. During the operation of block 204, the control circuit 14 can use a first subgroup of device sensors (e.g., the gaze tracking sensor 40) to determine the eye focus depth of the user. The eye focus depth can be different from the first depth from block 202.

[0107] To determine the eye focus depth, the control circuit 14 may filter data from the gaze tracking sensor during the operation of block 206. Filtering the data may include time averaging the data (as shown in block 208), removing one or more outliers from the data (as shown in block 210), and / or any other desired filtering operations. As a specific example, removing one or more outliers may include removing the highest 25% of the measured vergence angles from the data set used for time averaging and removing the lowest 25% of the measured vergence angles from the data set used for time averaging.

[0108] During the operation of block 212, the control circuit 14 may shift the virtual object from a first depth toward the eye focus depth while displaying the virtual object. The control circuit may cap the rate at which the depth of the virtual content is changed at a maximum rate. Thus, the virtual content effectively catches up with the current eye focus depth during the operation of block 212.

[0109] During the operation of block 214, based on determining that a first criterion (sometimes referred to as a locking criterion) associated with the current depth of the virtual object relative to the eye focus depth is satisfied, the virtual object may be locked at the current depth. The criterion may be that the depth is within a threshold distance (e.g., within 3 cm, within 10 cm, within 20 cm, within 50 cm, etc.) or within a threshold percentage (e.g., the current depth is within 10% of the eye focus depth, the current depth is within 5% of the eye focus depth, etc.). Alternatively or in addition, the first criterion may be checked in the angular domain. The vergence angle associated with the current eye focus depth may be compared with the vergence angle associated with the current depth of the virtual object. The criterion may be that the vergence angle is within a threshold amount (e.g., within 0.1 degrees, within 0.3 degrees, within 0.5 degrees, within 1 degree, etc.) or within a threshold percentage (e.g., 10%, 5%, etc.).

[0110] The criterion for block 214 may also include that the eye focus depth is relatively stable (e.g., the eye focus depth has remained within the target range for a target duration).

[0111] After locking the virtual content in block 214, the depth of the virtual content will no longer be shifted toward the current eye focus depth until the unlocking criterion is satisfied next.

[0112] During the operation of block 216 (see Figure 14B ), the control circuit 14 may determine whether a second criterion is satisfied. The second criterion may sometimes be referred to as an unlocking criterion. As an example, multiple unlocking criteria may be considered based on one or more parameters such as vergence angle, head position, head rotation, etc.

[0113] The unlocking criteria may include comparing a first vergence angle associated with the current eye focusing depth with a second vergence angle associated with the current depth of the virtual object (as shown in block 218). If the difference between the first vergence angle and the second vergence angle is greater than a threshold amount (e.g., 0.1 degrees, within 0.3 degrees, within 0.5 degrees, within 1 degree, etc.), the unlocking criteria may be satisfied. If the difference between the first vergence angle and the second vergence angle is less than the threshold amount, the unlocking criteria are not satisfied. Alternatively or additionally, the unlocking criteria may be checked in the distance domain (e.g., the current eye focusing depth may be compared with the current depth of the virtual object).

[0114] Other possible unlocking criteria include determining whether a change in the user's head position is greater than a threshold (as shown in block 220), determining whether a change in the user's head position acceleration is greater than a threshold (as shown in block 222), determining whether a change in the user's head rotation is greater than a threshold (as shown in block 224), and determining whether a change in the user's head rotation acceleration is greater than a threshold (as shown in block 226).

[0115] The head position in blocks 220 and 222 refers to the position of the user's head in three-dimensional space (regardless of the yaw, pitch, and roll of the user's head). Using the head position as an unlocking criteria as in blocks 220 and 220 may include using the change in the head position in any direction. The head rotation in blocks 224 and 226 refers to the change in the yaw, pitch, and roll of the user's head. Using the head rotation as an unlocking criteria as in blocks 224 and 226 may include using the change in one or more of the yaw, pitch, and roll.

[0116] Determining whether the unlocking criteria are satisfied may also include using information from a depth sensor such as depth sensor 42. The depth sensor may measure the depth of one or more physical objects in the physical environment, which may affect when the virtual content is locked and / or unlocked and / or the selected depth of the virtual content when the virtual content is locked and / or unlocked.

[0117] It should be understood that any possible unlocking conditions described herein may be used. When considering multiple criteria, the virtual content depth may be unlocked whenever any single criterion is satisfied, whenever any two criteria are satisfied, whenever any three criteria are satisfied, whenever all criteria are satisfied, etc.

[0118] During the operation of block 230, based on determining that the second criterion is not satisfied, the control circuit may continue to display the virtual content at the current depth (e.g., the locked depth), even when the fixation tracking sensor indicates a change in the vergence angle.

[0119] During operation of block 232, upon determining that a second criterion is met, the control circuit may unlock the virtual content depth and shift the virtual content from the current depth towards an updated eye focus depth indicated by the gaze tracking sensor. The virtual content depth may continuously chase the real-time eye focus depth until the locking criterion is met again (e.g., as in block 214).

[0120] As an example, during operation of block 202, a virtual object may be displayed at 1.0 meter. During operation of block 204, the gaze tracking sensor 40 may measure a filtered vergence angle associated with a depth of 0.8 meter. During operation of block 212, the virtual object may be shifted from 1.0 meter towards the real-time eye focus depth (0.8 meter). During operation of block 214, the control circuit may determine that the virtual object depth (e.g., 0.82 meter) is within a threshold distance of the current eye focus depth (0.8 meter) and that the eye focus depth is stable. Accordingly, the virtual object depth is locked at the current depth of 0.82 meter. During operation of block 216, one or more unlock criteria may be analyzed. The virtual object depth may be unlocked when the difference between the current vergence angle of the user's eyes and the vergence angle associated with the locked content depth is greater than a threshold amount (as in block 218), when the user's head position changes by more than a threshold amount (as in block 220), when the user's head position acceleration is greater than a threshold amount (as in block 222), when the user's roll, pitch, and / or yaw changes are greater than a threshold amount (as in block 224), and / or when the acceleration of the roll, pitch, and / or yaw is greater than a threshold amount (as in block 226). If desired, depth sensor information may also be used to unlock the virtual content depth (as in block 228).

[0121] During operation of block 230, when the unlock criteria are not met, the virtual object remains at the current depth (0.82 meter) even when the eye vergence changes (e.g., changes to a vergence angle associated with a depth of 0.78 meter). During operation of block 232, when the unlock criteria are met, the virtual object depth is unlocked and the virtual object is shifted from the current depth (0.82 meter) towards an updated eye focus depth (e.g., 0.75 meter) associated with the real-time vergence angle measurement from the gaze tracking sensor 40.

[0122] As described above, one aspect of the technology of the present invention is to collect and use information, such as sensor information. The present disclosure contemplates that, in some cases, data including personal information data may be collected, and such personal information data uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, user names, passwords, biometric information, or any other identifying or personal information.

[0123] The present disclosure recognizes that the use of such personal information in the technology of the present invention can be used to benefit users. For example, such personal information data can be used to deliver targeted content that the user is more interested in. Thus, the use of such personal information data enables the user to have control over the delivered content. In addition, the present disclosure also contemplates other uses of personal information data that are beneficial to users. For example, health and fitness data can be used to provide insights into the user's overall health status, or can be used as positive feedback for individuals who use technology to pursue health goals.

[0124] The present disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with sound privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to the use of privacy policies and measures that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable entity purposes and should not be shared or sold outside of these legitimate purposes. Additionally, such collection / sharing should occur after receiving informed consent from the user. Further, such entities should consider taking any necessary steps to protect and secure access to such personal information data and ensure that other entities with access to personal information data comply with their privacy policies and procedures. Additionally, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and measures. Further, the policies and measures should apply to the collection and / or access of specific types of personal information data and to applicable laws and standards that include considerations of special jurisdictions. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Thus, different privacy measures should be asserted for different types of personal data in each country.

[0125] Notwithstanding the foregoing, the present disclosure also anticipates embodiments where users selectively block the use or access of personal information data. That is, the present disclosure anticipates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, the inventive technology may be configured to allow users to select to "opt-in" or "opt-out" of the collection of personal information data at any time during or after registering for a service. In another example, a user may choose not to provide certain types of user data. In yet another example, a user may choose to limit the length of time that user-specific data is maintained. In addition to providing "opt-in" and "opt-out" options, the present disclosure also anticipates providing notifications related to the access or use of personal information. For example, a user may be notified when downloading an application ("app") that their personal information data will be accessed and then reminded again before the personal information data is accessed by the app.

[0126] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by restricting data collection and deleting the data. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect user privacy. In appropriate cases, de-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or characteristics of the data stored (e.g., collecting location data at the city level rather than the address level), controlling the way the data is stored (e.g., aggregating data among users), and / or other methods.

[0127] Thus, while the present disclosure broadly covers the use of information that may include personal information data to implement one or more of the various disclosed embodiments, the present disclosure also anticipates that the various embodiments may also be implemented without access to personal information data. That is, the various embodiments of the inventive technology will not fail to function properly due to the lack of all or a portion of such personal information data.

[0128] According to an embodiment, an electronic device is provided that includes one or more sensors, one or more displays, one or more processors, and a memory that stores instructions configured to be executed by the one or more processors, the instructions for: displaying a virtual object at a first depth via the one or more displays; using a first subgroup of the one or more sensors, determining that an eye focus depth is different from the first depth; while displaying the virtual object, causing the virtual object to shift from the first depth toward the eye focus depth; and locking the virtual object at the current depth based on determining that a first criterion associated with the current depth of the virtual object relative to the eye focus depth is satisfied.

[0129] According to another embodiment, the instructions optionally include instructions for: after locking the virtual object at the current depth, determining whether a second criterion is satisfied; and based on determining that the second criterion is not satisfied, continuing to display the virtual object at the current depth despite the first subgroup of the one or more sensors indicating a change in the eye focus depth.

[0130] According to another embodiment, the instructions optionally include instructions for: based on determining that the second criterion is satisfied, causing the virtual object to shift from the current depth toward an updated eye focus depth indicated by the first subgroup of the one or more sensors.

[0131] According to another embodiment, determining whether the second criterion is satisfied optionally includes using the first subgroup of the one or more sensors or using a second subgroup of the one or more sensors to determine whether the second criterion is satisfied.

[0132] According to another embodiment, displaying a virtual object at a first depth, determining that an eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at a current depth optionally include: displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth according to determining that the virtual object is a first type of virtual object having a position defined relative to a position corresponding to the electronic device or a user of the electronic device.

[0133] According to another embodiment, the instructions optionally include instructions for: receiving data from a first subgroup of the one or more sensors; and filtering the data to obtain filtered data, determining that the eye focus depth is different from the first depth optionally includes determining that the eye focus depth is different from the first depth using the filtered data, filtering the data optionally includes time averaging the data, and removing one or more outliers from the data.

[0134] According to another embodiment, the first subgroup of the one or more sensors optionally includes a gaze tracking system configured to measure eye vergence.

[0135] According to an embodiment, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device including one or more sensors and one or more displays, the one or more programs including instructions for: displaying a virtual object at a first depth via the one or more displays; using a first subgroup of the one or more sensors, determining that an eye focus depth is different from the first depth; while displaying the virtual object, shifting the virtual object from the first depth toward the eye focus depth; and locking the virtual object at the current depth according to determining that a first criterion associated with the current depth of the virtual object relative to the eye focus depth is satisfied.

[0136] According to another embodiment, the instructions optionally include instructions for: after locking the virtual object at the current depth, determining whether a second criterion is satisfied; and according to determining that the second criterion is not satisfied, continuing to display the virtual object at the current depth despite a change in the eye focus depth indicated by the first subgroup of the one or more sensors.

[0137] According to another embodiment, the instructions optionally include instructions for: according to determining that the second criterion is satisfied, shifting the virtual object from the current depth toward an updated eye focus depth indicated by the first subgroup of the one or more sensors.

[0138] According to another embodiment, determining whether the second criterion is met optionally includes using a first subgroup of the one or more sensors or a second subgroup of the one or more sensors to determine whether the second criterion is met.

[0139] According to another embodiment, displaying a virtual object at a first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth optionally include: displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth, based on determining that the virtual object is a first type of virtual object having a position defined relative to a position corresponding to the electronic device or a user of the electronic device.

[0140] According to another embodiment, the instructions further include instructions for: receiving data from a first subgroup of the one or more sensors; and filtering the data to obtain filtered data, determining that the eye focus depth is different from the first depth optionally includes using the filtered data to determine that the eye focus depth is different from the first depth, filtering the data optionally includes time averaging the data and removing one or more outliers from the data.

[0141] According to another embodiment, the first subgroup of the one or more sensors optionally includes a gaze tracking system configured to measure eye convergence.

[0142] According to an embodiment, a method of operating an electronic device including one or more sensors and one or more displays includes: displaying a virtual object at a first depth via the one or more displays; using a first subgroup of the one or more sensors, determining that the eye focus depth is different from the first depth; while displaying the virtual object, shifting the virtual object from the first depth toward the eye focus depth; and locking the virtual object at the current depth based on determining that a first criterion associated with the current depth of the virtual object relative to the eye focus depth is met.

[0143] According to another embodiment, the method optionally includes: after locking the virtual object at the current depth, determining whether a second criterion is met; and based on determining that the second criterion is not met, continuing to display the virtual object at the current depth despite an indication of a change in the eye focus depth by the first subgroup of the one or more sensors.

[0144] According to another embodiment, the method optionally includes: based on determining that the second criterion is met, shifting the virtual object from the current depth toward an updated eye focus depth indicated by the first subgroup of the one or more sensors.

[0145] According to another implementation, determining whether the second criterion is met optionally includes using a first subgroup of the one or more sensors or a second subgroup of the one or more sensors to determine whether the second criterion is met.

[0146] According to another implementation, displaying a virtual object at a first depth, determining that an eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth optionally include: displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth according to determining that the virtual object is a first type of virtual object having a position defined relative to a position corresponding to the electronic device or a user of the electronic device.

[0147] According to another implementation, the method optionally includes: receiving data from a first subgroup of the one or more sensors; and filtering the data to obtain filtered data, determining that the eye focus depth is different from the first depth optionally includes using the filtered data to determine that the eye focus depth is different from the first depth, and filtering the data optionally includes time averaging the data and removing one or more outliers from the data.

[0148] According to another implementation, the first subgroup of the one or more sensors optionally includes a gaze tracking system configured to measure eye convergence.

[0149] The foregoing is merely illustrative and various modifications may be made to the described implementations. The described implementations may be implemented individually or in any combination.

Claims

1. An electronic device, comprising: one or more sensors; one or more displays; one or more processors; and a memory storing instructions configured to be executed by the one or more processors, the instructions for: displaying a virtual object at a first depth via the one or more displays; using a first subgroup of the one or more sensors, determining that an eye focus depth is different from the first depth; while displaying the virtual object, shifting the virtual object from the first depth toward the eye focus depth; and locking the virtual object at the current depth according to determining that a first criterion associated with the current depth of the virtual object relative to the eye focus depth is satisfied.

2. The electronic device according to claim 1, wherein the instructions further include instructions for: after locking the virtual object at the current depth, determining whether a second criterion is satisfied; and continuing to display the virtual object at the current depth according to determining that the second criterion is not satisfied, although the first subgroup of the one or more sensors indicates a change in the eye focus depth.

3. The electronic device according to claim 2, wherein the instructions further include instructions for: shifting the virtual object from the current depth toward an updated eye focus depth indicated by the first subgroup of the one or more sensors according to determining that the second criterion is satisfied.

4. The electronic device according to claim 2, wherein determining whether the second criterion is satisfied includes using the first subgroup of the one or more sensors or using a second subgroup of the one or more sensors to determine whether the second criterion is satisfied.

5. The electronic device according to claim 1, wherein displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth towards the eye focus depth, and locking the virtual object at the current depth include: Displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth according to determining that the virtual object is a first type of virtual object having a position defined relative to a position corresponding to the electronic device or a user of the electronic device.

6. The electronic device according to claim 1, wherein the instructions further include instructions for: receiving data from the first subgroup of the one or more sensors; and filtering the data to obtain filtered data, wherein determining that the eye focus depth is different from the first depth includes using the filtered data to determine that the eye focus depth is different from the first depth, wherein filtering the data includes time-averaging the data, and wherein filtering the data includes removing one or more outliers from the data.

7. The electronic device according to claim 1, wherein the first subgroup of the one or more sensors includes a gaze tracking system configured to measure eye vergence.

8. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device including one or more sensors and one or more displays, the one or more programs including instructions for: displaying a virtual object at a first depth via the one or more displays; using a first subgroup of the one or more sensors to determine that an eye focus depth is different from the first depth; while displaying the virtual object, shifting the virtual object from the first depth toward the eye focus depth; and locking the virtual object at the current depth in accordance with determining that a first criterion associated with a current depth of the virtual object relative to the eye focus depth is satisfied.

9. The non-transitory computer-readable storage medium of claim 8, wherein the instructions further include instructions for: after locking the virtual object at the current depth, determining whether a second criterion is satisfied; and in accordance with determining that the second criterion is not satisfied, continuing to display the virtual object at the current depth despite an indication of a change in the eye focus depth by the first subgroup of the one or more sensors.

10. The non-transitory computer-readable storage medium of claim 9, wherein the instructions further include instructions for: in accordance with determining that the second criterion is satisfied, shifting the virtual object from the current depth toward an updated eye focus depth indicated by the first subgroup of the one or more sensors.

11. The non-transitory computer-readable storage medium of claim 9, wherein determining whether the second criterion is satisfied includes using the first subgroup of the one or more sensors or using a second subgroup of the one or more sensors to determine whether the second criterion is satisfied.

12. The non-transitory computer-readable storage medium according to claim 8, wherein displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth include: Displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth in accordance with determining that the virtual object is a first type of virtual object having a position defined relative to a position corresponding to the electronic device or a user of the electronic device.

13. The non-transitory computer-readable storage medium of claim 8, wherein the instructions further include instructions for: receiving data from the first subgroup of the one or more sensors; and filtering the data to obtain filtered data, wherein determining that the eye focus depth is different from the first depth includes using the filtered data to determine that the eye focus depth is different from the first depth, wherein filtering the data includes time-averaging the data, and wherein filtering the data includes removing one or more outliers from the data.

14. The non-transitory computer-readable storage medium according to claim 8, wherein the first subgroup of the one or more sensors includes a gaze tracking system configured to measure eye convergence.

15. A method of operating an electronic device, the electronic device including one or more sensors and one or more displays, the method including: Displaying a virtual object at a first depth via the one or more displays; Using a first subgroup of the one or more sensors, determining that an eye focus depth is different from the first depth; While displaying the virtual object, shifting the virtual object from the first depth toward the eye focus depth; And Locking the virtual object at the current depth according to determining that a first criterion associated with the current depth of the virtual object relative to the eye focus depth is satisfied.

16. The method according to claim 15, further including: After locking the virtual object at the current depth, determining whether a second criterion is satisfied; According to determining that the second criterion is not satisfied, continuing to display the virtual object at the current depth although the first subgroup of the one or more sensors indicates a change in the eye focus depth; And According to determining that the second criterion is satisfied, shifting the virtual object from the current depth toward an updated eye focus depth indicated by the first subgroup of the one or more sensors.

17. The method according to claim 15, further including: After locking the virtual object at the current depth, determining whether a second criterion is satisfied; And According to determining that the second criterion is not satisfied, continuing to display the virtual object at the current depth although the first subgroup of the one or more sensors indicates a change in the eye focus depth, wherein determining whether the second criterion is satisfied includes using the first subgroup of the one or more sensors or using a second subgroup of the one or more sensors to determine whether the second criterion is satisfied.

18. The method according to claim 15, wherein displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth towards the eye focus depth, and locking the virtual object at the current depth comprise: Displaying the virtual object at the first depth, determining that the eye focus depth is different from the first depth, shifting the virtual object from the first depth toward the eye focus depth, and locking the virtual object at the current depth according to determining that the virtual object is a first type of virtual object having a position defined relative to a position corresponding to the electronic device or a user of the electronic device.

19. The method according to claim 15, further including: Receiving data from the first subgroup of the one or more sensors; And Filtering the data to obtain filtered data, wherein determining that the eye focus depth is different from the first depth includes using the filtered data to determine that the eye focus depth is different from the first depth, wherein filtering the data includes time-averaging the data, and wherein filtering the data includes removing one or more outliers from the data.

20. The method according to claim 15, wherein the first subgroup of the one or more sensors includes a gaze tracking system configured to measure eye convergence.

Images