Method for controlling performance of an extended reality display system - Patents.com

Abstract

A system, method, and computer program product for displaying virtual content using a wearable electronic device determines locations of aiming centers of both eyes of a user wearing the wearable electronic device and estimates an error or precision for these aiming centers. A motion range may be determined for a focal length or a focal plane at the focal length based at least in part on an error or precision and a criterion for vergence-divergence and accommodation of the binocular vision of the virtual content using the wearable electronic device. The virtual content may be adjusted for presentation to the focal length or focal plane based at least in part on the motion range to an adjusted virtual content. The adjusted virtual content may be presented to the user for the focal length or focal plane.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 285,051, filed December 1, 2021, and entitled “METHODS FOR CONTROLLING PERFORMANCE OF EXTENDED REALITY DISPLAY SYSTEMS.” This application also cross-related to the following patent applications: U.S. Ser. No. 14 / 205,126, filed March 11, 2014; U.S. Ser. No. 14 / 212,961, filed March 14, 2014; U.S. Ser. No. 14 / 331,218, filed July 14, 2014; U.S. Ser. No. 14 / 555,585, filed November 27, 2014; U.S. Ser. No. 14 / 690,401, filed April 18, 2015; U.S. Ser. No. 14 / 738,877, filed June 13, 2015; and U.S. Ser. No. 16 / 215,477, filed December 10, 2018. The contents of the patent applications described herein are incorporated by reference in their entirety for all purposes.

[0002] The present disclosure relates to extended reality (i.e., virtual reality, augmented reality, and / or mixed reality) imaging, visualization, and display systems and methods. [Background technology]

[0003] Modern computing and display technologies have encouraged the development of so-called "extended reality" (XR) systems for "virtual reality" (VR), "augmented reality" (AR), or "mixed reality" (MR) experiences in which digitally reproduced images or parts thereof are presented to a user in a manner that appears or can be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to actual real-world visual input. AR scenarios typically involve the presentation of digital or virtual image information as an extension of the user's surrounding real-world visualization (i.e., transparency to real-world visual input). MR scenarios typically involve the presentation of digital or virtual objects that interact with real-world objects. Thus, AR and MR scenarios involve the presentation of digital or virtual image information with transparency to real-world visual input.

[0004] XR systems typically generate and display color data that increases the realism of the XR scenario. Many of these XR systems display color data by sequentially projecting partial images in different (e.g., primary) colors or "fields" (e.g., red, green, and blue) that correspond to a color image in rapid succession. Projecting color partial images at a sufficiently high rate (e.g., 60 Hz, 120 Hz, etc.) can deliver a smooth color XR scenario in the user's mind.

[0005] Various optical systems generate images, including color images at various depths, for displaying XR (VR, AR, and MR) scenarios. Several such optical systems are described in U.S. Utility Patent Application No. 14 / 555,585, filed November 27, 2014, the contents of which are incorporated herein by reference.

[0006] XR systems typically employ a wearable display device (e.g., a head-mounted display, helmet-mounted display, or smart glasses) that is at least loosely coupled to the user's head and thus moves as the user's head moves. When the user's head movement is detected by the display device, the data being displayed can be updated to account for changes in head pose (i.e., the orientation and / or location of the user's head).

[0007] Head-mounted display devices that enable AR and MR provide a side-by-side viewing of both real and virtual objects. With an "optical see-through" display, a user can see through a transparent (or semi-transparent) element in the display system and view light directly from real objects in the environment. The transparent element, often referred to as a "combiner," superimposes the light from the display over the user's view of the real world, which projects an image of the virtual content over the see-through view of the real objects in the environment. A camera may be mounted on the head-mounted display device to capture an image or video of the scene being viewed by the user.

[0008] Current optical systems, such as those in XR systems, render virtual content optically. The content is "virtual" in that it does not correspond to actual physical objects located at discrete positions in space. Instead, the virtual content exists only in the brain (optical centers) of a user of a head-mounted display device when stimulated by light beams directed at the user's eyes. XR systems attempt to present immersive XR scenarios that are colored, photorealistic, and realistic.

[0009] The brain's visualization center obtains useful perceptual information from the movement of the eyes and their components relative to one another. The convergence-divergence movement of the two eyes relative to one another (i.e., the movement of the pupils toward or away from one another to converge the line of sight of the eyes and fixate on an object) is closely linked to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, changing the focus of the eye's lenses, i.e., causing the eyes to accommodate and focus on an object at a different distance, will automatically cause a matching change in convergence-divergence to the same distance, in a relationship known as the "accommodation-vergence-divergence reflex." Similarly, a change in convergence-divergence will induce a matching change in accommodation, under normal conditions. Countering this reflex, as most conventional stereoscopic XR configurations do, is known to cause eye fatigue, headaches, or other forms of discomfort to the user.

[0010] Due to the complexity of the human visual perception system, it is challenging to produce XR technologies that facilitate comfortable, natural-feeling, rich presentation of virtual image elements among other virtual or real-world image elements. For example, three-dimensional (3D) image display systems can cause users to experience vergence-accommodation conflict problems. This problem arises when two optical depth-related biological processes send competing depth signals to the viewer / user's brain. Vergence is related to the tendency of the viewer's eyes to rotate to align the optical axis(es) with the viewer's attentional object at a certain distance. In a binocular system, the point where the optical axes intersect can be called the "vergence point." The amount of rotation of the viewer's eyes during vergence is interpreted by the viewer's brain as an estimated depth. Accommodation is related to the tendency of the viewer's eye lens to focus so that the viewer's attentional object is at a certain distance. The focus of the viewer's eyes during convergence and divergence is interpreted by the viewer's brain as different estimated depths. When the convergence and accommodation signals are interpreted by the viewer's brain as the same or similar estimated depths, the 3D viewing experience is natural and comfortable for the viewer. On the other hand, when the convergence and accommodation signals are interpreted by the viewer's brain as substantially different estimated depths, the 3D viewing experience may be suboptimal for the viewer and may result in discomfort (eye strain, headaches, etc.) and fatigue. Such a problem is known as convergence-accommodation conflict.

[0011] Stereoscopic wearable glasses generally feature two displays for the left and right eyes that are configured to display images with slightly different presentations of elements such that 3D perspective is perceived by the human visual system. Such configurations have been found to be uncomfortable for many users due to the mismatch between vergence and accommodation, which causes vergence-accommodation conflicts that must be overcome to perceive images in three dimensions. In fact, some users are not able to tolerate stereoscopic configurations. These limitations apply to all typical XR systems. Thus, most conventional XR systems are not optimally suited to present rich binocular 3D experiences in a manner that would be comfortable and maximally useful to users, in part because conventional systems fail to address some of the fundamental aspects of the human perceptual system, including vergence-accommodation conflicts.

[0012] The XR system must also be capable of displaying virtual digital content at various perceived positions and distances to the user for a 3D experience. The design of a portable XR system also presents numerous other challenges that can adversely affect XR system performance, including the speed of the system in delivering the virtual digital content, the quality of the virtual digital content, the user's eye relief (addressing vergence-accommodation conflicts), system size and portability, battery life, system heating, processing power, memory, bandwidth, and other system and optical challenges. These limitations increase the importance of 3D image rendering for natural vergence and accommodation.

[0013] Improved systems and techniques for processing image data and displaying images are needed, including, for example, systems and techniques for minimizing vergence-divergence-accommodation conflicts while rendering and displaying 3D images to a viewer / user, and for doing so while minimizing demands on the limited graphical processing capabilities of portable XR systems. The systems and methods described herein are configured to address these challenges. [Prior art documents] [Patent documents]

[0014] [Patent Document 1] U.S. Patent No. 9,791,700 Summary of the Invention [Means for solving the problem]

[0015] Embodiments are directed to methods for controlling the performance of an XR system. In particular, embodiments are directed to methods for modifying content displayed by an XR system to reduce vergence-accommodation conflicts experienced by a user.

[0016] In one embodiment, a method for displaying virtual content using an expanded reality (XR) system includes an eye tracking subsystem of the XR system obtaining a rendering position error. The method also includes the XR system determining a motion range from at least the rendering position error. The method further includes modifying the virtual content based on at least the motion range before the XR system displays the virtual content.

[0017] In another embodiment, a method for displaying virtual content using an expanded reality (XR) system includes the XR system obtaining a rendering position error. The method also includes the XR system determining a motion range from at least the rendering position error. The method further includes modifying the virtual content based on at least the motion range before the XR system displays the virtual content.

[0018] In one or more embodiments, modifying the virtual content includes removing a portion of the virtual content, where the portion corresponds to a depth outside of a working range from a focal plane of the XR system. Modifying the virtual content may include configuring a portion of the virtual content to be displayed monocularly, where the portion may correspond to a depth outside of a working range from a focal plane of the XR system.

[0019] In one or more embodiments, the rendering position error corresponds to an aim center error. The method may include the XR system tracking a user's eye position, and the XR system estimating the aim center error from at least the user's eye position. The method may include the XR system determining a system error corresponding to the eye tracking by the XR system, and the XR system modifying the operating range based on the system error. The method may include the XR system acquiring a user's eye characteristic, and the XR system modifying the system error based on at least the user's eye characteristic. The XR system may detect the user's eye characteristic using a camera. The XR system may receive the user's eye characteristic from the user through a user interface.

[0020] In one or more embodiments, the method includes the XR system updating the rendering position error in real time. The method may include the XR system reducing the operating range to a predetermined minimum operating range when the XR system cannot update the rendering position error. The operating range may be a binocular operating range. The rendering position error may be a population level rendering position error.

[0021] In one or more embodiments, the method includes the XR system detecting a movement of the XR system and the XR system estimating an aim center error from at least the movement of the XR system. The movement of the XR system may be relative to a user. The XR system may detect the movement using an accelerometer. The method may include the XR system detecting the movement of the XR system and the XR system reducing the motion range to a predetermined minimum motion range when the movement of the XR system is more than a predetermined threshold amount.

[0022] In yet another embodiment, a method for displaying virtual content using an expanded reality (XR) system includes the XR system obtaining a time of a persistent motion. The method also includes the XR system determining a motion range from at least the time of the persistent motion. The method further includes modifying the virtual content based on at least the motion range before the XR system displays the virtual content.

[0023] Some embodiments are directed to a system for displaying virtual content using a wearable electronic device. In these embodiments, the wearable electronic device includes a virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR) goggles, smart glasses, or headset, a three-dimensional or stereoscopic wearable device, or a stereoscopic display device or pair thereof, capable of presenting virtual content to a user, by itself or when connected with another computing device (e.g., a smartphone, a tablet computing device, a laptop computer, a desktop computer, a gaming console, or a remote computing device such as a remote server).

[0024] The system comprises a display device that presents virtual content to a user, a microprocessor operably coupled to the display device, and a memory that stores a sequence of instructions that, when executed by the microprocessor, cause the microprocessor to perform a set of actions. In these embodiments, the set of actions includes determining, by the wearable electronic device, an aiming center for a first eye of a user wearing the wearable electronic device, and estimating, by the wearable electronic device or a remote computing device connected to the wearable electronic device via a network, an error or accuracy related to the aiming center.

[0025] The set of acts further includes determining a motion range for a focal length or a focal plane at the focal length based at least in part on an error or precision and a criterion for binocular vergence and accommodation of the virtual content using the wearable electronic device, and adjusting the virtual content for presentation to the focal plane or focal length based at least in part on the motion range to an adjusted virtual content.

[0026] In some of these embodiments, the set of actions further includes identifying a characteristic of a first eye of the user, the characteristic related to an eye disease of the user, and adjusting the aiming center based at least in part on the characteristic of the first eye of the user.

[0027] Additionally or alternatively, the adjusted virtual content may be presented to the user by using a wearable electronic device that projects a light beam related to the adjusted virtual content relative to at least a focal plane or focal length. Further, the aforementioned error or accuracy includes at least one of a rendering camera position error or accuracy specific to the user and the wearable electronic device, a system level error or accuracy specific to the wearable electronic device, or a population level residual error for multiple users.

[0028] In some embodiments, a determination may be made to determine whether a rendering position error or accuracy should be updated based at least in part on one or more criteria, where a first error or accuracy may be estimated with respect to the aiming center.

[0029] In some of the immediately preceding embodiments, the operating range may be adjusted to a smaller operating range, where the smaller operating range includes a distance range for a focal plane or focal length, and virtual content for the focal plane or focal length is rendered according to the smaller operating range.

[0030] Additionally or alternatively, the system may be configured in a manner to receive a signal indicative of a change in relative position, movement, or motion between the wearable electronic device and the user, and in response to receiving the signal, determine an aim center error or accuracy or a rendering camera position error or accuracy based at least on the signal. The operating range may be adjusted based, at least in part, on the aim center error or accuracy or the rendering camera position error or accuracy.

[0031] In some of the preceding embodiments, adjusting the operating range may include performing a determination of a rendering camera position for the first eye of the user by executing at least an eye tracking module of the wearable electronic device or by performing a visual task. Further, an estimated aiming center error or accuracy may be determined with respect to the rendering camera position for the first eye of the user, and a focal plane or focal length corresponding to the ground truth for the multiple users may be identified.

[0032] In some of the immediately preceding embodiments, adjusting the operating range may further include adjusting the operating range for the focal plane or focal length to an adjusted operating range based, at least in part, on the estimated aim center error or accuracy.

[0033] In some embodiments, determining the aiming center may include presenting a first marker at a first location to a first eye of the user, the first marker including a first hole in the object or the first rendered object, and presenting a target to the first eye of the user at a focal plane or focal length.

[0034] In some of these embodiments, the target may be moved about the first marker. A first signal may be received from the user when the target becomes visible by a first eye of the user. In response to receiving the first signal, a first reference entity connecting the first marker and the target may be determined.

[0035] In some of the immediately preceding embodiments, a second marker may be presented to the user's first eye at a second location, the second marker including a second hole in the object or a second rendered object, and a target or separate target may also be presented to the user's first eye at a focal plane or focal length.

[0036] In some of the immediately preceding embodiments, the target or separate target may be moved around the second marker. A second signal may be received from the user when the target or separate target becomes visible by the first eye of the user. In response to receiving the second signal, a second reference entity may be determined that connects the second marker and the target or separate target. Additionally or alternatively, a center of sight may be determined for the first eye of the user based at least in part on the first reference entity and the second reference entity.

[0037] In some embodiments in which the system determines the aiming center, one or more light sources may emit a light ray or beam toward the user's eye, and a first characteristic associated with the interaction of the user's eye with a reflection of the light ray or beam may be detected using one or more sensors.

[0038] In some of these embodiments, the one or more sensors may include a photodiode and the one or more light sources may include a light emitting diode. In some embodiments, the first characteristic may include a return amount, a reflection, or a specific pattern detected by the one or more sensors in response to at least a portion of the first light ray or beam from the user's eye. Additionally, a first reference entity, including, for example, a first gaze direction, may be determined at least in part based on the first characteristic.

[0039] In some of these embodiments, the one or more light sources may emit a second light ray or beam toward the same eye of the user, and a second characteristic associated with the interaction of the second light ray or beam with the user's eye may be detected using one or more camera sensors mounted to the user's eye.

[0040] In some of these embodiments, the one or more sensors may include a photodiode and the one or more light sources may include a light emitting diode. In some embodiments, the second characteristic may include a return amount, a reflection, or a specific pattern detected by the one or more sensors in response to at least a portion of the second light ray or beam from the user's eye. Additionally, a second reference entity, including, for example, a second gaze direction, may be determined based at least in part on the first characteristic.

[0041] The aiming center of the user's eye may be determined based at least in part on the first reference entity and the second reference entity, for example, the aiming center of the user's eye may be determined to be an intersection of the first reference entity and the second reference entity.

[0042] Some of these embodiments assume that the user's eyes move in unison, and thus only one eye of the user is equipped with one or more of the above-mentioned light sources and one or more sensors. Some other embodiments equip the user's eyes with one or more of the above-mentioned light sources and one or more sensors for each of the user's two eyes. In these latter embodiments, the individual aiming centers of each of the user's two eyes may be determined independently. In some of these embodiments, the individual aiming centers may be used to characterize the focal depth, focal plane, convergence-divergence movement, accommodation, working range with respect to the focal plane or focal depth, etc., along with the individual gaze directions.

[0043] Some embodiments are directed to methods for displaying virtual content using a wearable electronic device. In these embodiments, the wearable electronic device includes virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR) goggles, smart glasses, or headset, three-dimensional or stereoscopic wearable device, or stereoscopic display device, or a pair thereof, capable of presenting virtual content to a user, either by itself or when connected with another computing device (e.g., a smartphone, a tablet computing device, a laptop computer, a desktop computer, a gaming console, or a remote computing device such as a remote server).

[0044] In these embodiments, the wearable electronic device may determine an aiming center for a first eye of a user wearing the wearable electronic device, and the wearable electronic device or a remote computing device connected to the wearable electronic device via a network may further estimate an error or precision for the aiming center. A motion range may be determined for a focal length or a focal plane at a focal length based at least in part on an error or precision and criteria for vergence-divergence and accommodation of binocular vision of virtual content using the wearable electronic device. The virtual content may be adjusted for presentation relative to the focal plane or focal length based at least in part on the motion range.

[0045] In some of these embodiments, a characteristic of a first eye of a user may be identified, the characteristic relating to an eye disease of the user, and the center of aim is adjusted based at least in part on the characteristic of the first eye of the user.

[0046] Additionally or alternatively, the adjusted virtual content may be presented to the user by using a wearable electronic device that projects a light beam related to the adjusted virtual content relative to at least a focal plane or focal length. Further, the aforementioned error or accuracy includes at least one of a rendering camera position error or accuracy specific to the user and the wearable electronic device, a system level error or accuracy specific to the wearable electronic device, or a population level residual error for multiple users.

[0047] In some embodiments, a determination may be made to determine whether a rendering position error or accuracy should be updated based at least in part on one or more criteria, where a first error or accuracy may be estimated with respect to the aiming center.

[0048] In some of the immediately preceding embodiments, the operating range may be adjusted to a smaller operating range, where the smaller operating range includes a distance range for a focal plane or focal length, and virtual content for the focal plane or focal length is rendered according to the smaller operating range.

[0049] Additionally or alternatively, the system may be configured in a manner to receive a signal indicative of a change in relative position, movement, or motion between the wearable electronic device and the user, and based on receiving the signal, determine an aim center error or accuracy or a rendering camera position error or accuracy based at least on the signal. The operating range may be adjusted based, at least in part, on the aim center error or accuracy or the rendering camera position error or accuracy.

[0050] In some of the preceding embodiments, adjusting the operating range may include performing a determination of a rendering camera position for the first eye of the user by executing at least an eye tracking module of the wearable electronic device or by performing a visual task. Further, an estimated aiming center error or accuracy may be determined with respect to the rendering camera position for the first eye of the user, and a focal plane or focal length corresponding to the ground truth for the multiple users may be identified.

[0051] In some of the immediately preceding embodiments, adjusting the operating range may further include adjusting the operating range for the focal plane or focal length to an adjusted operating range based, at least in part, on the estimated aim center error or accuracy.

[0052] In some embodiments, determining the aiming center may include presenting a first marker at a first location to a first eye of the user, the first marker including a first hole in the object or the first rendered object, and presenting a target to the first eye of the user at a focal plane or focal length.

[0053] In some of these embodiments, the target may be moved about the first marker. A first signal may be received from the user when the target becomes visible by a first eye of the user. In response to receiving the first signal, a first reference entity connecting the first marker and the target may be determined.

[0054] In some of the immediately preceding embodiments, a second marker may be presented to the user's first eye at a second location, the second marker including a second hole in the object or a second rendered object, and a target or separate target may also be presented to the user's first eye at a focal plane or focal length.

[0055] In some of the immediately preceding embodiments, the target or separate target may be moved around the second marker. A second signal may be received from the user when the target or separate target becomes visible by the first eye of the user. In response to receiving the second signal, a second reference entity may be determined that connects the second marker and the target or separate target. Additionally or alternatively, a center of sight may be determined for the first eye of the user based at least in part on the first reference entity and the second reference entity.

[0056] In some embodiments in which the system determines the aiming center, one or more light sources may emit a first light ray or beam toward the user's eye, and a first characteristic associated with an interaction of the first light ray or beam with the user's eye may be detected using one or more sensors.

[0057] In some of these embodiments, the one or more sensors may include a photodiode and the one or more light sources may include a light emitting diode. In some embodiments, the first characteristic may include a return amount, a reflection, or a specific pattern detected by the one or more sensors in response to at least a portion of the first light ray or beam from the user's eye. Additionally, a first reference entity, including, for example, a first gaze direction, may be determined at least in part based on the first characteristic.

[0058] In some of these embodiments, the one or more light sources may emit a second light ray or beam toward the same eye of the user, and a second characteristic associated with the interaction of the second light ray or beam with the user's eye may be detected using one or more sensors mounted to the user's eye.

[0059] In some of these embodiments, the one or more sensors may include a photodiode and the one or more light sources may include a light emitting diode. In some embodiments, the second characteristic may include a return amount, a reflection, or a specific pattern detected by the one or more sensors in response to at least a portion of the second light ray or beam from the user's eye. Additionally, a second reference entity, including, for example, a second gaze direction, may be determined based at least in part on the first characteristic.

[0060] The aiming center of the user's eye may be determined based at least in part on the first reference entity and the second reference entity, for example, the aiming center of the user's eye may be determined to be an intersection of the first reference entity and the second reference entity.

[0061] Some of these embodiments assume that the user's eyes move in unison, and thus only one eye of the user is equipped with one or more of the above-mentioned light sources and one or more sensors. Some other embodiments equip the user's eyes with one or more of the above-mentioned light sources and one or more sensors for each of the user's two eyes. In these latter embodiments, the individual aiming centers of each of the user's two eyes may be determined independently. In some of these embodiments, the individual aiming centers may be used to characterize the focal depth, focal plane, convergence-divergence movement, accommodation, working range with respect to the focal plane or focal depth, etc., along with the individual gaze directions.

[0062] Some embodiments are directed to a computer program product comprising a non-transitory machine-readable storage medium having stored thereon a sequence of instructions that, when executed by a microprocessor, cause the microprocessor to perform a set of acts for displaying virtual content using a wearable electronic device. In these embodiments, the wearable electronic device includes a virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR) goggles, smart glasses, or headset, a three-dimensional or stereoscopic wearable device, or a stereoscopic display device or pair thereof, capable of presenting virtual content to a user, by itself or when connected with another computing device (e.g., a smartphone, a tablet computing device, a laptop computer, a desktop computer, a gaming console, or a remote computing device such as a remote server).

[0063] In these embodiments, the set of actions includes determining, by the wearable electronic device, a center of aiming for a first eye of a user wearing the wearable electronic device, and estimating, by the wearable electronic device or a remote computing device connected to the wearable electronic device via a network, an error or accuracy related to the center of aiming.

[0064] The set of acts further includes determining a motion range for a focal length or a focal plane at the focal length based at least in part on an error or precision and a criterion for binocular vergence and accommodation of the virtual content using the wearable electronic device, and adjusting the virtual content for presentation to the focal plane or focal length based at least in part on the motion range to an adjusted virtual content.

[0065] In some of these embodiments, the set of actions further includes identifying a characteristic of a first eye of the user, the characteristic related to an eye disease of the user, and adjusting the aiming center based at least in part on the characteristic of the first eye of the user.

[0066] Additionally or alternatively, the adjusted virtual content may be presented to the user by using a wearable electronic device that projects a light beam related to the adjusted virtual content relative to at least a focal plane or focal length. Further, the aforementioned error or accuracy includes at least one of a rendering camera position error or accuracy specific to the user and the wearable electronic device, a system level error or accuracy specific to the wearable electronic device, or a population level residual error for multiple users.

[0067] In some embodiments, a determination may be made to determine whether a rendering position error or accuracy should be updated based at least in part on one or more criteria, where a first error or accuracy may be estimated with respect to the aiming center.

[0068] In some of the immediately preceding embodiments, the operating range may be adjusted to a smaller range, where the smaller operating range includes a distance range for a focal plane or focal length, and virtual content for the focal plane or focal length is rendered according to the smaller operating range.

[0069] Additionally or alternatively, the system may be configured in a manner to receive a signal indicative of a change in relative position, movement, or motion between the wearable electronic device and the user, and in response to receiving the signal, determine an aim center error or accuracy or a rendering camera position error or accuracy based at least on the signal. The operating range may be adjusted based at least in part on the aim center error or accuracy or the rendering camera position error or accuracy.

[0070] In some of the preceding embodiments, adjusting the operating range may include performing a determination of a rendering camera position for the first eye of the user by executing at least an eye tracking module of the wearable electronic device or by performing a visual task. Further, an estimated aiming center error or accuracy may be determined for the rendering camera position for the first eye of the user, and a focal plane or focal length corresponding to the ground truth for multiple users may be identified.

[0071] In some of the immediately preceding embodiments, adjusting the operating range may further include adjusting the operating range for the focal plane or focal length to an adjusted operating range based, at least in part, on the estimated aim center error or accuracy.

[0072] In some embodiments, determining the aiming center may include presenting a first marker at a first location to the user's first eye, where the first marker includes a first hole in the object or the first rendered object, and presenting a target to the user's first eye at a focal plane or focal length.

[0073] In some of these embodiments, the target may be moved about the first marker. A first signal may be received from the user when the target becomes visible by a first eye of the user. In response to receiving the first signal, a first reference entity connecting the first marker and the target may be determined.

[0074] In some of the immediately preceding embodiments, a second marker may be presented to the user's first eye at a second location, the second marker including a second hole in the object or a second rendered object, and a target or separate target may also be presented to the user's first eye at a focal plane or focal length.

[0075] In some of the immediately preceding embodiments, the target or separate target may be moved around the second marker. A second signal may be received from the user when the target or separate target becomes visible by the user's first eye. In response to receiving the second signal, a second reference entity may be determined that connects the second marker and the target or separate target. Additionally or alternatively, a center of sight may be determined for the user's first eye based at least in part on the first reference entity and the second reference entity.

[0076] In some embodiments in which the system determines the aiming center, one or more light sources may emit a light ray or beam toward the user's eye, and a first characteristic associated with an interaction of the first light ray or beam with the user's eye may be detected using one or more sensors.

[0077] In some of these embodiments, the one or more sensors may include a photodiode and the one or more light sources may include a light emitting diode. In some embodiments, the first characteristic may include a return amount, a reflection, or a specific pattern detected by the one or more sensors in response to at least a portion of the first light ray or beam from the user's eye. Additionally, a first reference entity, including, for example, a first gaze direction, may be determined at least in part based on the first characteristic.

[0078] In some of these embodiments, the one or more light sources may emit a second light ray or beam toward the same eye of the user, and a second characteristic associated with the interaction of the second light ray or beam with the user's eye may be detected using one or more sensors mounted to the user's eye.

[0079] In some of these embodiments, the one or more sensors may include a photodiode and the one or more light sources may include a light emitting diode. In some embodiments, the second characteristic may include a return amount, a reflection, or a specific pattern detected by the one or more sensors in response to at least a portion of the second light ray or beam from the user's eye. Additionally, a second reference entity, including, for example, a second gaze direction, may be determined based at least in part on the first characteristic.

[0080] The aiming center of the user's eye may be determined based at least in part on the first reference entity and the second reference entity, for example, the aiming center of the user's eye may be determined to be an intersection of the first reference entity and the second reference entity.

[0081] Some of these embodiments assume that the user's eyes move in unison, and thus only one eye of the user is equipped with one or more of the above-mentioned light sources and one or more sensors. Some other embodiments equip the user's eyes with one or more of the above-mentioned light sources and one or more sensors for each of the user's two eyes. In these latter embodiments, the individual aiming centers of each of the user's two eyes may be determined independently. In some of these embodiments, the individual aiming centers may be used to characterize the focal depth, focal plane, convergence-divergence movement, accommodation, working range with respect to the focal plane or focal depth, etc., along with the individual gaze directions.

[0082] Additional and other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims. [Brief description of the drawings]

[0083] The above and other aspects of the embodiments will be described in further detail with reference to the accompanying drawings, in which identical elements in different figures are referred to by common reference numerals.

[0084] [Figure 1] FIG. 1 depicts a user's view of AR / MR through a wearable AR / MR user device, according to some embodiments.

[0085] [Diagram 2] FIG. 2 diagrammatically depicts an XR system and its subsystems, according to some embodiments.

[0086] [Figure 3A] 3A and 3B diagrammatically depict a user's eye viewing a display, according to some embodiments. [Figure 3B] 3A and 3B diagrammatically depict a user's eye viewing a display, according to some embodiments.

[0087] [Figure 4] FIG. 4 diagrammatically depicts inputs for determining a motion range for an XR system, according to some embodiments.

[0088] [Diagram 5] FIG. 5 diagrammatically depicts determining / estimating aim center / rendering camera position using user input, according to some embodiments.

[0089] [Figure 6A] 6A and 6B diagrammatically depict a user's eye viewing a display, according to some embodiments. [Figure 6B] 6A and 6B diagrammatically depict a user's eye viewing a display, according to some embodiments.

[0090] [Figure 7] FIG. 7 is a flowchart depicting a method for modifying virtual content before display based on rendering position error, according to some embodiments.

[0091] [Figure 8] FIG. 8 illustrates an example computing device and some example computing components that may be used to implement at least the computational and data processing portions of the wearable electronic devices described herein in some embodiments.

[0092] [Figure 9] FIG. 9 illustrates a flow diagram for determining operating ranges for binocular perception of virtual content exhibiting vergence-divergence-accommodation conflicts in one or more embodiments.

[0093] [Figure 10] FIG. 10 illustrates a flow diagram for dynamically and real-time updating the center of aim of the eye of a user wearing a wearable electronic device in one or more embodiments.

[0094] [Figure 11] FIG. 11 illustrates a block diagram with further details regarding adjusting the operating range of FIG. 10 in some embodiments.

[0095] [Figure 12] FIG. 12 illustrates a block diagram with further details regarding determining the aiming center of the user's eye of FIG. 9 in some embodiments.

[0096] [Figure 13] FIG. 13 illustrates a schematic implementation of eye tracking instrumentation, simplified for purposes of illustration and explanation, in some embodiments.

[0097] [Figure 14] FIG. 14 illustrates an example output of reflection of light, patterns, or objects from a structure in a user's eye captured by one or more sensors (e.g., one or more photodiodes) in some embodiments.

[0098] [Figure 15] FIG. 15 illustrates a simplified schematic diagram of an eye according to some embodiments.

[0099] [Figure 16] FIG. 16 shows another perspective view of an eye and one or more cameras for gaze tracking and center of aim identification in accordance with one or more embodiments.

[0100] [Figure 17] FIG. 17 illustrates yet another perspective view of an eye and one or more cameras for gaze tracking and center of aim identification, in accordance with one or more embodiments.

[0101] [Figure 18] FIG. 18 illustrates yet another perspective view of an eye and one or more cameras for gaze tracking and center of aim identification, in accordance with one or more embodiments.

[0102] [Figure 19] FIG. 19 illustrates a transformation matrix diagram for gaze tracking and center of aim identification, in accordance with one or more embodiments.

[0103] [Figure 20] FIG. 20 illustrates an exemplary high-level block diagram of gaze tracking and center of aim identification, according to one embodiment.

[0104] [Figure 21] FIG. 21 illustrates a block diagram of a method or system for determining the center of aim using eye tracking techniques in some embodiments.

[0105] [Figure 22] FIG. 22 illustrates a simplified schematic diagram of a wearable electronic device in some embodiments.

[0106] [Figure 23] FIG. 23 illustrates an example architecture for electronics for an augmented reality device, according to one or more illustrated embodiments.

[0107] In order to more clearly understand how to obtain the above-listed and other advantages and objects of the various embodiments, a more detailed description of the embodiments is provided with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and that elements of similar structure or function are represented by like reference numerals throughout. It should be understood that these drawings depict only certain illustrated embodiments, and therefore should not be considered as limiting the scope of the embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0108] DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for controlling the performance of an XR system in a single embodiment or in multiple embodiments. Other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims.

[0109] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples of the present disclosure, so as to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and examples are not meant to limit the scope of the present disclosure. Where certain elements of the present disclosure can be partially or fully implemented using known components (or methods or processes), only those parts of such known components (or methods or processes) that are necessary for the understanding of the present disclosure will be described, and detailed descriptions of other parts of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Furthermore, various embodiments encompass current and future known equivalents of the components referenced herein as examples.

[0110] Although the performance control system may be implemented independently of an XR system, some embodiments below are described in relation to an AR system for illustrative purposes only. For example, the performance control system described herein may also be used in conjunction with VR (virtual reality), MR (mixed reality), and XR (extended reality) systems in the same manner. In various embodiments described herein, the terms "augmented reality (AR)", "virtual reality (VR)", "mixed reality (MR)", and "extended reality (XR)" may be used synonymously unless otherwise explicitly distinguished or contrasted.

[0111] Illustrative AR Scenarios and Systems The following description relates to an exemplary AR system whose performance may be controlled / modified, however, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of XR systems such as VR and MR systems) and thus should not be limited to only the exemplary systems disclosed herein.

[0112] AR scenarios often involve the presentation of virtual content (e.g., color images and sounds) that correspond to virtual objects in association with real-world objects. For example, referring to FIG. 1, an AR scene 100 is depicted in which a user of the AR technology sees a real-world physical park-like setting 102 featuring people, trees, and buildings in the background, and a real-world physical concrete platform 104. In addition to these items, the user of the AR technology also perceives that they "see" a virtual robot figure 106 standing on the physical concrete platform 104 and a virtual cartoon-like avatar character 108, which appears to be an anthropomorphic bumblebee, flying beside it, even though these virtual objects 106, 108 do not exist in the real world.

[0113] 2, an embodiment of an AR system 200 is illustrated, according to some embodiments. The AR system 200 may be operated in conjunction with a projection subsystem 208 to provide images of virtual objects mixed with physical objects in the field of view of a user 250. The approach employs one or more at least partially transparent surfaces through which the surrounding real-world environment, including physical objects, may be viewed and through which the AR system 200 produces images of virtual objects. The projection subsystem 208 is stored within a control subsystem 201, which is operably coupled to a display system / subsystem 204 through a link 207. The link 207 may be a wired or wireless communication link.

[0114] For AR applications, it may be desirable to spatially position various virtual objects relative to individual physical objects within the field of view of the user 250. Virtual objects may take any of a wide variety of forms, having any of a variety of data, information, concepts, or logical constructs that can be represented as images. Non-limiting examples of virtual objects may include a virtual text object, a virtual number object, a virtual alphanumeric object, a virtual tag object, a virtual field object, a virtual chart object, a virtual map object, a virtual instrumentation object, or a virtual visual representation of a physical object.

[0115] AR system 200 comprises a frame structure 202 worn by a user 250, a display system 204 carried by frame structure 202 such that display system 204 is positioned in front of the eye of user 250, and a speaker 206 incorporated into or connected to display system 204. In the illustrated embodiment, speaker 206 is carried by frame structure 202 such that speaker 206 is positioned adjacent to (in or around) the ear canal of user 250, e.g., a plug-in earphone or headphone.

[0116] The display system 204 is designed to present to the eyes of the user 250 with a photo-based radiation pattern that can be comfortably perceived as an extension to the surrounding environment, including both two-dimensional and three-dimensional content. The display system 204 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene. To achieve this goal, the display system 204 includes a projection subsystem 208 and a partially transparent display screen through which the projection subsystem 208 projects images. The display screen is positioned within the field of view of the user 250 between the eyes of the user 250 and the surrounding environment.

[0117] In some embodiments, the projection subsystem 208 takes the form of a scanning-based projection device and the display screen takes the form of a waveguide-based display in which scanned light from the projection subsystem 208 is injected to produce, for example, an image at a single optical viewing distance (e.g., arm's length) closer than infinity, images at multiple discrete optical viewing distances or focal planes, and / or stacked image layers at multiple viewing distances or focal planes to represent a volumetric 3D object. These layers in the light field may be stacked close enough together to appear persistent to the human visual subsystem (e.g., one layer is within the circle of confusion or cone of confusion of an adjacent layer). A circle of confusion is an optical spot caused in an optical system, in some embodiments, by a light cone from a lens that does not reach a perfect focus when imaging a point source, and thus may also be referred to as a cone of confusion, a disk of confusion, a circle of obscuration, a blur circle, or a blur spot in some embodiments of the present application. The clutter cone in these embodiments is thus distinguishable from the clutter cone in the context of navigation or the context of the biological ear. Additionally or alternatively, pixels may be blended across two or more layers to increase the perceived continuity of the transition between layers in the light field, even if those layers are more sparsely stacked (e.g., one layer is outside the clutter cone of an adjacent layer). The display system 204 may be monocular or binocular. The scanning assembly includes one or more light sources that produce light beams (e.g., emit different colored light in a defined pattern). The light sources may take any of a wide variety of forms, for example, a set of red, green, and blue (RGB) sources (e.g., laser diodes capable of outputting red, green, and blue light), each operable to produce red, green, and blue coherent collimated light according to a defined pixel pattern, each of which is defined within a separate frame of pixel information or data. Laser light provides high color saturation and is very energy efficient.The light coupling subsystem includes a light guide input device, such as, for example, one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms, for optically coupling light into an edge of the display screen. The light coupling subsystem further includes a collimation element that collimates the light from the optical fiber. Optionally, the light coupling subsystem includes an optical modulation device configured to converge the light from the collimation element toward a focal point in the center of the light guide input device, thereby allowing the size of the light guide input device to be minimized. Thus, the display subsystem 204 generates a series of composite image frames of pixel information that presents an undistorted image of one or more virtual objects to the user. The display subsystem 204 may also generate a series of color composite partial image frames of pixel information that presents an undistorted color image of one or more virtual objects to the user. Further details describing the display subsystem are provided in US patent application Ser. Nos. 14 / 212,961 and 14 / 331,218, the contents of which are previously incorporated by reference herein.

[0118] The AR system 200 further includes one or more sensors mounted on the frame structure 202 to detect the position (including orientation) and movement of the user's 250 head and / or the position and interocular distance of the user's 250 eyes. Such sensors may include image capture devices, microphones, inertial measurement units (IMUs), accelerometers, compasses, GPS units, wireless devices, gyroscopes, and the like. For example, in one embodiment, the AR system 200 includes a head-mounted transducer subsystem including one or more inertial transducers for capturing inertial measurements indicative of the movement of the user's 250 head. Such devices may be used to sense, measure, or collect information about the head movement of the user 250. For example, these devices may be used to detect / measure the movement, speed, acceleration, and / or position of the user's 250 head. The position (including orientation) of the user's 250 head is also known as the "head pose" of the user 250.

[0119] 2 may include one or more forward-facing cameras. The cameras may be employed for any number of purposes, such as recording images / video from a front direction of the system 200. In addition, the cameras may be used to capture information about the environment in which the user 250 is located, i.e., information indicative of the distance, orientation, and / or angular position of the user 250 relative to the environment and specific objects within that environment, etc.

[0120] The AR system 200 may further include a rear-facing camera for tracking the angular position of the user's 250 eyes (the direction one or both eyes are pointing), eye blinks, and depth of focus (by detecting eye convergence). Such eye tracking information may be determined, for example, by projecting light onto the end user's eye and detecting a return or reflection of at least a portion of the projected light.

[0121] The augmented reality system 200 further includes a control subsystem 201, which may take any of a wide variety of forms. The control subsystem 201 includes a number of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), e.g., field programmable gate arrays (FPGAs), and / or programmable logic controllers (PLUs). The control subsystem 201 may include a digital signal processor (DSP), a central processing unit (CPU) 251, a graphics processing unit (GPU) 252, and one or more frame buffers 254. The CPU 251 controls the overall operation of the system, while the GPU 252 renders frames (i.e., converts a 3D scene into a two-dimensional image) and stores these frames in the frame buffer 254. Although not shown, one or more additional integrated circuits may control the loading and / or retrieval of frames into and / or from the frame buffer 254 and the operation of the display system 204. The loading and / or retrieval of frames into and / or from the frame buffer 254 may employ dynamic addressing, for example, if a frame is over-rendered. The control subsystem 201 further includes a read-only memory (ROM) and a random access memory (RAM). The control subsystem 201 further includes a 3D database 260 from which the GPU 252 may access 3D data of one or more scenes to render the frames and synthesized sound data associated with virtual sound sources contained within the 3D scenes.

[0122] The augmented reality system 200 further includes a user orientation detection module 248. The user orientation module 248 may detect the instantaneous position of the head of the user 250 and predict the position of the head of the user 250 based on the position data received from the sensors. The augmented reality system 200 further includes an eye tracking module 249. The eye tracking module 249 tracks the eyes of the user 250, and in particular, the direction and / or distance at which the user 250 is focused based on the tracking data received from the sensors.

[0123] Rendering position errors causing user discomfort Errors in the placement of the rendering camera for rendering virtual content can lead to severe visual discomfort. Similar to vergence-divergence and accommodation conflicts, discomfort is more likely to arise as virtual content is rendered farther away from the focal plane.

[0124] Stereoscopic 3D displays may render virtual content that appears in front of or behind the image plane by applying binocular disparity to the images displayed to the left and right eyes. However, after extended viewing of such stereoscopic 3D displays, many users experience visual discomfort such as eye strain, headaches, and nausea. The further away the content is displayed from the physical image plane / focal plane, the more likely these symptoms of visual discomfort are to occur.

[0125] Vergence and accommodation conflicts that occur in stereoscopic 3D displays are related to the visual discomfort associated with them. Vergence eye movements ensure that both lines of sight of the user's eyes are directed at the object of interest. This causes the eyes to cross when looking at nearby objects and to be parallel when the object is viewed at infinity. Near and distant objects also require the eyes to individually adjust the power of their lenses to maintain a sharp image (a process known as accommodation). In real-world viewing, vergence and accommodation work together, with near vergence correlating with strong accommodation and far vergence correlating with a focus at infinity. However, in stereoscopic 3D displays, this coupling is disrupted, as the image is always sharpest when the eyes remain focused on the depth of the physical display, regardless of vergence and the apparent depth of the content.

[0126] With larger convergence-divergence / accommodation conflicts, geometric distortions are also more likely to occur. This occurs when left and right eye images are rendered from rendering camera positions (i.e., the viewing positions at which the images are constructed / rendered) that do not properly align with the respective aiming centers of the user's eyes (e.g., in some embodiments, the aiming centers of the eyes may include the center of the eye or portions thereof, such as the cornea, iris, lens, retina, etc., of the eye, as represented in a two-dimensional or three-dimensional model of the eye). Distortion of the geometry of the rendered scene may force each eye to make unnatural eye movements in their attempt to stay pointed at the content of interest. For example, when the rendering camera position is offset vertically, the eyes make vertical movements in opposite directions. Such vertical convergence-divergence movements are unnatural because they never occur in real-world viewing. Inducing a vertical separation of the two lines of vision as small as 8-10 minutes of arc can cause discomfort within a few seconds of viewing. Depending on the geometric distortion introduced, different symptoms may be experienced and may be as severe as motion sickness.

[0127] 3A diagrammatically depicts a user's eye 310 viewing a display 312 when a rendering camera 310A is co-located with the center of aim of the eye 310. As a result, an image 314 on the display 312 is aligned with the line of sight 316 of the eye 310. Content 318 can be rendered with binocular disparity (i.e., closer or further away from the display 312) to appear in a manner similar to real-world objects to the eye 310.

[0128] 3B diagrammatically depicts a user's eye 310 viewing a display 312 when a rendering camera 310B is misaligned from the eye's center of aim. As a result, an image 314' on the display 312 is misaligned with the eye's natural line of sight 316 (see FIG. 3A ). When the rendering camera 310B is misaligned with the eye's center of aim, the eye is forced to rotate (322) and reposition the line of sight 316' in a way that would never occur when viewing the real world, which may cause discomfort.

[0129] Motion range for XR systems An important operating parameter of a 3D display system (e.g., an XR system) is how far away from the focal plane content can be presented and still be comfortably viewed. This operating parameter, which may be called the "operating range" of the display system, relates to the amount of binocular disparity that is tolerable to a user. Accurately determining / estimating the operating range facilitates the design of XR systems, applications, and content that are comfortable to use and consume.

[0130] A comfortable operating range for an XR system may be determined based, at least in part, on the accuracy / error of the estimate of the eye aiming center / rendering camera position. In some embodiments, the determined operating range is used to constrain the range of stereoscopic depth using, for example, clip planes in front of and behind the focal plane (implemented in software). In other embodiments, the determined operating range is used to provide a guideline to the content creator as to where to comfortably position 3D objects in the virtual image.

[0131] FIG. 4 diagrammatically depicts the inputs for determining the operating range for an XR system. Visual comfort and perceived accuracy are inputs related to the desired user experience. These inputs determined the maximum acceptable spatial rendering error. The expected aiming center / rendering camera position error is the limit for the spatial rendering accuracy. The expected aiming center / rendering camera position error and the maximum acceptable spatial rendering error together determine the operating range (and focal plane location). Assuming the system / user's maximum tolerable spatial rendering error is known, the operating range can be calculated by determining or estimating the error in the aiming center / rendering camera position.

[0132] The error of the location of the content in the display (i.e., spatial rendering error, compare FIGS. 3A and 3B) is related to the position of the rendering camera relative to the aiming center or pupil center of the human eye. The rendering camera position for users of 3D (three-dimensional) wearable and other stereoscopic displays can be determined / estimated using eye tracking techniques and / or methods that require the user to perform a visual task. However, even when these methods are used, a residual error in the rendering camera position remains for most users. This residual error can be understood at a population level using the error distribution of a representative group of users collected in a ground truth test setting. With this known population level of rendering camera position error, a comfortable operating range given the focal plane location can be calculated. In various embodiments, the VR / AR / MR / XR smart glasses, goggles, or headset, 3D wearable or stereoscopic device, stereoscopic display, or any electronic device that presents virtual content to a user may be collectively referred to as a wearable electronic device or a wearable electronic stereoscopic display device.

[0133] Most commercially available 3D display headsets do not use eye tracking or visual tasks to determine the aiming center / rendering camera position. In some cases, the headset does not even allow the user to set its interpupillary distance (IPD), which may provide a very rough estimate of the aiming center / rendering camera position. Even in these cases, position errors may still result from using population IPD data to determine a very rough aiming center / rendering camera position.

[0134] 5 diagrammatically depicts determining / estimating aiming center / rendering camera position using user input (e.g., via a user interface). The aiming center of the eye can be found using a task in which target dots (e.g., 502, 504, and / or 506) on a display are moved until they visually align with an array of holes (e.g., 508, 510, and / or 512) in front of the display 522. The observer looks through each of the holes (e.g., 508, 510, and / or 512) in turn with one eye, and moves the dots (e.g., 502, 504, and / or 506) on the display 512 until they are visible through the holes (e.g., 508, 510, and / or 512). The aiming center 520 is where all the lines (e.g., 514, 516, and / or 518) connecting the holes (e.g., 508, 510, and / or 512) and their corresponding dots (e.g., 502, 504, and / or 506) intersect.

[0135] Vertical Misalignment and Operating Range for XR Systems Vertical binocular misalignment is expressed in terms of binocular vertical disparity in arc minutes. Figures 6A and 6B diagrammatically depict a user's eyes viewing a display without (Figure 6A) and with (Figure 6B) vertical binocular misalignment. Vertical binocular misalignment becomes noticeable at approximately 8-10 arc minutes (see Figures 3A and 3B). A comfortable operating range can be calculated using this value (i.e., 8-10 arc minutes) and the known accuracy of the aiming center / rendering camera position estimate.

[0136] For example, if the accuracy of the aiming center / rendering camera position estimate is 1 mm (i.e., for 95% of users, the error is 0.5 mm or less) and a worse scenario is when the aiming center is misestimated with one rendering camera position 1 mm higher than the aiming center and the rendering camera for the other eye 1 mm lower than the aiming center, then the maximum working range for an XR system with a focal plane at 74 cm (1.35 dpt) is +1.35 diopters to -1.35 diopters, corresponding to a working range of 37 cm to infinity. The relationship between working range, focal plane, and accuracy of the aiming center estimate can be generically described as follows: Higher accuracy in terms of mm means lower error, resulting in a larger working range in terms of diopters. Changing the position of the focal plane alters the working range, which is measured nonlinearly in units of distance, because the focal plane, measured in diopters, varies nonlinearly with distance from the eye.

[0137] FIG. 6A shows two rendering cameras, one for the left eye (Cam_L) 610′ and one for the right eye (Cam_R) 611′, which draw an image of a target object (M) 618 on a display 612. For simplicity, the geometry is described only for the right eye 611′. The stereoscopic display 612′ is at a certain distance from the rendering camera 611′. The position of the target object M 618 on the display 612 is determined by projecting a ray 616′ from the camera 616′ through the target object M 618 to the display 612. Where the display 612 and the ray 618 intersect, a target image T 614′ is drawn on the display 612 based on the right eye rendering camera 611′.

[0138] FIG. 6B shows the user's eyes 610, 611 looking at the same target object M618 as depicted on the display 612 using the rendering cameras 610′, 611′ as in FIG. 6A. The eyes 610, 611 are vertically offset from the cameras 610′, 611′. The right eye Eye_R 611 is aligned to the target location T′ 614 on the screen through the target object M618 and therefore looks upwards to see the target image T614′. The gaze direction 616 is shown as a solid red line. A dashed line 616″ from Eye_R 611 through M618 to the display 612 shows the correct location of the target on the screen 612 (T′ 614) where it should be. The vertical error for the right eye is the angle alpha (T, Eye_R, T′=622). Assuming that the geometry of the eyes is symmetric here, the total vertical binocular misalignment is 2 x angle alpha. The binocular misalignment in the diagram is shown only for objects in front of the focal plane. The misalignment is symmetric around the focal plane in diopters, such that an object at z diopters in front of the focal plane leads to an identical binocular misalignment at z diopters behind the focal plane.

[0139] Modifying virtual content based on motion range Developer Guidelines In some embodiments, developer guidelines can be developed / modified based on the determined motion range. For example, developers can be advised to keep 3D content within a certain distance from the focal plane based on the determined motion range.

[0140] Clipping Planes In some embodiments, the XR system can clip the content when it is outside the determined motion range (e.g., beyond the farthest value within the determined motion range). Figure 7 is a flow chart depicting a method 700 of modifying virtual content before display based on rendering position error.

[0141] In step 712, the XR system obtains the rendering position error. In some embodiments, the XR system estimates the rendering position error by using population data, IPD data, system specific data, user specific data, etc. The user specific data may be obtained from a database, entered by the user using a user interface, and / or obtained from the eye tracking subsystem.

[0142] In step 714, the XR system determines the motion range from at least the rendering position error as described herein.

[0143] In step 716, the XR system modifies the virtual content based at least on the determined motion range. For example, the XR system may remove portions of the virtual content that correspond to depths that are outside (e.g., beyond) the motion range from the focal plane of the XR system. In other embodiments, the XR system may configure portions of the virtual content that correspond to depths that are outside (e.g., beyond) the motion range from the focal plane of the XR system to be displayed monocularly instead of binocularly.

[0144] In step 718, the XR system displays the modified virtual content for the user.

[0145] The methods described herein modify the virtual content based on the rendering position error to prevent binocular depth cues from deviating beyond a comfortable operating range.

[0146] Dynamic clipping planes based on estimated aim center / rendering camera position In some embodiments, the virtual content can be modified based on the rendering camera position error based on a real-time dynamic estimate of the rendering camera error. In some embodiments where the rendering camera position is estimated using eye tracking, the aiming center / rendering camera position signal can be noisy. Such a noisy rendering camera position signal is likely to involve insufficient accuracy in aiming center estimation. To maintain visual comfort in such embodiments, the binocular operating range can be dynamically increased or decreased based on the estimate of the noise. Similarly, if the eye tracking input is interrupted for any reason, the XR system can revert to a default small binocular operating range.

[0147] In embodiments where the rendering camera position is estimated using eye tracking, an individual's aim center error can be estimated while using the XR system in real time, and the virtual content can be modified based on the determined range of motion accordingly. This would be an individual-level application of the concepts disclosed herein in addition to, or instead of, a population-level application of those concepts.

[0148] In other embodiments, XR systems without an eye tracking subsystem implement a dynamic operating range using a separate signal that indicates a change in the aiming center relative to the headset. Some XR systems ask the user to perform a visual calibration at the beginning of a user session, which is used to calculate the aiming center. However, during use, the device may slide relative to the user's head, thereby changing the position of the aiming center relative to the device and therefore the rendering camera. The XR system may detect the slide using, for example, an accelerometer, and then account for the change by making the binocular operating range smaller. The XR system may make estimated changes to the operating range based on slide data from a user study, or the XR system may apply a general reduction to the operating range when a slide is detected.

[0149] In some embodiments, the XR system reduces the binocular motion range as a function of motion time (e.g., sustained motion).

[0150] Adjusting the operating range based on the characteristics of the user's eyes In some embodiments, when the XR system uses a tracking subsystem to estimate the aiming center / rendering camera position, the accuracy of the position estimate may be affected by the shape of the eye's cornea. The shape of the user's cornea may deviate from that of a "normal eye" when the user suffers from keratoconus. As a result, the eye tracking subsystem may not be able to produce an accurate estimate of the aiming center / rendering camera position. By using information about the user's eyes, the XR system may adjust the operating range of both eyes to accommodate expected errors resulting from corneal abnormalities. Information about the user's eyes may be detected by the camera or it may be typed into the XR system by the user.

[0151] System Architecture Overview 8 is a block diagram of an exemplary computing system 800 suitable for implementing certain embodiments of the present disclosure. The computer system 800 includes a processor 807, a system memory 808 (e.g., RAM), a static storage device 809 (e.g., ROM), a disk drive 810 (e.g., magnetic or optical), a communication interface 814 (e.g., a modem or Ethernet card), a display 811 (e.g., CRT or LCD), an input device 812 (e.g., a keyboard), and a bus 806 or other communication mechanism for communicating information that interconnects subsystems and devices such as a cursor control device.

[0152] According to one embodiment of the invention, computer system 800 performs specific operations by processor 807 executing one or more sequences of one or more instructions contained in system memory 808. Such instructions may be read into system memory 808 from another computer-readable / usable medium, such as static storage device 809 or disk drive 810. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or a portion of the present disclosure.

[0153] As used herein, the terms "computer-readable medium" or "computer usable medium" refer to any medium that participates in providing instructions to the processor 807 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as the disk drive 810. Volatile media include dynamic memory, such as the system memory 808.

[0154] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card with a pattern of holes, paper tape, any other physical medium from which a computer can read, a RAM, a PROM, an EPROM, a flash EPROM (e.g., NAND flash, NOR flash), any other memory chip or cartridge, or any other medium.

[0155] In some embodiments of the present disclosure, execution of sequences of instructions for practicing the present disclosure is performed by a single computer system 800. According to other embodiments of the present disclosure, two or more computing systems 800 coupled by a communications link 815 (e.g., a LAN, PTSN, or wireless network) may cooperate with each other to perform the sequences of instructions required to practice the present disclosure.

[0156] Computer system 800 may transmit and receive messages, data, and instructions, including programs, i.e., application code, through communications link 815 and communications interface 814. Received program code may be executed by processor 807 as it is received, and / or stored in disk drive 810 or other non-volatile storage for later execution. A database 832 in storage medium 831 may be used to store data accessible by system 800 via data interface 833.

[0157] Certain aspects, advantages, and features of the present disclosure are described herein.It should be understood that not all such advantages can be achieved according to any particular embodiment of the present disclosure.Thus, the present disclosure can be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein, without necessarily achieving other advantages as taught or suggested herein.

[0158] The embodiments are described in conjunction with the accompanying drawings. However, it should be understood that the figures are not drawn to scale. Distances, angles, etc. are merely illustrative and do not necessarily bear a strict relationship to the actual dimensions and layout of the devices shown. In addition, the foregoing embodiments have been described at a level of detail to enable those skilled in the art to make and use the devices, systems, methods, and equivalents described herein. A wide variety of variations are possible. Components, elements, and / or steps may be altered, added, removed, or rearranged.

[0159] The devices and methods described herein can advantageously be implemented, at least in part, using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. A software module can include computer executable code stored in a computer's memory to perform the functions described herein. In some embodiments, the computer executable code is executed by one or more general-purpose computers. However, those skilled in the art will understand in light of this disclosure that any module that can be implemented using software to be executed on a general-purpose computer can also be implemented using a different combination of hardware, software, or firmware. For example, such modules can be implemented entirely in hardware using a combination of integrated circuits. Alternatively, or in addition, such modules can be implemented entirely or partially using a specialized computer designed to perform the specific functions described herein, rather than by a general-purpose computer. In addition, where a method is described that is or can be performed, at least in part, by computer software, it should be understood that such a method can be provided on a non-transitory computer-readable medium that, when read by a computer or other processing device, causes it to perform the method.

[0160] While certain embodiments are explicitly described, other embodiments will be apparent to those of ordinary skill in the art based on this disclosure.

[0161] The various processors and other electronic components described herein are suitable for use with any optical system to project light. The various processors and other electronic components described herein are also suitable for use with any audio system to receive voice commands.

[0162] FIG. 9 illustrates a flow diagram for determining the operating range of binocular perception of virtual content exhibiting vergence-divergence-accommodation conflicts in one or more embodiments. In these embodiments, an aiming center may be determined for one or both eyes of a user wearing a wearable electronic device, at 902. Once the aiming center of an eye has been determined, at 902, an error may be estimated for the aiming center, at 904. For example, the wearable electronic device or a remote computing device (e.g., a server) operably connected to the wearable electronic device may estimate the rendering position error or accuracy, the system error or accuracy of the wearable electronic device, the residual error or accuracy at a population level of multiple users, in addition to, for example, the rendering position error and / or system error, or any other suitable error or accuracy that characterizes the accuracy and / or deviation of the aiming center determined from the ground truth. In some embodiments, the error estimated at 904 includes at least one of a rendering camera position error or accuracy specific to a user and a wearable electronic device, a system level error or accuracy specific to the wearable electronic device, or a population level residual error for multiple users.

[0163] Characteristics of one eye (or both eyes, in embodiments in which separate aiming centers are determined for both eyes) may optionally be identified at 906. For example, health characteristics such as keratoconus represent conditions that cause the cornea to develop a distorted shape resulting in blurred vision, myopia (e.g., nearsightedness) that causes the cornea to be too large, hyperopia (e.g., farsightedness) that causes the cornea, and therefore the eye, to be too short, astigmatism that causes the eye to resemble a football shape, or any other characteristic that may affect the shape or geometry of the eye such that the shape or geometry of the eye deviates from the general assumption of a spherical shape. In some embodiments, eye characteristics may be provided by a user. In some other embodiments, eye characteristics may be determined or detected by using one or more inwardly facing devices (e.g., one or more cameras) having the user's eye (or both eyes) within their respective fields of view, and these one or more inwardly facing devices may, for example, capture data regarding the eye of interest from one or more perspectives (e.g., multiple images captured from different angles of at least a portion of the eye) and use the captured data to determine whether, how, and / or by how much the eye deviates from a spherical shape.

[0164] In some embodiments where one or more characteristics of the eye are identified at 906, the aiming center determined at 902 may optionally be adjusted to an adjusted aiming center based at least in part on those one or more characteristics at 908. In the preceding examples where one or more inwardly pointing devices captured data regarding the user's eye and determined whether, how, and / or the amount the eye deviates from a spherical shape, the aiming center estimated at 904 may be adjusted based at least in part on the data regarding whether, how, and / or the amount the eye deviates from a spherical shape.

[0165] A focal length of a focal plane or a motion range for a focal plane of a wearable electronic device may be determined at 910 based, at least in part, on the error determined at 904 and one or more criteria, factors, or characteristics related to binocular vergence and accommodation of virtual content (e.g., virtual content presented to a user by a wearable electronic device described herein). The motion range may include a binocular motion range and may be determined to address horizontal binocular misalignment, vertical binocular misalignment, and / or vergence-accommodation conflicts, etc., which may be common in perceiving virtual content when a user's brain receives mismatch cues between ocular vergence and accommodation above a certain threshold (e.g., vergence-accommodation above a first threshold arc minute or below a second threshold arc minute, etc.) to provide a comfortable user experience in perceiving the virtual content. In some of these embodiments, the operating range may be determined at 910 further based in part on the error or precision for one or more characteristics optionally identified at 906.

[0166] The virtual content to be rendered according to the aforementioned focal plane or focal length may be adjusted at 912 to an adjusted virtual content based, at least in part, on the operating range determined for the focal plane or focal length at 910. For example, the portion of the virtual content to be rendered beyond the determined operating range may be suppressed, clipped, blurred, or removed, for example, by using a blending or clipping plane. As another example, the portion of the virtual content to be rendered beyond the determined operating range may be rendered for a monocular display or the like to reduce or mitigate vergence-accommodation conflicts. The adjusted virtual content may then be presented at 914, for example, by projecting a light beam corresponding to the adjusted virtual content to the user's eye using one or more microprojectors or optical fibers of the wearable electronic device.

[0167] 10 illustrates a flow diagram for dynamically and in real-time updating the center of sight of a user wearing a wearable electronic device in one or more embodiments. In these one or more embodiments, it may be determined at 1002 whether a rendering position error is required or desired to be updated. For example, such a determination may be made based at least in part on a threshold time period since the last update on the rendering position error. As another example, such a determination may be made when a position, movement, or motion sensor (e.g., an accelerometer, a movement sensor, a motion sensor) detects a threshold magnitude of movement, movement, displacement, or change in position, etc. (e.g., acceleration, movement, displacement, etc.) of the wearable electronic device relative to the user wearing it.

[0168] The error may be determined with respect to the aiming center of the user's eyes at 1004. In some of these embodiments, the rendering position error corresponds to the aiming center error and may be determined on an individual user basis, for example, by using an eye tracking module to estimate the aiming center, i.e., center location, of the user's eyes. Images may be presented to the user's left and right eyes, for example, by separate projectors.

[0169] The left and right projectors are located at their respective rendering camera positions and project image data or signals (e.g., light beams) to the assumed aiming center of the user's eye. Nevertheless, rendering position errors may be caused by respective misalignments (e.g., horizontal and / or vertical misalignments) between the aiming center assumed by the rendering device based on the estimated aiming center of the user's eye and the true aiming center of the user's eye, and are therefore determined accordingly.

[0170] In some embodiments, the rendering position error may be determined on an individual basis for a user based on an estimated aiming center determined, at least in part, by using an eye tracking module, by performing a visual task that determines the aiming center, the interpupillary distance of the user, or one or more characteristics of the user's eyes, or any other suitable individual factor or characteristic of the user. In some of these embodiments, the rendering position error may further be statistically determined or adjusted at a population level. For example, the rendering position error or a component thereof (e.g., a residual error component) may be determined at a population level, for example, by using an error distribution of a representative group of users collected in a ground truth test setting.

[0171] In some of these embodiments, the existing motion range (e.g., the motion range determined according to one or more embodiments illustrated in FIG. 9 and described above) may optionally be adjusted at 1006 to a smaller motion range when it is determined that the rendered position error is required or desired to be updated at 1002. Note that the adjustment at 1006 may be an optional implementation in which the wearable electronic device first adjusts the currently effective motion range to the smaller adjusted motion range (e.g., by reducing the motion range, by setting the motion range to a predetermined minimum motion range) such that discomfort resulting from a lack of updates to the rendered position error may be reduced, and that this adjustment is performed prior to the determination of the aim center error at 1010 and the adjustment of the motion range at 1012 (or to a smaller motion range if the adjustment is performed at 1006 in some embodiments).

[0172] In some of these embodiments, a signal indicative of a change in relative position, movement, or motion between the wearable electronic device and a user wearing the wearable electronic device is received at 1008. In some embodiments, such a signal may be generated, for example, by an accelerometer, a movement sensor, a motion sensor, or any other sensor, device, and / or routine that may determine the relative position, movement, or room for motion between the user (e.g., one or more anchor points, such as one or more points used by the wearable electronic device to align its position relative to the user) and the wearable electronic device.

[0173] The aiming center may then be determined at 1010 based at least in part on the signal received at 1008. In some embodiments, the aiming center may be determined anew at 1010 (e.g., by using an eye tracking module or by a vision task described herein) after receiving the signal at 1008. In some other embodiments, the aiming center may be determined by modifying an existing aiming center based at least in part on the signal. For example, in some embodiments where the signal indicates a change in the relative position between the user and the wearable electronic device, the aiming center may be determined based on the detected change.

[0174] As another example, in some embodiments where the signal indicates the presence of acceleration over a duration, a resultant force (e.g., a two-dimensional or three-dimensional force vector on a plane parallel to a plane connecting the two aiming centers of the eye) imparted on the wearable electronic device may be determined, and a relative movement of the wearable electronic device with respect to the user may be predicted by using said resultant force and duration. An existing aiming center may then be determined based on the predicted relative movement. Once the aiming center is determined, an aiming center error may be determined at 1010 based at least in part on the signal by using techniques described herein.

[0175] Once the aiming error is determined at 1010, the operating range may be adjusted at 1012 (e.g., increasing or decreasing the existing operating range) based at least in part on the aiming center error (or in some embodiments, an optional adjustment of the smaller operating range is performed at 1006). For example, the signal received at 1008 may indicate a slippage of the wearable electronic device relative to the wearing user. The corresponding aiming center error may then be determined to increase. Due to the increase in the aiming center error, the diopters for the focal plane (e.g., the focal plane at 74 cm in the above example) may increase (e.g., from + / - 1.35 diopters to + / - 1.45 diopters), thus reducing the operating range from the original 37 cm to infinity to 35 cm (e.g., the focal length is now 100 / 1.45 to 70 cm, thus the operating range is now reduced to 35 cm to infinity).

[0176] It should be noted that the signal does not necessarily indicate at all times that the aiming center is shifted in a manner that reduces the operating range. In some embodiments, the signal may indicate that a change in the position, movement, or motion of the wearable electronic device relative to the user actually moves the aiming center closer to the true aiming center. In these embodiments, the operating range may thus be increased at least in part based on the signal. It should be further noted that the various embodiments illustrated in FIG. 10 may be implemented periodically (e.g., once every fixed duration), dynamically, or in real time, with or without reference to a threshold time period for the performance of these techniques.

[0177] 11 illustrates a block diagram with further details on adjusting the operating range at 1012 of FIG. 10 in some embodiments. In these embodiments, a rendering camera position may be determined for a user at 1102 by using an eye tracking module or by performing a visual task to find the rendering camera position. An estimated aiming center error or accuracy may be determined at 1104 based at least in part on the rendering camera position determined at 1102 by using techniques described herein.

[0178] A residual error for the aim center position or rendering camera position may be determined at a population level at 1106. For example, the rendering position error or a component thereof (e.g., a residual error component) may be determined at a population level, for example, by using an error distribution of a representative group of users collected in a ground truth test setting. A focal length or a focal plane corresponding to a focal length that corresponds to the ground truth for a plurality of users may be identified at 1108. A focal plane or focal length is identified at 1108 because an operating range will be determined based on a diopter that corresponds to a particular focal plane or focal depth, length, or distance.

[0179] An adjusted operating range may be determined for the focal plane or focal length, in some embodiments, based at least in part on the estimated aim center error or accuracy, at 1110. In some of these embodiments, the adjusted operating range may be determined for the focal plane or focal length, further based at least in part on the residual error, at 1110. Once the adjusted operating range is determined at 1110, the virtual content to be presented for the focal plane (or focal length, distance, or depth) may be modified at 1112 to a modified virtual content based at least in part on the adjusted operating range.

[0180] FIG. 12 illustrates a block diagram with further details regarding determining a user's eye aiming center at 902 of FIG. 9 in some embodiments. More specifically, FIG. 12 illustrates a block diagram for determining an eye aiming center using a vision task. In these embodiments, an object having a first hole at a first location and a second hole at a second location is presented to the user's eye at 1202. In some of these embodiments, the object includes a virtual object that includes multiple through holes such that the user can see through the multiple holes to see the real world environment or another virtual object behind the object.

[0181] In some other embodiments, rather than presenting an object to a user with a through-hole, a first marker (e.g., a crosshair, a dot, a regular or irregular shape, a pattern, etc.) may be presented to the user's eye at a first location and a second marker may be presented to the user's eye at a second location. Thus, references to "hole" may be used synonymously with references to "marker" in various embodiments illustrated in FIG. 12 and described herein.

[0182] A target of any shape or pattern may be presented to the user's eye at 1204 and may have a size (e.g., a shape is considered to have a size) or may not have any size (e.g., a dot is considered to have no size). In some of these embodiments, the rendering depth of the target is greater than that of the hole in the object or marker. In some other embodiments, the rendering depth of the target may be equal to that of the hole in the object or marker. In still other embodiments, the rendering depth of the target may be less than that of the hole in the object or marker. One of the purposes of the target and hole (or marker) is to align the hole (or marker) with the target to establish a reference line along which the center of sight of the eye resides. Thus, whether the target is at the same depth as the hole (or marker), a shallower depth, or a deeper depth may be determined based at least in part on the type or types of hole and target selected.

[0183] The target may be moved around the first hole by either the user with input (e.g., by using a physical or virtual pointing device, the up, down, left, and / or right arrow keys on a physical or virtual keyboard, gestures, or interaction with an on-display widget for moving the target, etc.) at 1206. In some embodiments where the target is rendered at a longer focal length or depth than the first hole in the object, the object occludes the target until it is moved to a location where at least a portion of the target is visible by the user's eye through the first hole.

[0184] When the user's eye perceives at least a portion of the target through the hole, a signal may be received at 1208 indicating that at least a portion of the target is visible by the eye through the first hole. For example, the user may issue a visual, audible, or other type of command or instruction to indicate that at least a portion of the target is visible by the user's eye through the first hole. In response to receiving the signal, the wearable electronic device is made to recognize that the target and the first hole are aligned within a certain tolerance (e.g., the alignment precision depends on the size of the first hole, the size of the target, or both).

[0185] A smaller target and / or a smaller hole may provide a high accuracy for the alignment between the target and the hole, but may be less perceptible by the eye vision. On the other hand, a larger target and / or a larger hole may provide better visibility, but the alignment error will be larger. A balanced approach may be to use a larger size target and / or a larger hole, and when the target is first visible, the target is moved along the same direction across the hole until the moment when the target is no longer visible by the user's eye. The final alignment position for the moving target is therefore the midpoint between the first point when the target is first visible and the second point when the target is last visible.

[0186] In some embodiments where a marker is used instead of a through hole in the object, the target and marker may be devised in such a way as to provide a visible distinction when any part of the target is not aligned with the marker. For example, the target may be implemented as a crosshair of a first color and the marker may be implemented as another crosshair of a second color different from the first color. During the alignment process, any non-overlapping representation between the target and the marker may be highlighted due to the two contrasting colors. In some embodiments, the wearable electronic device may further enlarge the display area showing the target and the hole (or marker) to provide a closer and therefore enlarged view of the target and the hole (or marker) to achieve higher accuracy. In some embodiments, the target may be rendered in such a way that the target appears smaller than the hole so that the user may decide whether to trigger a first signal, described in 1208, when the gap between the outer edge of the target and the edge of the hole appears uniform.

[0187] In response to receiving the first signal at 1208, a first reference entity (e.g., a line, a plane, etc.) connecting the first hole (or marker) and the target may be determined at 1210. More specifically, the first signal indicates that the target and the hole (or marker) are aligned with each other as perceived by the user's eye. Thus, an assumption may be made that the aiming center of the user's eye may lie along the first reference entity.

[0188] A substantially similar process may be performed for the user's eye by moving the target (or a different target) around the second hole (or second marker). For example, the same target (or second target) may be moved around the second hole (or second marker) at 1212. A signal may then be triggered by the user and received from the target (or second target) becoming visible to the user's eye through the second hole (or aligned with the second marker) at 1214. In response to receiving the second signal, the target (or second target) and the second hole (or second marker) are considered to be aligned with each other as perceived by the user's eye. Therefore, another assumption may be made that the aiming center of the user's eye may also be along the second reference entity. Using the first and second reference entities, the aiming center of the eye may then be determined to be the intersection of the first and second reference entities. A substantially similar process may be performed for the user's other eye.

[0189] Eye Tracking In one or more embodiments, an AR system may track eye pose (e.g., orientation, direction) and / or eye movement of one or more users within a physical space or environment (e.g., a physical room). The AR system may employ information (e.g., captured images or image data) collected by one or more sensors or transducers (e.g., cameras) positioned and oriented to detect the user's eye pose and / or movement. For example, a head-worn component of an individual AR system may include one or more inwardly facing cameras and / or light sources to track the user's eyes.

[0190] As described above, the AR system may track the user's eye posture (e.g., orientation, direction) and eye movements and build a "heat map." A heat map may be a map of the world that tracks and records the time, frequency, and number of eye posture instances directed at one or more virtual or real objects. For example, a heat map may provide information about the virtual and / or real objects that produced the highest number / time / frequency of gazes or fixations. This may further enable the system to understand the user's interest in a particular virtual or real object.

[0191] Advantageously, in one or more embodiments, the heat map may be used in some embodiments for advertising or marketing purposes to determine the effectiveness of an advertising campaign. The AR system may generate or determine a heat map that represents areas in a space to which a user is paying attention. In one or more embodiments, the AR system may render virtual content (e.g., virtual objects, virtual tools, and other virtual constructs, such as applications, features, characters, text, numbers, and other symbols) with optimized positions and / or optical properties (e.g., color, brightness, luminance) based on, for example, eye tracking and / or the heat map.

[0192] Pseudorandom Pattern In one or more embodiments, the AR system may employ pseudorandom noise in tracking eye posture or eye movement. For example, the head-mounted components of each AR system may include one or more light sources (e.g., LEDs) positioned and oriented to illuminate the user's eyes when the head-mounted components are worn by the user. The camera detects light from the light sources returned from the eyes. For example, the AR system may use Purkinje images, e.g., reflections of objects from structures in the eye.

[0193] The AR system may vary parameters of the light emitted by the light source to impart a recognizable pattern on the detected light emitted and thus reflected from the eye. For example, the AR system may pseudo-randomly vary the operating parameters of the light source to pseudo-randomly vary the parameters of the emitted light. For example, the AR system may vary the length (on / off) of the emission of the light source. This facilitates automatic detection of the emitted and reflected light from the light emitted and reflected from the ambient light sources.

[0194] Figure 13 illustrates a schematic implementation of eye tracking instrumentation, simplified for purposes of illustration and explanation, in some embodiments. Figure 14 illustrates an example output of light, patterns, or object reflections from a structure in a user's eye captured by one or more sensors (e.g., one or more photodiodes) in some embodiments.

[0195] As illustrated in FIG. 13 and FIG. 14, in one implementation, a light source (e.g., LED) 10102 is positioned on a frame on one side (e.g., top) of the eye, and a sensor (e.g., photodiode) is positioned on a bottom portion of the frame. The eye can be considered as a reflector. It is noted that since a pair of eyes tend to move in tandem, only one eye needs to be equipped and tracked. The light source 10102 (e.g., LED) is typically turned on and off one at a time (e.g., time slice) to produce a patterned code (e.g., amplitude variation or modulation). The AR system performs autocorrelation of the signal produced by the sensor (e.g., photodiode) to determine a time-of-flight signal. In one or more embodiments, the AR system employs a known geometry of light source (e.g., LED), sensor (e.g., photodiode), and distance to the eye.

[0196] The sum of vectors along with the known geometry of the eye allows for eye tracking. When estimating the position of the eye, the geometry can be represented as two circles layered on top of each other, since the eye has a cornea and an eyeball. Using this system 10100, the vector of the eye's orientation can be determined or calculated without a camera. Also, the eye's center of rotation can be estimated, since the cross section of the eye is circular and the cornea swings through a certain angle. This actually results in a vector distance, not just a ray trace, but due to the autocorrelation of the received signal against the known transmitted signal. The output can be seen as a Purkinje image 1400, as shown in FIG. 14, which can then be used to track the eye's movements.

[0197] In some implementations, the light source may emit light in the infrared (IR) range of the electromagnetic spectrum, and the light sensor may be selectively responsive to electromagnetic energy in the IR range.

[0198] In one or more embodiments, a light beam is emitted toward the user's eye as shown in the illustrated embodiment. The AR system is configured to detect one or more characteristics associated with the interaction of the light with the user's eye (e.g., Purkinje image, range of backscattered light detected by a photodiode, direction of backscattered light, etc.), which may be captured by a photodiode as shown in the illustrated embodiment. One or more parameters of the interaction may be measured by the photodiode. These parameters may then be used to extrapolate characteristics of eye movement or eye posture.

[0199] Eye tracking It should be understood that the concepts outlined with respect to eye tracking may be applied to any of the user scenarios and embodiments described further below. In one or more embodiments, the various user interfaces described below may also be activated / created to return to the detected eye gaze. The principles described herein may be applied to any other part of this disclosure and should not be read as limiting.

[0200] The AR system may, in some embodiments, track the gaze. For gaze tracking, there are three main components: an eye tracking module (pupil detection and corneal center detection), a head tracking module, and a correlation module that correlates the eye tracking module and the head tracking module. The correlation module correlates information between world coordinates (e.g., the position of an object in the real world) and eye coordinates (e.g., eye movements relative to an eye tracking camera, etc.).

[0201] The eye tracking module is configured to determine the center of the cornea and the center of the pupil. Referring first to Figure 15, a schematic diagram of an eye 1502 is illustrated. As shown in Figure 15, a line 1504 is shown passing through the center of the cornea, the center of the pupil, and the center of the eye. The main line 1504 may be referred to as the optical axis.

[0202] FIG. 15 also shows another line of sight 1506 passing through the cornea. The main line may be referred to as the visual axis. As shown in FIG. 15, the visual axis is an inclined line related to the optical axis. It should be understood that the area of ​​the fovea 1508 through which the visual axis 1506 intersects is considered a very dense area of ​​photoreceptors and is therefore crucial for the eye to see the outside world. The visual axis 1506 is typically at a deviation of 1-5° from the optical axis (not necessarily a perpendicular deviation).

[0203] In traditional eye-tracking techniques, one of the main assumptions is that the head is not moving. This makes it easier to determine the visual axis relative to the optical axis for eye-tracking purposes. However, in the context of AR systems, it is expected that the user will be constantly moving their head, and therefore traditional eye-tracking mechanisms may not be viable.

[0204] To achieve this goal, the AR system is configured to normalize the position of the cornea in relation to the system. It should be appreciated that the position of the cornea is very important in eye tracking since both the optical and visual axes pass through the cornea, as shown in FIG. 15 above.

[0205] 16, the AR system includes a world camera system (e.g., a camera mounted on the user's head to capture the surrounding set, the camera moving with the user's head movement) 1604 attached to a wearable AR system 1606. Also, as shown in FIG. 16, the AR system 1606 may further include one or more eye tracking cameras 1608 that track the movement of the eye 1602. Since both cameras (e.g., the eye tracking camera 1608 and the world camera 1604) are moving, the system may take into account both head movement and eye movement. Both head movement (e.g., calculated based on the field of view (FOV) camera 1604) and eye movement (e.g., calculated based on the eye tracking camera 1608) may be tracked to normalize the position of the cornea.

[0206] It should be appreciated that the eye tracking camera 1608 measures the distance from the camera to the center of the cornea. Therefore, the distance to the center of the cornea is normalized to compensate for any changes in how the wearable AR system 1606 moves relative to the eye. For example, with eyeglass movement, there may be a slight rotation and / or translation of the camera away from the cornea. However, the system compensates for this movement by normalizing the distance to the center of the cornea.

[0207] It should be understood that since both the eye tracking camera and the head camera (world camera) are rigid bodies (e.g., the frame of the AR system), any normalization or correction of the eye tracking camera must also be performed on the world camera as well. For example, the same rotation and translation vectors may be applied to the world camera system as well. Thus, this step identifies the relationship (e.g., rotation vector, translation vector, etc.) between the eye tracking system and the head tracking system.

[0208] Once the rotation and / or translation vectors are identified, a calibration step is performed at various depths away from the user. For example, there may be known points that are fixed distances away from the user. The world camera 1604 may measure the distance between the points fixed in space from the user. As discussed above, the location of the center of the cornea is also known based on calculations associated with the eye tracking camera 1608.

[0209] In addition, as discussed above, the relationship between the eye tracking camera 1608 and the world camera is also known (e.g., any translation or rotation vectors). Thus, it should be appreciated that once the location of the target (e.g., a fixed, known point in space) and the location of the cornea are identified, the line of sight (from the cornea to the target) can be readily identified. This information may be used in mapping and / or rendering to accurately depict virtual objects in space in relation to one or more real objects in the physical world.

[0210] More specifically, to determine the relationship between the world camera 1604 and the eye tracking camera 1606, at least two fixed images may be presented to both the eye camera and the world camera, and the differences in the images may be used to calibrate both cameras. For example, if the center of the cornea is known relative to the eye tracking system 1608, the center of the cornea may be determined relative to the world coordinate system 1604 by utilizing the known relationship between the eye camera and the world camera.

[0211] In one or more embodiments, during a calibration process (e.g., during a setup process when a user first receives the AR device, etc.), a first fixed image is captured by the eye camera 1606 and then the world camera 1604. For illustrative purposes, the first image capture performed by the eye camera may be considered an "E" and the first image capture performed by the world camera may be considered a "W". A second fixed image is then captured by the eye camera 1606 and then by the world camera 1604. The second fixed image may be in a slightly different position than the first fixed image.

[0212] The second image capture of the eye camera may be referred to as E' and the second image capture of the world camera may be referred to as W'. Since Z=WXE and Z=W'XE', X may be easily calculated using the two equations above. This information may therefore be used to reliably map points to naturally calibrate the position of the camera in relation to the world. By establishing this mapping information, the line of sight 1506 may be easily determined, which in turn may be used to strategically provide virtual content to the user.

[0213] 17, to detect the center of the cornea using an eye tracking module, the AR system utilizes either one camera with two flashes (e.g., LED lights) or two cameras with one flash each. In the illustrated embodiment, only one flash 1702 is shown in association with the eye 1602 and the eye tracking camera 1606. It should be understood that the surface of the cornea is highly reflective, and thus, if a camera (e.g., an eye tracking camera) is present that tracks the eye, there may be a flash formed on the image plane of the camera.

[0214] Since the 3D position of the LED light 1702 is known, and the line from the camera's image plane to the flash 1710 is also known, a 3D plane is generated that comprises the flash and the image plane. The center of the cornea is located on this generated 3D plane 1704 (represented as a line in FIG. 17). Similarly, if another flash (from another LED light) is used, the two 3D planes will intersect with each other such that the other 3D plane also has the center of the cornea. It should therefore be understood that the intersection of both 3D planes produces a line that holds the center of the cornea. Now the exact point of the cornea within that line can be determined.

[0215] It should be understood that there is a unique location on the line (from the flash to the projector) that satisfies the law of reflection. As is well known in physics, the law of reflection states that when a ray of light reflects off a surface, the angle of incidence is equal to the angle of reflection. This law may be used to find the center of the cornea.

[0216] Referring to FIG. 18, now the distance from the center of the cornea to the origin (e.g., flash 1710) may be determined (r′, not shown). Similarly, the same analysis may be performed for other lines 1804 (from other flashes 1802 to other projectors) to find r″ (e.g., distance from the intersection line to the other line) (not shown). The center of the cornea may be estimated based on the values ​​of r′ and r″ that are closest in value to each other. While the above exemplary embodiment describes two planes, it should be understood that the location of the cornea may be more easily found if more planes are used. This may be accomplished by using multiple LED lights (e.g., more flashes).

[0217] It is important that the eye tracking system produces at least two flashes on the eye. To increase accuracy, more flashes may be produced on the eye. However, additional flashes are produced on the surface of the eye, making it difficult to determine which flash was produced by which LED. To achieve this goal, one LED may be turned on for one frame and another may be turned on after the first one is turned off, rather than reflecting the flashes on each frame at the same time to understand the correspondence between the flashes and the LEDs. This approach may make the AR system more reliable.

[0218] In some embodiments, due to differences caused by refraction, it may be difficult to determine the exact center of the pupil. To detect the center of the pupil, an image of the eye may be captured. To find the pupil, one may move around the center of the image in a "star" pattern radially outward from the center point. Once found, the same process may be performed starting from a point within the pupil to find the edges of the pupil. This information may be used to infer the pupil center. It should be understood that if this process is repeated several times, some centers may be outliers. However, these outliers may be filtered out. However, even with this approach, the center of the pupil may still not be in the correct location due to the refraction principles discussed above.

[0219] Now, referring to FIG. 19, a calibration may be performed to determine the deviation between the visual axis and the optical axis. When calibrating the system, the actual center of the pupil may not be an issue, but it is important to determine the distance between the world and the eye for mapping in the world (e.g., considering the world as being in 2D, for example). Given the pupil center and the image plane, it is important to find a mapping to find the correlation coordinate in the 2D world, as shown in FIG. 19. To achieve this goal, a parabolic mapping may be used to find the corresponding coordinate in the image plane. A sample equation may be used, such as:

[0220] [ka]

[0221] [ka]

[0222] [ka]

[0223] As shown in 1900 of Figure 19, an equation similar to the above may be used to determine (Xs, Ys) from the determined (Xe, Ye), where the total parameters are 12. Each point provides two equations, so at least six points (e.g., a1-a6) may be required to solve this equation.

[0224] Now, once the center of the cornea is known and the location of the target point is known, a line can be drawn from the center of the cornea to the target point. The world camera 1604 has a fixed plane of image capture that can capture images at a fixed point in space. Another target point is then displayed to the person and then an intersection plane that is virtually connected to the world camera is determined.

[0225] The mapping techniques described above, as described in detail above, can be used to determine the corresponding point in that intersection plane. Knowing the center of the cornea, the mapping techniques described above can identify a point on the image plane that is virtually attached to the world camera. Given that all these points are now known, a line of sight can be created from the center of the cornea to a point on the image plane. It should be understood that a line of sight is created separately for each eye.

[0226] 20, an exemplary method 2000 for determining a line of sight is illustrated. First, at 2002, the center of the cornea may be determined (e.g., through the LED triangulation approach described above, etc.). Then, at 2004, the relationship between the eye camera and the world camera may be determined. At 2006, the target location may be determined. Finally, at 2008, mapping techniques may be utilized to create a line of sight based on all the determined information.

[0227] 21 illustrates a block diagram for a method or system for determining the aiming center using eye tracking techniques in some embodiments. In these embodiments, a plurality of first light rays or beams may be emitted from one or more light sources toward the user's eye at 2102. In some of these embodiments, the one or more light sources may include an LED (light emitting diode) light source.

[0228] A first characteristic associated with an interaction of the first light ray or beam with the eye may be detected at 2104 using one or more sensors. In some embodiments, the one or more sensors include a photodiode. The first characteristic may include a return amount, a reflection, or a specific pattern responsive to at least a portion of the first light ray or beam from the user's eye and detected by the one or more sensors. A first gaze direction for the eye may be determined at 2106 based, at least in part, on the first characteristic detected at 2104.

[0229] A plurality of second light rays or beams may be emitted from one or more light sources towards the user's eye at 2108. A second characteristic associated with an interaction of the second light rays or beams with the eye may be detected using one or more sensors at 2110. A second gaze direction for the eye may be determined at 2112 based at least in part on the second characteristic detected at 2110. A aiming center for the eye may be determined at 2114 based at least in part on the first gaze direction and the second gaze direction. It should be noted that a similar process may be performed for the user's other eye in some of these embodiments to determine a corresponding aiming center for the user's other eye.

[0230] Some of these embodiments assume that the user's eyes move in tandem and thus equip only one eye with one or more of the above-mentioned light sources and one or more sensors, rather than both eyes of the user. Some other embodiments equip both of the user's eyes with one or more of the above-mentioned light sources and one or more sensors for each of the user's two eyes.

[0231] In these latter embodiments, the respective aiming centers for each of the user's two eyes may be determined independently, and in some of these embodiments, the respective aiming centers, along with the respective gaze directions, may be used to characterize focal depth, focal plane, convergence / divergence, accommodation, range of motion, etc. with respect to focal plane or focal depth.

[0232] FIG. 22 illustrates a simplified schematic diagram of a wearable electronic device in some embodiments. As illustrated in FIG. 22, the audio transducers may be integrated with the visual components, e.g., each audio transducer is supported from a common frame with the visual components. Alternatively, the audio transducers may be distinct from the frame carrying the visual components. For example, the audio transducers may be part of a belt pack, such as that shown in FIG. 22 (2202).

[0233] As illustrated in Figure 22, the augmented reality system 100 may include a distinct computing component (e.g., a processing subsystem 102 as shown in Figure 22) that is separate from a head-mounted component (e.g., an optical subsystem 100 as shown in Figure 22). The processing subsystem or computing component 102 may take the form of, for example, a belt pack that may be conveniently coupled to a belt or trouser waistband during use. Alternatively, the computing component 102 may take the form of, for example, a personal digital assistant or smartphone type device.

[0234] The computing component 102 may include one or more processors, such as one or more microcontrollers, microprocessors, graphical processing units, digital signal processors, application specific integrated circuits (ASICs), programmable gate arrays, programmable logic circuits, or other circuits that either embody logic or are capable of executing logic embodied in instructions encoded in software or firmware. The computing component 102 may include one or more non-transitory computer or processor readable media, such as volatile and / or non-volatile memory, such as read only memory (ROM), random access memory (RAM), static RAM, dynamic RAM, flash memory, EEPROM, etc.

[0235] The computing component 102 may be communicatively coupled to the head-worn component. For example, the computing component 102 may be communicatively tethered to the head-worn component via one or more wires or optical fibers via a cable with an appropriate connector. The computing component 102 and the head-worn component 100 may communicate according to any of a variety of tethered protocols, such as USB, USB2, USB3, Ethernet, Thunderbolt, and Lightning protocols.

[0236] Alternatively or in addition, the computing component 102 may be wirelessly communicatively coupled to the head-worn component. For example, the computing component 102 and the head-worn component 100 may each include a transmitter and a receiver or transceiver (collectively, a radio) and an associated antenna to establish wireless communication therebetween. The radio and antenna may take various forms. For example, the radio may be capable of short-range communication and may employ a communication protocol such as BLUETOOTH, WI-FI, or some IEEE 802.11 compliant protocol (e.g., IEEE 802.11n, IEEE 802.11a / c).

[0237] 23 shows an example architecture for electronics for an augmented reality device, according to one or more illustrated embodiments. The AR device may include one or more printed circuit board components, such as a left (2302) and a right (2304) printed circuit board assembly (PCBA). As shown, the left PCBA 2302 contains the majority of the active electronics, while the right PCBA 2304 primarily supports the display or projector elements.

[0238] The right PCBA 2304 may include several projector driver structures that provide image information and control signals to the image generation components. For example, the right PCBA 2304 may carry a first or left projector driver structure 2306 and a second or right projector driver structure 2308. The first or left projector driver structure 2306 splices a first or left projector fiber 2310 and a set of signal lines (e.g., piezoelectric driver wires). The second or right projector driver structure 2308 splices a second or right projector fiber 2312 and a set of signal lines (e.g., piezoelectric driver wires). The first or left projector driver structure 2306 is communicatively coupled to the first or left image projector, while the second or right projector driver structure 2308 is communicatively coupled to the second or right image projector.

[0239] In operation, the image projector renders virtual content to the user's left and right eyes (e.g., retinas) via separate optical components, e.g., waveguides and / or compensatory lenses.

[0240] The image projector may include, for example, left and right projector assemblies. The projector assemblies may use a variety of different image formation or fabrication technologies, for example, fiber scanning projectors, liquid crystal displays (LCD), liquid crystal on silicon (LCOS) displays, digital light processing (DLP) displays. When a fiber scanning projector is employed, the image may be delivered through the tip of the optical fiber, along the optical fiber, to be projected therefrom. The tip may be oriented to be fed into a waveguide. The end of the optical fiber with the tip from which the image is projected may be supported to flex or oscillate. A number of piezoelectric actuators may control the oscillation (e.g., frequency, amplitude) of the tip. A projector driver structure provides the image to the individual optical fibers, controls the signals to control the piezoelectric actuators, and projects the image to the user's eye.

[0241] Continuing with the right PCBA 2304, a button board connector 2314 may provide communication and physical coupling to a button board 2316 carrying various user accessible buttons, keys, switches, or other input devices. The right PCBA 2304 may include a right earbud or speaker connector 2318 to communicatively couple audio signals to a right earbud 2320 or speaker of the head worn component. The right PCBA 2304 may also include a right microphone connector 2322 to communicatively couple audio signals from a microphone of the head worn component. The right PCBA 2304 may further include a right shielded driver connector 2324 to communicatively couple shielding information to a right shielded display 2326 of the head worn component. The right PCBA 2304 may also include a board-to-board connector to provide communication with the left PCBA 2302 via its board-to-board connector 2334.

[0242] The right PCBA 2304 may be communicatively coupled to one or more right outward facing or world view cameras 2328, either body or head mounted, and optionally to a right camera visual indicator (e.g., LED) that illuminates to indicate to others when an image is being captured. The right PCBA 2304 may be communicatively coupled to one or more right eye cameras 2332, carried by a head mounted component, positioned and oriented to capture right eye images and enable tracking, detection, or monitoring of the right eye orientation and / or movement. The right PCBA 2304 may optionally be communicatively coupled to one or more right eye illumination sources 2330 (e.g., LED) that illuminate the right eye with a pattern (e.g., temporal, spatial) of illumination to facilitate tracking, detection, or monitoring of the right eye orientation and / or movement, as described herein.

[0243] The left PCBA 2302 may include a control subsystem, which may include one or more controllers (e.g., a microcontroller, a microprocessor, a digital signal processor, a graphical processing unit, a central processing unit, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) 2340, and / or a programmable logic unit (PLU)). The control system may include one or more non-transitory computer or processor readable media that store executable logic or instructions and / or data or information. The non-transitory computer or processor readable media may take various forms, e.g., volatile and non-volatile forms, e.g., read only memory (ROM), random access memory (RAM, DRAM, SD-RAM), flash memory, etc. The non-transitory computer or processor readable media may be formed, for example, as one or more registers of a microprocessor, an FPGA, or an ASIC.

[0244] The left PCBA 2302 may include a left earphone or speaker connector 2336 to communicatively couple an audio signal to a left earphone or speaker 2338 of the head worn component. The left PCBA 2302 may include an audio signal amplifier (e.g., a stereo amplifier) ​​2342 that is communicatively coupled to the driver earphone or speaker. The left PCBA 2302 may also include a left microphone connector 2344 to communicatively couple an audio signal from a microphone of the head worn component. The left PCBA 2302 may further include a left occlusion driver connector 2346 to communicatively couple occlusion information to a left occlusion display 2348 of the head worn component.

[0245] The left PCBA 2302 may also include one or more sensors or transducers that detect, measure, capture, or otherwise sense information about the surrounding environment and / or about the user. For example, an acceleration transducer 2350 (e.g., a three-axis accelerometer) may detect acceleration in three axes, thereby detecting movement. A gyroscope sensor 2352 may detect orientation and / or the pointing or orientation of a magnet or compass. Other sensors or transducers may be employed as well.

[0246] The left PCBA 2302 may be communicatively coupled to one or more left outward facing or world view cameras 2354, body or head mounted, and optionally a left camera visual indicator (e.g., LED) 2356 that illuminates to indicate to others when an image is being captured. The left PCBA may be communicatively coupled to one or more left eye cameras 2358 carried by a head mounted component and positioned and oriented to capture images of the left eye and enable tracking, detection, or monitoring of the orientation and / or movement of the left eye. The left PCBA 2302 may optionally be communicatively coupled to one or more left eye illumination sources (e.g., LED) 2356 that illuminate the left eye with a pattern (e.g., temporal, spatial) of illumination to facilitate tracking, detection, or monitoring of the orientation and / or movement of the left eye, as described herein.

[0247] The PCBAs 2302 and 2304 are communicatively coupled to distinct computing components (e.g., belt packs) via one or more ports, connectors, and / or paths. For example, the left PCBA 2302 may include one or more communication ports or connectors to provide communication (e.g., bidirectional communication) with the belt pack. The one or more communication ports or connectors may also provide power from the belt pack to the left PCBA 2302. The left PCBA 2302 may include power conditioning circuitry 2380 (e.g., DC / DC power converters, input filters) electrically coupled to the communication ports or connectors and operable to condition (e.g., boost, step-down, current smoothing, reduce transient voltages).

[0248] The communications port or connector may take the form of, for example, a data and power connector or transceiver 2382 (e.g., Thunderbolt port, USB port). The right PCBA 2304 may include a port or connector and receive power from the belt pack. The image generation element may receive power from a portable power source (e.g., chemical battery cells, primary or secondary battery cells, ultracapacitor cells, fuel cells), which may be located, for example, within the belt pack.

[0249] As shown, the left PCBA 2302 contains the majority of the active electronics, while the right PCBA 2304 primarily supports the display or projector and associated piezoelectric drive signals. Electrical and / or fiber optic connections are employed across the front, back, or top of the body or head-worn components of the AR system.

[0250] Both PCBAs 2302 and 2304 are communicatively (e.g., electrically and optically) coupled to the belt pack. The left PCBA 2302 includes a power subsystem and a high speed communication subsystem. The right PCBA 2304 handles the fiber display piezoelectric drive signals. In the illustrated embodiment, only the right PCBA 2304 needs to be optically connected to the belt pack. In other embodiments, both the right and left PCBAs may be connected to the belt pack.

[0251] Although illustrated as employing two PCBAs 2302 and 2304, the electronics of the body or head worn component may employ other architectures. For example, some implementations may use a fewer or greater number of PCBAs. Also, for example, various components or subsystems may be arranged differently than illustrated in FIG. 23. For example, in some alternative embodiments, some of the components illustrated in FIG. 23 as residing on one PCBA may be located on the other PCBA without loss of generality.

[0252] In some embodiments, each user may use his or her own individual AR system (generally referred to as an individual AR system in the following discussion). In some implementations, the individual augmented reality systems may communicate with each other. For example, two or more closely located AR systems may communicate with each other. As further described herein, the communication may occur after implementation of a handshaking protocol in one or more embodiments. The AR systems may communicate wirelessly via one or more radios. As discussed above, such radios may be capable of short-range direct communication or may be capable of longer-range direct communication (e.g., without repeaters, extenders, etc.). Additionally or alternatively, indirect longer-range communication may be achieved via one or more intermediate devices (e.g., wireless access points, repeaters, extenders).

[0253] The head-mounted component of the XR system may have one or more "outward facing" cameras (e.g., 2328, 2354). In one or more embodiments, the head-mounted component may have one or more "inward facing" cameras. As used herein, "outward facing" means that the camera captures images of the surrounding environment, not the user wearing the head-mounted component. Notably, an "outward facing" camera may have a field of view that encompasses areas in front, to the left, right, or even behind the user. This is in contrast to an inward facing camera that captures an image of the individual wearing the head-mounted component, e.g., the camera, which faces the user's face and captures the user's facial expressions or eye movements.

[0254] Various exemplary embodiments of the present disclosure are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate the broader applicable aspects of the present disclosure. Various changes may be made to the described disclosure, and equivalents may be substituted without departing from the true spirit and scope of the present disclosure. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process act, or step to the objective, spirit, or scope of the present disclosure. Moreover, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present disclosure. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0255] The present disclosure includes methods that may be performed using the subject devices. The methods may include the act of providing such a suitable device. Such providing may be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, locate, configure, activate, launch, or otherwise act to provide the device needed in the subject method. The methods recited herein may be carried out in any order of the recited events, and in the recited order of events, that is logically possible.

[0256] Exemplary aspects of the present disclosure, together with details regarding material selection and manufacture, have been described above. As for other details of the present disclosure, these can be understood in conjunction with the patents and publications referenced above, and generally understood or understood by those skilled in the art. The same can be true with respect to the method-based aspects of the present disclosure in terms of additional acts as generally or theoretically adopted.

[0257] In addition, although the present disclosure has been described with reference to several embodiments that optionally incorporate various features, the present disclosure should not be limited to those described or shown as possible for each variation of the present disclosure. Various modifications may be made to the described disclosure, and equivalents (whether listed herein or not included for some brevity) may be substituted without departing from the true spirit and scope of the present disclosure. In addition, when a range of values ​​is provided, it is understood that all intervening values ​​between the upper and lower limits of the range, and any other stated or intervening values ​​within the stated range, are encompassed within the present disclosure.

[0258] It is also envisioned that any optional feature of the described variations may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular object includes the possibility that there are multiple identical items. More specifically, as used in this specification and the claims associated herewith, the singular forms "a," "an," "said," and "the" include multiple supports unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the subject article in the above description and in the claims associated herewith. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, this description is intended to serve as a prelude for the use of exclusive language such as "solely," "only," and equivalents, or the use of "negative" limitations in connection with the recitation of claim elements.

[0259] Without the use of such exclusive terms, the term "comprising" in claims associated with this disclosure shall permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim, or the addition of features may be viewed as changing the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein shall be given their broadest possible, commonly understood meaning while maintaining claim legitimacy.

[0260] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the process flows described above are described with reference to a particular ordering of process actions. However, the ordering of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are therefore to be regarded in an illustrative and not a restrictive sense.

[0261] The scope and breadth of the present invention should not be limited to the examples provided and / or this specification, but rather should be limited only by the scope of the claim language associated with this disclosure.

Claims

1. 1. A system for displaying virtual content using a wearable electronic device, comprising: a display device for presenting virtual content to a user; a microprocessor operably coupled to the display device; a memory having a sequence of instructions stored therein that, when executed by the microprocessor, cause the microprocessor to perform a set of actions, the set of actions including: determining, by a wearable electronic device, a center of aim for a first eye of the user wearing the wearable electronic device; estimating an error or accuracy related to the aim center by the wearable electronic device or a remote computing device connected to the wearable electronic device via a network; determining a range of motion for a focal length or a focal plane at the focal length based, at least in part, on the error or the accuracy and criteria for vergence and accommodation of binocular vision of the virtual content using the wearable electronic device; and adjusting virtual content for presentation to the focal plane or the focal length based at least in part on the range of motion; and determining whether the rendering position error or accuracy should be updated based at least in part on one or more criteria; estimating a first error or accuracy relative to the sighting center; adjusting the motion range to a smaller motion range, the smaller motion range including a distance range for the focal plane or the focal length, and the virtual content for the focal plane or the focal length being rendered according to the smaller motion range; Including memory and A system comprising:

2. The memory stores the sequence of instructions that, when executed by the microprocessor, cause the microprocessor to perform the set of actions, the set of actions including: identifying the first eye characteristic of the user, the characteristic related to an eye condition of the user; and adjusting the center of aim based at least in part on the characteristic of the first eye of the user; The system of claim 1 further comprising:

3. The memory stores the sequence of instructions that, when executed by the microprocessor, cause the microprocessor to perform the set of actions, the set of actions including: presenting the adjusted virtual content to the user by using the wearable electronic device to project a light beam related to the adjusted virtual content onto at least the focal plane or the focal length, the error or accuracy includes at least one of a rendering camera position error or accuracy that is specific to the user and the wearable electronic device, a system level error or accuracy that is specific to the wearable electronic device, or a population level residual error for multiple users. The system of claim 1 further comprising:

4. The memory stores the sequence of instructions that, when executed by the microprocessor, cause the microprocessor to perform the set of actions, the set of actions including: receiving a signal indicative of a change in relative position, movement, or motion between the wearable electronic device and the user; determining a boresight center error or accuracy or a rendering camera position error or accuracy based on at least said signal; adjusting the operating range based at least in part on the aim center error or accuracy or the rendering camera position error or accuracy; The system of claim 1 further comprising:

5. The set of acts includes adjusting the operating range based at least in part on the aim center error or accuracy or the rendering camera position error or accuracy, and adjusting the operating range further comprises: determining a rendering camera position for the first eye of the user by executing at least an eye tracking module of the wearable electronic device or by performing a visual task; determining an estimated aim center error or accuracy for the rendering camera position for the first eye of the user; identifying the focal plane or the focal length corresponding to ground truth for a plurality of users; The system of claim 4 , comprising:

6. The set of acts includes adjusting the operating range based at least in part on the aim center error or accuracy or the rendering camera position error or accuracy, and adjusting the operating range further comprises: determining a residual error related to the rendering camera position for the first eye of the user; The system of claim 5 , comprising:

7. The set of acts includes adjusting the operating range based at least in part on the aim center error or accuracy or the rendering camera position error or accuracy, and adjusting the operating range further comprises: adjusting the operating range for the focal plane or the focal length to an adjusted operating range based, at least in part, on the estimated boresight center error or accuracy. The system of claim 5 , comprising:

8. The set of acts includes determining the aiming center for the first eye of the user wearing the wearable electronic device, wherein determining the aiming center includes: presenting a first marker at a first location to the first eye of the user, the first marker comprising a first hole in an object or a first rendered object; presenting a target to the first eye of the user at the focal plane or at the focal distance; The system of claim 1 , comprising:

9. The set of acts includes determining the aiming center for the first eye of the user wearing the wearable electronic device, wherein determining the aiming center includes: moving the target around the first marker; receiving a first signal from the user when the target becomes visible to the first eye of the user; determining a first reference entity connecting the first marker and the target in response to receiving the first signal; The system of claim 8 , comprising:

10. The set of acts includes determining the aiming center for the first eye of the user wearing the wearable electronic device, wherein determining the aiming center includes: presenting a second marker at a second location to the first eye of the user, the second marker comprising a second hole in the object or a second rendered object; presenting the target or a separate target to the first eye of the user at the focal plane or focal length; The system of claim 9 , comprising:

11. The set of acts includes determining the aiming center for the first eye of the user wearing the wearable electronic device, wherein determining the aiming center includes: moving the target or the separate target around the second marker; receiving a second signal from the user when the target or the distinct target becomes visible to the first eye of the user; determining a second reference entity connecting the second marker and the target or the separate targets in response to receiving the second signal; The system of claim 10, comprising:

12. The set of acts includes determining the aiming center for the first eye of the user wearing the wearable electronic device, wherein determining the aiming center includes: determining the center of aim for the first eye of the user based at least in part on the first reference entity and the second reference entity; The system of claim 11 , comprising:

13. 1. A method for displaying virtual content using a wearable electronic device, comprising: determining, by a wearable electronic device, a center of aim for a first eye of a user wearing the wearable electronic device; estimating an error or accuracy related to the aim center by the wearable electronic device or a remote computing device connected to the wearable electronic device via a network; determining a range of motion for a focal length or a focal plane at the focal length based, at least in part, on the error or the accuracy and criteria for vergence and accommodation of binocular vision of virtual content using the wearable electronic device; and adjusting virtual content for presentation to the focal plane or the focal length based at least in part on the range of motion; and determining whether the rendering position error or accuracy should be updated based at least in part on one or more criteria; estimating a first error or accuracy relative to the sighting center; adjusting the motion range to a smaller motion range, the smaller motion range including a distance range for the focal plane or the focal length, and the virtual content for the focal plane or the focal length being rendered according to the smaller motion range; A method comprising:

14. identifying the first eye characteristic of the user, the characteristic related to an eye condition of the user; and adjusting the center of aim based at least in part on the characteristic of the first eye of the user; presenting the adjusted virtual content to the user by using the wearable electronic device to project a light beam related to the adjusted virtual content onto at least the focal plane or the focal length, the error or accuracy includes at least one of a rendering camera position error or accuracy specific to the user and the wearable electronic device, a system level error or accuracy specific to the wearable electronic device, or a population level residual error for multiple users; 14. The method of claim 13, further comprising:

15. determining whether the rendering position error or accuracy should be updated based at least in part on one or more criteria; estimating a first error or accuracy relative to the sighting center; adjusting the motion range to a smaller motion range, the smaller motion range including a distance range for the focal plane or the focal length, and the virtual content for the focal plane or the focal length being rendered according to the smaller motion range; receiving a signal indicative of a change in relative position, movement, or motion between the wearable electronic device and the user; determining a boresight center error or accuracy or a rendering camera position error or accuracy based on at least said signal; adjusting the operating range based at least in part on the aim center error or accuracy or the rendering camera position error or accuracy; 14. The method of claim 13, further comprising:

16. 1. A computer program product comprising a non-transitory machine-readable storage medium having stored thereon a sequence of instructions that, when executed by a microprocessor, cause the microprocessor to perform a set of actions, the set of actions including: determining, by a wearable electronic device, a center of aim for a first eye of the user wearing the wearable electronic device; estimating an error or accuracy related to the aim center by the wearable electronic device or a remote computing device connected to the wearable electronic device via a network; determining a range of motion for a focal length or a focal plane at the focal length based, at least in part, on the error or the accuracy and criteria for vergence and accommodation of binocular vision of the virtual content using the wearable electronic device; and adjusting virtual content for presentation to the focal plane or the focal length based at least in part on the range of motion; and determining whether the rendering position error or accuracy should be updated based at least in part on one or more criteria; estimating a first error or accuracy relative to the sighting center; adjusting the motion range to a smaller motion range, the smaller motion range including a distance range for the focal plane or the focal length, and the virtual content for the focal plane or the focal length being rendered according to the smaller motion range; a computer program product,

17. The non-transitory machine-readable storage medium has stored thereon the sequence of instructions that, when executed by a microprocessor, cause the microprocessor to perform the set of actions, the set of actions including: presenting a first marker at a first location to the first eye of the user, the first marker comprising a first hole in an object or a first rendered object; presenting a target to the first eye of the user at the focal plane or focal length; moving the target around the first marker; receiving a first signal from the user when the target becomes visible to the first eye of the user; determining a first reference entity connecting the first marker and the target in response to receiving the first signal; 17. The computer program product of claim 16, further comprising:

18. The non-transitory machine-readable storage medium has stored thereon the sequence of instructions that, when executed by the microprocessor, cause the microprocessor to perform the set of actions, the set of actions including: presenting a second marker at a second location to the first eye of the user, the second marker comprising a second hole in the object or a second rendered object; presenting the target or a separate target to the first eye of the user at the focal plane or focal length; moving the target or the separate target around the second marker; receiving a second signal from the user when the target or the distinct target becomes visible to the first eye of the user; determining a second reference entity connecting the second marker and the target or the separate target in response to receiving the second signal; determining the center of aim for the first eye of the user based at least in part on the first reference entity and the second reference entity; 20. The computer program product of claim 17, further comprising:

Images