Dynamic display calibration based on eye tracking

Abstract

To provide favorable dynamic display calibration based on eye tracking.SOLUTION: Examples of a light field metrology system for use with a display are disclosed. The light field metrology may capture images of a projected light field, and determine focus depths (or lateral focus positions) for various regions of the light field using the captured images. The determined focus depths (or lateral positions) may then be compared with intended focus depths (or lateral positions), to quantify the imperfections of the display. Based on the measured imperfections, an appropriate error correction may be performed on the light field to correct for the measured imperfections. The display can be an optical display element in a head-mounted display, for example, an optical display element capable of generating multiple depth planes or a light field display.SELECTED DRAWING: Figure 26

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of priority to U.S. Application No. 62 / 250,925, filed November 4, 2015, entitled LIGHT FIELD DISPLAY METROLOGY; U.S. Application No. 62 / 278,779, filed January 14, 2016, entitled LIGHT FIELD ERROR CORRECTION; U.S. Application No. 62 / 250,934, filed November 4, 2015, entitled AUTOMATED CALIBRATION IMAGE PROJECTION AND CAPTURE FOR DISPLAY CALIBRATION; U.S. Application No. 62 / 278,824, filed January 14, 2016, entitled DYNAMIC CALIBRATION OF A DISPLAY BASED ON EYE - TRACKING; U.S. Application No. 62 / 278,794, filed January 14, 2016, entitled CHROMATIC BALANCING A DISPLAY HAVING VARYING CHROMATICITY ACROSS A FIELD OF VIEW, all of which are hereby incorporated by reference in their entirety.

[0002] (Field) The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to metrology systems for measuring and calibrating the optical properties of imaging and visualization systems. The present disclosure also relates to dynamic calibration of virtual reality and augmented reality imaging and visualization systems based on eye - tracking.

Background Art

[0003] (Background) Modern computing and display technologies are facilitating the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or can be perceived as being real. Virtual reality, i.e., "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, while augmented reality, i.e., "AR" scenarios typically involve the presentation of digital or virtual image information as an augmentation of the visualization of the actual world around the user, or mixed reality "MR" is related to the fusion of the real and virtual worlds to create a new environment in which physical and virtual objects coexist and interact in real time. Stated in conclusion, the human visual perception system is very complex, and it is difficult to generate VR, AR, or MR technologies that facilitate a comfortable and natural-feeling rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention Means for Solving the Problems

[0004] (Abstract) An embodiment of an imaging system is a projection device for projecting an image towards a viewer's eye, the image comprising a light field representing light from a virtual object, the virtual object being configured to be projected as if it were located at one or more intended depths of focus, a projection device, and a light field measurement device for measuring an imperfection in the light field. The light field measurement device captures one or more images corresponding to a portion of the light field, analyzes the one or more captured images, identifies one or more perceived depths of focus corresponding to the depth at which a portion of the light field is in focus, creates a depth map at least in part based on the identified depths of focus, and may be configured to compare the created depth map with one or more intended depths of focus. The system can generate a calibration for spatial and / or color imperfections that can be used to dynamically calibrate a wearable display system.

[0005] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, the drawings, and the claims. Neither this summary nor any of the following detailed descriptions purports to define or limit the scope of the subject matter of the invention. This specification also provides, for example, the following items. (Item 1) A display system, an eye-tracking camera, a display, a non-transitory data storage device configured to store a plurality of calibrations for the display, each calibration in the plurality of calibrations being associated with a calibration position for the display, a non-transitory data storage device, a hardware processor that communicates with the eye-tracking camera, the display, and the non-transitory data storage device, Determining the eye position of the user of the display with respect to the display based on information from the eye-tracking camera; Accessing one or more than one of the plurality of calibrations, at least in part, based on the determined eye position; Calculating a correction for application to the display to correct for imperfection within the display, at least in part, based on one or more than one of the plurality of calibrations; Applying the correction to the display; A hardware processor programmed to perform the above; A display system comprising the above. (Item 2) The display system according to item 1, wherein the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more than that. (Item 3) The display system according to item 1, wherein the calibration positions are distributed across the display within a grid. (Item 4) The display system according to item 3, wherein the grid comprises a 2×2, 3×3, 5×5, or 9×9 grid. (Item 5) The display system according to item 1, wherein one or more than one of the plurality of calibrations comprises a calibration associated with the calibration position closest to the eye position. (Item 6) To calculate the correction, the hardware processor is programmed to interpolate or extrapolate between one or more than one of the plurality of calibrations, at least in part, based on the calibration positions of one or more than one of the plurality of calibrations and the determined eye position. The display system according to item 1. (Item 7) The display includes a first display associated with the user's first eye and a second display associated with the user's second eye, and the hardware processor is programmed to determine the user's eye position with respect to the first display and apply the determined eye position to calculate a correction for the second display, the display system according to item 1. (Item 8) The display includes a first display associated with the user's first eye and a second display associated with the user's second eye, and at least some of the plurality of calibrations represent an average calibration for the first display and the second display, the display system according to item 1. (Item 9) The display includes a light field display, the display system according to item 1. (Item 10) The display includes a stackable waveguide assembly including a plurality of waveguides, the display system according to item 1. (Item 11) The display is configured as a head-mounted wearable display system, the display system according to item 1. (Item 12) Each calibration in the plurality of calibrations corrects a spatial imperfection of the display, a color imperfection of the display, or both the spatial imperfection and the color imperfection, the display system according to any one of items 1-11. (Item 13) The spatial imperfection includes one or more of in-plane translation, rotation, scaling, or warping errors, or out-of-plane or depth-of-focus errors, the display system according to item 12. (Item 14) The display system according to item 12, wherein the color imperfection includes one or more of luminance flatness or color uniformity errors associated with colors that can be displayed by the display. (Item 15) A method for calibrating a display, under the control of a dynamic calibration system performed by computer hardware, determining an eye position of a user of the display; accessing calibration for the display, at least in part, based on the determined eye position, wherein the calibration is selected based on an associated calibration position and the determined eye position; calculating a correction for application to the display to correct an imperfection within the display, at least in part, based on the accessed calibration; applying the correction to the display. A method comprising the above steps. (Item 16) The step of accessing the calibration includes selecting one or more calibrations from a plurality of calibrations, each calibration being associated with a different calibration position for the display, according to the method of item 15. (Item 17) The calibration positions are arranged in a grid across the display, according to the method of item 16. (Item 18) The step of calculating the correction includes interpolating or extrapolating between one or more of the plurality of calibrations based on the associated calibration positions and the determined eye position of one or more of the plurality of calibrations, according to the method of item 16. (Item 19) The method according to item 15, further comprising accessing an image of the eyes of the user of the display and determining the eye position, at least in part, based on the image of the eyes. (Item 20) The step of calculating the correction includes the step of correcting the spatial non-uniformity of the display, the color non-uniformity of the display, or both the spatial non-uniformity and the color non-uniformity, according to any one of items 15-19. (Item 21) A wearable display system, An imaging system facing inward, A display, A non-transitory data storage device configured to store a plurality of calibrations for the display, wherein each calibration in the plurality of calibrations is associated with a calibration position for the display, a non-transitory data storage device; A hardware processor that communicates with the inward-facing imaging system, the display, and the non-transitory data storage device, Determining, using the inward-facing imaging system, an eye position of a user of the display with respect to the display; Calculating a correction for application to the display to correct, at least in part, one or more of a spatial non-uniformity within the display or a color non-uniformity within the display, based at least in part on the determined eye position and one or more of the plurality of calibrations; Applying the correction to the display A hardware processor programmed to perform A wearable display system comprising (Item 22) The hardware processor is programmed to apply the correction via a feedback loop that monitors changes in the eye position, according to the wearable display system of item 21. (Item 23) The wearable display system according to item 21, wherein the hardware processor is programmed to determine a change in the eye position with respect to the previous eye position and, if the change exceeds a threshold value, calculate the correction. (Item 24) The wearable display system according to any one of items 21-23, wherein the spatial imperfection comprises one or more of in-plane translation, rotation, scaling, or warping error, or out-of-plane or depth-of-field error. (Item 25) The wearable display system according to any one of items 21-23, wherein the color imperfection comprises one or more of luminance flatness or color uniformity error associated with the colors that can be displayed by the display.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 9E

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 11

Figure 12A

Figure 12B

Figure 13

Figure 14A

Figure 14B

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 19C

Figure 20

Figure 21

Figure 22

Figure 23A

Figure 23B

Figure 24

Figure 25A

Figure 25B

Figure 26

Figure 27

Figure 28

[0007] Throughout the drawings, reference numerals may be reused to indicate corresponding between referenced elements. The drawings are provided to illustrate the exemplary embodiments described herein and are not intended to limit the scope of the present disclosure.

[0008] (Overview) For a three-dimensional (3D) display to provide a true sense of depth, more specifically a simulated sense of surface depth, it is desirable to generate a focusing response corresponding to the virtual depth for each point within the field of view of the display. If the focusing adjustment in response to a display point does not correspond to the virtual depth of that point as determined by the binocular depth cues of convergence and stereopsis, the human eye can suffer from focusing conflict, resulting in unstable imaging, harmful eye strain, headaches, and, in the absence of focusing adjustment information, an almost complete lack of surface depth.

[0009] VR and AR experiences can be provided by a display system having a display that provides an image corresponding to a plurality of depth planes to a viewer. The images can be different for each depth plane (e.g., providing a somewhat different presentation of a scene or object), and are separately focused by the viewer's eyes, thereby serving to provide depth cues to the user based on the eye focusing adjustment required to bring into focus different image features for scenes located on different depth planes and / or based on the observation of different image features on different depth planes that are out of focus. As discussed anywhere herein, such depth cues provide a perception of realistic depth.

[0010] (3D display) Figure 1 depicts an illustration of an augmented reality scenario with a virtual reality object and an actual reality object as viewed by a person. Figure 1 depicts an augmented reality scene 100, where a user of AR technology can see a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of AR technology also "sees" a robotic image 130 standing on the real-world platform 120 and an avatar character 140 in the form of a flying cartoon that appears anthropomorphic like a bumblebee, although these elements do not exist in the real world.

[0011] It is desirable for a three-dimensional (3D) display to generate a perspective adjustment response corresponding to the virtual depth for each point within the field of view of the display in order to generate a true sense of depth, more specifically, a simulated sense of surface depth. If the perspective adjustment response for a display point does not correspond to the virtual depth of that point as determined by both the convergence and stereoscopic binocular depth cues, the human eye will experience a vergence conflict, resulting in unstable imaging, harmful eye strain, headaches, and, in the absence of vergence information, a nearly complete lack of surface depth.

[0012] VR, AR, and MR experiences can be provided by a display system having a display that provides an image corresponding to a plurality of depth planes to a viewer. The images may vary for each depth plane (e.g., providing a somewhat different presentation of a scene or object), are separately focused by the viewer's eyes, thereby based on the eye vergence required to focus on different image features for scenes located on different depth planes and / or based on observing different image features on different depth planes that are out of focus, can help provide depth cues to the user. As discussed anywhere in this specification, such depth cues provide a reliable perception of depth.

[0013] Figure 2 illustrates an example of a wearable display system 200 that can be used to present a VR, AR, or MR experience to a display system wearer or viewer 204. The display system 200 includes a display 208 and various mechanical and electronic modules and systems to support the functions of the display 208. The display 208 may be coupled to a frame 212 that is wearable by a display system user, wearer, or viewer 204 and configured to position the display 208 in front of the wearer 204's eyes. The display 208 may be a light field display. In some embodiments, a speaker 216 is coupled to the frame 212 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / formable acoustic control). The display 208 is operably coupled to a local data processing module 224 and can be mounted in various configurations, such as fixed to the frame 212 by a wired conductor or wireless connection, fixed to a helmet or hat worn by the user, incorporated within headphones, or removably attached to the user 204 in another manner (e.g., in a backpack configuration, in a belt attachment configuration).

[0014] The local processing and data module 224 may include a hardware processor and a non-transitory digital memory such as a non-volatile memory (e.g., flash memory), both of which may be utilized to assist in the processing, caching, and storage of data. The data may include (a) data captured from sensors such as an image capture device (e.g., a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope (e.g., operatively coupled to frame 212 or otherwise attachable to user 204), and / or (b) data obtained and / or processed using remote processing module 228 and / or remote data repository 232, optionally for passage through display 208 after such processing or retrieval. The local processing and data module 224 may be operatively coupled to remote processing module 228 and remote data repository 232 via communication links 236 and / or 240, such as via a wired or wireless communication link, such that these remote modules 228, 232 are available as resources to the local processing and data module 224. Additionally, remote processing module 228 and remote data repository 232 may be operatively coupled to each other.

[0015] In some embodiments, the remote processing module 228 may include one or more processors configured to analyze and process data such as video information and / or image information captured by an image capture device. The video data may be stored locally within the local processing and data module 224 and / or the remote data repository 232. In some embodiments, the remote data repository 232 may comprise a digital data storage facility, which may be available through the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the local processing and data module 224, enabling complete autonomy from the remote modules.

[0016] The human visual system is complex and it is difficult to provide a realistic perception of depth. Although not limited by theory, it is thought that an object viewer can perceive an object in three dimensions due to a combination of vergence and accommodation. The vergence of the two eyes relative to each other (i.e., the rotational movement of the pupils towards or away from each other to converge the lines of sight of the eyes and fix them on an object) is closely associated with the focusing (or “accommodation”) of the eye's lens. Under normal conditions, changing the focus of the eye's lens, or accommodating the eye, to change the focus from one object to another object at a different distance will automatically cause a matching change in vergence at the same distance under a relationship known as the “accommodation-vergence reflex”. Similarly, a change in vergence will, under normal conditions, induce a matching change in accommodation. A display system that provides better matching between accommodation and vergence can form a more realistic or comfortable simulation of a three-dimensional image.

[0017] Figure 3 illustrates an aspect of an approach for simulating a 3D image using multiple depth planes. Referring to Figure 3, objects at various distances from eyes 302 and 304 on the z-axis are focused by eyes 302 and 304 such that those objects are in focus. Eyes 302 and 304 take on a particular focused state and focus objects at different distances along the z-axis. As a result, a particular focused state can be said to be associated with a particular one of depth planes 306 having an associated focal length such that an object or a portion of an object in a particular depth plane is in focus when the eye is in the focused state with respect to that depth plane. In some embodiments, the 3D image may be simulated by providing different presentations of the image for each of eyes 302 and 304 and also by providing different presentations of the image corresponding to each of the depth planes. For purposes of clarity of illustration, the fields of view of eyes 302 and 304 are shown as being separate, but it should be understood that they may overlap, for example, as the distance along the z-axis increases. Additionally, for purposes of ease of illustration, the contours of the depth planes are shown as being flat, but it should be understood that they may be curved in physical space such that all features within a depth plane are in focus with the eye in a particular focused state. Without being limited by theory, it is believed that the human eye can typically interpret a finite number of depth planes in order to provide depth perception. As a result, a highly realistic simulation of the perceived depth can be achieved by providing the eye with different representations of the image corresponding to each of these limited number of depth planes.

[0018] (Waveguide stack assembly) FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. Display system 400 includes a stack of waveguides or a stacked waveguide assembly 405 that can be utilized to provide three-dimensional perception to eye 410 or the brain using a plurality of waveguides 420, 422, 424, 426, 428. In some embodiments, display system 400 may correspond to system 200 of FIG. 2, and FIG. 4 schematically shows some portions of that system 200 in more detail. For example, in some embodiments, waveguide assembly 405 may be integrated within display 208 of FIG. 2.

[0019] Continuing to refer to FIG. 4, waveguide assembly 405 may also include a plurality of features 430, 432, 434, 436 between the waveguides. In some embodiments, features 430, 432, 434, 436 may be lenses. In some embodiments, features 430, 432, 434, 436 may not be lenses. Rather, they may be spacers (e.g., cladding layers and / or structures for forming an air gap).

[0020] Waveguides 420, 422, 424, 426, 428 and / or a plurality of lenses 430, 432, 434, 436 may be configured to transmit image information to the eye with various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 440, 442, 444, 446, 448 may be utilized to input image information into waveguides 420, 422, 424, 426, 428, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits from the output surface of the image input devices 440, 442, 444, 446, 448 and is input into the corresponding input edges of waveguides 420, 422, 424, 426, 428. In some embodiments, a single beam of light (e.g., a collimated beam) is input into each waveguide and outputs an entire field of cloned collimated beams directed toward the eye 410 at a particular angle (and amount of divergence) corresponding to the depth plane associated with the particular waveguide.

[0021] In some embodiments, image input devices 440, 442, 444, 446, 442 are discrete displays that generate, respectively, image information for input into their respective corresponding waveguides 420, 422, 424, 426, 428. In some other embodiments, image input devices 440, 442, 446, 446, 448 are the output ends of a single multiplexed display that may pipe image information, e.g., via one or more optical waveguides (such as an optical fiber cable), to each of image input devices 440, 442, 444, 446, 448.

[0022] Controller 450 controls the operation of the stacked waveguide assemblies 405 and the image input devices 440, 442, 444, 446, 448. In some embodiments, controller 450 includes programming (e.g., instructions in a non-transitory computer-readable medium) that adjusts the timing and provision of image information to waveguides 420, 422, 424, 426, 428. In some embodiments, controller 450 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. Controller 450 may, in some embodiments, be part of processing modules 224 and / or 228 (illustrated in FIG. 2). In some embodiments, the controller may communicate with an inward-facing imaging system 452 (e.g., a digital camera), an outward-facing imaging system 454 (e.g., a digital camera), and / or a user input device 466. The inward-facing imaging system 452 (e.g., a digital camera) captures an image of the eye 410 and can be used, for example, to determine the size and / or orientation of the pupil of the eye 410. The outward-facing imaging system 454 can be used to image a portion of the world 456. The user can input commands to controller 450 via user input device 466 and interact with display system 400.

[0023] Waveguides 420, 422, 424, 426, 428 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 420, 422, 424, 426, 428 may each be planar, or have another shape (e.g., curved), with a major top and bottom surface and an edge extending between their major top and bottom surfaces. In the illustrated configuration, the waveguides 420, 422, 424, 426, 428 each include light extraction optical elements 460, 462, 464, 466, 468 configured to redirect light, propagate it within each individual waveguide, and extract the light from the waveguide by outputting image information to the eye 410 from the waveguide. The extracted light may also be referred to as external coupled light, and the light extraction optical elements may also be referred to as external coupling optical elements. The beam of the extracted light is output by the waveguide at the location where the light propagating within the waveguide impinges on the light redirecting element. The light extraction optical elements (460, 462, 464, 466, 468) may be, for example, reflective and / or diffractive optical features. For ease of explanation and clarity of the drawings, they are shown disposed on the bottom major surface of the waveguides 420, 422, 424, 426, 428, but in some embodiments, the light extraction optical elements 460, 462, 464, 466, 468 may be disposed on the top and / or bottom major surfaces and / or directly within the volume of the waveguides 420, 422, 424, 426, 428. In some embodiments, the light extraction optical elements 460, 462, 464, 466, 468 may be attached to a transparent substrate and formed within a layer of the material forming the waveguides 420, 422, 424, 426, 428. In some other embodiments, the waveguides 420, 422, 424, 426, 428 may be a monolithic piece of material, and the light extraction optical elements 460, 462, 464, 466, 468 may be formed on and / or within the surface of that piece of material.

[0024] Continuing to refer to FIG. 4, as discussed herein, each waveguide 420, 422, 424, 426, 428 is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 420 closest to the eye may be configured to deliver collimated light to the eye 410 as it is input into such waveguide 420. The collimated light may represent an optically infinite focal plane. The next waveguide 422 may be configured to output collimated light that passes through a first lens 430 (e.g., a negative lens) before reaching the eye 410. The first lens 430 may be configured to generate some convex wavefront curvature such that the eye / brain interprets the light resulting from the next waveguide 422 as originating from a first focal plane that is closer inwardly from the optically infinite towards the eye 410. Similarly, the third waveguide 424 passes its output light through both the first lens 430 and a second lens 432 before reaching the eye 410. The combined refractive power of the first and second lenses 430 and 432 may be configured to generate another increment of wavefront curvature such that the eye / brain interprets the light resulting from the third waveguide 424 as originating from a second focal plane that is even closer inwardly from the optically infinite, where the light from the next waveguide 422 was, towards the person.

[0025] Similarly, other waveguide layers (e.g., waveguides 426, 428) and lenses (e.g., lenses 434, 436) are configured to send their output through all of the lenses between them and the eye for the aggregated focus power representing the focal plane closest to the person using the highest waveguide 428 in the stack. When viewing / interpreting light originating from the world 456 on the other side of the stacked waveguide assembly 405, a compensation lens layer 438 may be disposed on top of the stack to compensate for the stack of lenses 430, 432, 434, 436. Such a configuration provides the same number of perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements 460, 462, 464, 466, 468 of waveguides 420, 422, 424, 426, 428 and the focusing sides of lenses 430, 432, 434, 436 may be static (e.g., not dynamic or electrically active). In some alternative embodiments, either or both may be dynamic using electrically active features.

[0026] Continuing to refer to FIG. 4, the light extraction optical elements 460, 462, 464, 466, 468 may be configured to both redirect light from their respective waveguides and output this light using an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides having different associated depth planes may have light extraction optical elements of different configurations that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 460, 462, 464, 466, 468 may be three-dimensional or surface features configured to output light at a specific angle. For example, the light extraction optical elements 460, 462, 464, 466, 468 may be three-dimensional holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published Jun. 25, 2015, which is incorporated herein by reference in its entirety. In some embodiments, the features 430, 432, 434, 436 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers and / or structures for forming an air gap).

[0027] In some embodiments, the light extraction optical elements 460, 462, 464, 466, 468 are diffraction features that form a diffraction pattern, i.e., a “diffractive optical element” (also referred to herein as a “DOE”). Preferably, the DOE has a relatively low diffraction efficiency such that only a portion of the light of the beam is deflected toward the eye 410 using each intersection of the DOE, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is thus split into several associated output beams that exit the waveguide at multiple locations, resulting in a very uniform output emission pattern toward the eye 410 with respect to this particular collimated beam that bounces within the waveguide.

[0028] In some embodiments, one or more DOEs may be switchable between an “on” state that actively diffracts and an “off” state that does not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystals in which microdroplets form a diffraction pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match that of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0029] In some embodiments, the number and distribution of depth planes and / or depth of field may vary dynamically based on the pupil size and / or orientation of the viewer's eye. In some embodiments, an imaging system 452 (e.g., a digital camera) facing inward may be used to capture an image of the eye 410 and determine the pupil size and / or orientation of the eye 410. In some embodiments, the imaging system 452 facing inward may be attached to the frame 212 (as shown in FIG. 2) and communicate electrically with processing modules 224 and / or 228 that can process the image information from the imaging system 452 facing inward and determine, for example, the pupil diameter and / or orientation of the user's eye 204.

[0030] In some embodiments, the inward-facing imaging system 452 (e.g., a digital camera) can observe user movements such as eye movements and face movements. The inward-facing imaging system 452 can capture an image of the eye 410 and be used to determine the pupil size and / or orientation of the eye 410. The inward-facing imaging system 452 can be used to determine the direction the user is looking (e.g., eye pose), or to obtain an image for user biometric identification (e.g., via iris identification). The image obtained by the inward-facing imaging system 452 can be analyzed to determine the user's eye pose and / or mood and can be used by the display system 400 to determine the audio or visual content to be presented to the user. The display system 400 can also use sensors such as an inertial measurement unit (IMU), accelerometer, gyroscope, etc. to determine the head pose (e.g., head position or head orientation). The head pose can be used, alone or in combination with the eye pose, to interact with the support tracking and / or to present audio content.

[0031] In some embodiments, one camera can be utilized per eye to separately determine the pupil size and / or orientation of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some embodiments, at least one camera can be utilized per eye to separately and independently determine the pupil size and / or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, only the pupil diameter and / or orientation of one eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the viewer 204.

[0032] For example, the depth of field may vary inversely with the viewer's pupil size. As a result, as the size of the viewer's eye pupil decreases, one plane that was indistinguishable because its location in the plane exceeded the depth of focus of the eye becomes distinguishable, and as the pupil size decreases and the corresponding depth of field increases, it increases so that it can appear more in focus. Similarly, the number of separated depth planes used to present different images to the viewer may decrease with the decreased pupil size. For example, a viewer may not be able to clearly perceive the details of both a first depth plane and a second depth plane at one pupil size without adjusting the eye's focusing from one depth plane to the other. However, these two depth planes may be sufficient for the user to focus at another pupil size without changing the focusing adjustment at the same time.

[0033] In some embodiments, the display system may vary the number of waveguides that receive image information based on the determination of pupil size and / or orientation, or in response to receiving an electrical signal indicating a particular pupil size and / or orientation. For example, if the user's eye is indistinguishable between two depth planes associated with two waveguides, the controller 450 may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this can reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0034] In some embodiments, it may be desirable to satisfy the condition that the emitted beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition can be difficult in view of the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the emitted beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the emitted beam may also decrease. In some embodiments, the emitted beam size may be varied using a variable aperture.

[0035] The display system 400 can include an outward-facing imaging system 454 (e.g., a digital camera) that images a portion of the world 456. This portion of the world 456 can be referred to as the field of view (FOV), and the imaging system 454 is sometimes also referred to as the FOV camera. The entire area available for viewing or imaging by the viewer 204 can be referred to as the field of regard (FOR). The FOR may include a solid angle of 4π steradians surrounding the display system 400. In some implementations of the display system 400, the FOR may include substantially all of the solid angle around the user 204 of the display system 400 because the user 204 can move his head and eyes to view objects surrounding the user (in front of, behind, above, below, or to the side of the user). Images obtained from the outward-facing imaging system 454 can be used to track gestures made by the user (e.g., hand or finger gestures) and to detect objects within the world 456 in front of the user.

[0036] The display system 400 can include a user input device 466 through which a user can input commands to the controller 450 and interact with the display system 400. For example, the user input device 466 can include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., functioning as a virtual user input device), etc. In some cases, the user may use a finger (e.g., the thumb) to press or swipe on a touch sensor-based input device to provide input to the display system 400 (e.g., to provide user input to a user interface provided by the display system 400). The user input device 466 may be held by the user's hand during use of the display system 400. The user input device 466 can communicate with the display system 400 either wired or wirelessly.

[0037] FIG. 5 shows an embodiment of an output beam output by a waveguide. One waveguide is shown, but other waveguides within waveguide assembly 405 may function similarly, and it should be understood that waveguide assembly 405 includes a plurality of waveguides. Light 505 is introduced into waveguide 420 at input edge 510 of waveguide 420 and propagates within waveguide 420 by TIR. At the point where light 505 impinges on DOE 460, a portion of the light exits the waveguide as output beam 515. Output beams 515 are shown as being substantially parallel, but they may also be redirected to propagate at an angle to eye 410 depending on the depth plane associated with waveguide 420 (e.g., to form a diverging output beam). It should be understood that a substantially parallel output beam may represent a waveguide with an optical extraction optical element that externally couples the light and forms an image that appears to be set on a depth plane at a long distance (e.g., optical infinity) from eye 410. Other waveguides or other sets of optical extraction optical elements may output a more diverging output beam pattern that requires eye 410 to focus at a closer distance and be interpreted by the brain as light from a distance closer to eye 410 than optical infinity.

[0038] FIG. 6 shows another embodiment of a display system 400 that includes a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The display system 400 can be used to generate a multi-focus stereo, image, or light field. The display system 400 can include one or more primary planar waveguides 604 (only one is shown in FIG. 6) and one or more DOEs 608 associated with at least some of the primary waveguides 604. The planar waveguide 604 can be similar to the waveguides 420, 422, 424, 426, 428 discussed with reference to FIG. 4. The optical system can employ a diffractive waveguide device to relay light along a first axis (vertical or Y-axis in the FIG. 6 drawing) and expand the effective exit pupil of the light along the first axis (e.g., the Y-axis). The diffractive waveguide device can include, for example, a diffractive planar waveguide 612 and at least one DOE 616 (illustrated by the dashed line) associated with the diffractive planar waveguide 612. The diffractive planar waveguide 612 can be similar or identical to the primary planar waveguide 604 having a different orientation at at least some points. Similarly, at least one DOE 616 can be similar or identical to the DOE 608 at at least some points. For example, the diffractive planar waveguide 612 and / or the DOE 616 can each be made of the same material as the primary planar waveguide 604 and / or the DOE 608. The optical system shown in FIG. 6 can be integrated into the wearable display system 200 shown in FIG. 2.

[0039] The relayed light with an expanded exit pupil is optically coupled into one or more primary planar waveguides 604 from the diffractive waveguide device. The primary planar waveguide 662 preferably relays light along a second axis (e.g., horizontal or X-axis in the view of FIG. 6) that is orthogonal to the first axis. It should be noted that the second axis can be a non-orthogonal axis with respect to the first axis. The primary planar waveguide 604 expands the effective exit path of the light along its second axis (e.g., X-axis). For example, the diffractive planar waveguide 612 can pass the light through the primary planar waveguide 604 that relays and expands the light along the vertical or Y-axis and relays and expands the light along the horizontal or X-axis.

[0040] The display system 400 may include one or more colored light sources (e.g., red, green, and blue laser light) 620 that can be optically coupled into the proximal end of the single-mode optical fiber 624. The distal end of the optical fiber 624 may be screwed or received through a hollow tube 628 of piezoelectric material. The distal end projects from the tube 628 as a flexible cantilever 632 that is not fixed. The piezoelectric tube 628 can be associated with four quadrant electrodes (not shown). The electrodes may be deposited, for example, on the outside, outer surface, or outer periphery of the tube 628, or on the diameter. A core electrode (not shown) is also located at the core, center, inner periphery, or inner diameter of the tube 628.

[0041] For example, the drive electronics 636, which are electrically coupled via wires 640, drive a pair of opposing electrodes to independently bend the piezoelectric tube 628 in two axes. The protruding distal tip of the optical fiber 624 has a mechanical resonance mode. The resonance frequency can depend on the diameter, length, and material properties of the optical fiber 624. By vibrating the piezoelectric tube 628 near the first mechanical resonance mode of the fiber cantilever 632, the fiber cantilever 632 can be vibrated and swept through a large deflection.

[0042] By stimulating resonant vibrations along two axes, the tip of the fiber cantilever 632 is scanned in two axial directions within the area filling a two-dimensional (2-D) scan. By modulating the intensity of the light source 620 in synchronization with the scan of the fiber cantilever 632, light emitted from the fiber cantilever 632 forms an image. An explanation of such an arrangement is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0043] The component 644 of the optical coupler subsystem collimates the light emitted from the scanning fiber cantilever 632. The collimated light is reflected by the mirrored surface 648 into the narrow-dispersion planar waveguide 612 containing at least one diffractive optical element (DOE) 616. The collimated light propagates vertically (with respect to the view in the figure of FIG. 6) along the dispersion planar waveguide 612 by total internal reflection, thereby repeatedly intersecting the DOE 616. The DOE 616 preferably has a low diffraction efficiency. This diffracts a portion of the light (e.g., 10%) towards the edge of the larger primary planar waveguide 604 at each point of intersection with the DOE 616 and allows a portion of the light to continue on its original trajectory along the length of the dispersion planar waveguide 612 via TIR.

[0044] At each point of intersection with the DOE 616, additional light is diffracted towards the entrance of the primary waveguide 612. By splitting the incident light into a plurality of external coupling sets, the exit pupil of the light is vertically expanded by the DOE 616 within the dispersion planar waveguide 612. The vertically expanded light externally coupled from the dispersion planar waveguide 612 enters the edge of the primary planar waveguide 604.

[0045] The light entering the first waveguide 604 propagates horizontally along the first waveguide 604 (with respect to the view in FIG. 6) via TIR. As the light intersects the DOE 608 at multiple points, it propagates horizontally along at least a portion of the length of the first waveguide 604 via TIR. The DOE 608 preferably has a phase profile that is advantageously the sum of a linear diffraction pattern and a radially symmetric diffraction pattern and can be designed or configured to generate both deflection and focusing of the light. The DOE 608 preferably has a low diffraction efficiency (e.g., 10%) such that only a portion of the light of the beam is deflected towards the viewer's eye at each intersection of the DOE 608, while the remainder of the light continues to propagate through the waveguide 604 via TIR.

[0046] At each point of intersection between the propagating light and the DOE 608, a portion of the light is diffracted towards an adjacent surface of the first waveguide 604, allowing the light to escape from TIR and be emitted from the surface of the first waveguide 604. In some embodiments, the radially symmetric diffraction pattern of the DOE 608 additionally imparts a certain focal level to the diffracted light and shapes (e.g., imparts curvature to) the wavefront of the individual beams, as well as steering the beams to an angle that matches the designed focal level.

[0047] Therefore, these different paths can couple light outside the primary planar waveguide 604 by resulting in different multiplicity of the DOE 608, focus levels, and / or filling patterns at different angles in the exit pupil. Different filling patterns in the exit pupil can advantageously be used to generate a light field display with multiple depth planes. Each layer within the waveguide assembly or a set of layers within a stack (e.g., three layers) may be employed to generate an individual color (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers may each be employed to generate red, blue, and green light at a first depth of focus. A second set of three adjacent layers may each be employed to generate red, blue, and green light at a second depth of focus. Multiple sets may be employed to generate a full 3D or 4D color image light field with various depths of focus.

[0048] (Other components of the AR system) In many implementations, the AR system may include other components in addition to the wearable display system 80 (or the optical system 100). The AR device may include, for example, one or more haptic devices or components. The haptic device or component may be operable to provide a haptic sensation to the user. For example, the haptic device or component may provide a haptic sensation of pressure and / or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual constructs). The haptic sensation may reproduce the feel of a physical object represented by the virtual object, or the feel of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the haptic device or component may be worn by the user (e.g., on a user wearable glove). In some implementations, the haptic device or component may be held by the user.

[0049] The AR system may include one or more physical objects that are operable by a user to enable input or interaction with the AR system, for example. These physical objects are referred to herein as totems. Some totems may take the form of inanimate objects, such as pieces of metal or plastic, walls, or the surface of a table. Alternatively, some totems may take the form of living objects, such as a user's hand. As described herein, a totem may not actually have any physical input structure (such as a key, trigger, joystick, trackball, rocker switch). Instead, a totem may simply provide a physical surface, and the AR system may render a user interface so as to appear to the user to be on one or more surfaces of the totem. For example, the AR system may render an image of a computer keyboard and trackpad so as to appear to be resident on one or more surfaces of the totem. For example, the AR system may render a virtual computer keyboard and virtual trackpad so as to appear on the surface of a thin rectangular plate of aluminum that serves as a totem. The rectangular plate itself does not have any physical keys or trackpad or sensors. However, the AR system may detect user operations or interactions or touches using the rectangular plate as selections or inputs made via the virtual keyboard and / or virtual trackpad.

[0050] Examples of haptic devices and totems that are compatible with the AR devices, HMDS, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0051] (Examples of performing error correction on a display system) As described above, the display system may include a stacked waveguide assembly, such as that illustrated in FIGS. 4-6, having a plurality of display layers of substrate material with diffraction gratings for redirecting light to produce a digitized light field that impinges on the eye. In some embodiments, the waveguide assembly includes one substrate layer per color per depth. For example, a 2-depth plane RGB display can have a total of six waveguide layers. The display system can be an embodiment of the wearable display system 80.

[0052] In stacked waveguide assemblies, there is a range of potential phenomena that can introduce artifacts and result in degraded image quality. These can include ghosting (multiple images), distortion, misregistration (between colors or depths), and color intensity variations across the field of view. Additionally, certain types of artifacts can also occur under other types of conditions, such as when illuminating with a laser as opposed to an LED (e.g., speckle, banding, Newton's fringes), or when the density of the externally coupled beam is below a certain amount (e.g., wavefront sparsity, which can be perceived as if looking through a screen door or picket fence).

[0053] Due to imperfections in the optics of the light field display, a perfect three-dimensional grid within the rendering engine can be distorted when displayed through the optics. To identify and correct for the distortion between the expected and the actual displayed image, a calibration pattern, such as a moiré pattern, can be projected using the display system.

[0054] FIG. 7 illustrates exemplary distortions that can occur when the calibration pattern 702 is projected by a display system. The calibration pattern 702 can be any type of pattern suitable for performing spatial or color calibration (e.g., a checkerboard pattern comprising a plurality of checkerboard squares). The calibration pattern 702 can include any type of test or calibration pattern, such as a geometric pattern or a random probabilistic pattern. The projected calibration pattern 702 results in the generated light field image 704. The distortions present in the image 704 can include spatial distortions (e.g., when visible pixels are not in the expected locations within the field of view) and color distortions (e.g., when the color values of visible pixels are different from what is expected). For example, the checkerboard squares of the pattern 702 can be shifted from their expected positions within the image 704 (e.g., a spatial error). Additionally, instead of checkerboard squares appearing in black and white, some of the checkerboard squares within the image 704 can appear in other colors, such as purple (e.g., a color error). The display error can be measured using a light field measurement system that can include a digital camera positioned to acquire an image of the calibration pattern projected by the display. In some embodiments, multiple images corresponding to calibration images shifted to different locations can be captured to obtain more granular information regarding the expected position versus the actual position. The digital camera can be configured to focus at different depths of focus to determine the depth at which different regions of the displayed image (e.g., features on the displayed calibration pattern) are in focus.

[0055] The step of capturing multiple images at different depths of focus and determining the depth of different regions of the displayed image, according to some embodiments, is described in more detail below in connection with FIGS. 17 - 20. Different types of calibration patterns that can be used in various embodiments are described in more detail below in connection with FIGS. 22 - 24.

[0056] (Spatial error) Spatial errors can include several different manifestations. For example, spatial misregistration can include translation or rotation of the display layer. Spatial errors can also involve non-linear spatial distortion that varies across the field of view (FOV) of the depth plane of the display.

[0057] Spatial errors can be manifestations of mechanical or optical defects within the display system. By interpreting the measured spatial errors, metrics can be derived that quantify the opto-mechanical quality of the system and suggest ways for improvement. For example, a spatial error representing rotation in the depth plane can suggest that the display is mechanically rotated relative to the desired position. Scaling for each color plane can suggest that the lens system is not sufficiently achromatic.

[0058] To identify spatial errors, a light field measurement system equipped with an image capture device such as a digital camera can be used to capture one or more images projected by the display system (e.g., projection of a calibration pattern) and generate a vector field that represents the deviation of the actual displayed image from the expected image. The vector field can be a three-dimensional vector field that includes in-plane deviations in the x-y plane of the display and out-of-plane deviations in the z-direction (depth), or a two-dimensional vector field that includes deviations only in the x-y plane. In some embodiments, the vector field may be generated for each depth plane or color plane of the display system. In some embodiments, the depth may be measured in diopters (which represents the reciprocal of the focal length of a layer in meters).

[0059] FIG. 8 illustrates an example of a vector field that can be generated from one or more captured images that map the deviation between the expected position of a point within a projected light field and its actual displayed position. Points within the projected light field can correspond to features within a calibration image (e.g., the centers and corners of calibration grid squares). Each vector within the vector field represents the distortion between the expected position within the light field and its corresponding actual position. In this example, the distortion vector field is 2D. In the illustrated vector field, the expected position of a feature is marked using a first color and marker type (e.g., "O" 802 for the expected position), while the actual displayed position of the feature is marked using a second color (e.g., "X" 804 for the detected position). Each pair of corresponding expected and displayed positions is connected by a line 806 that can include an arrow indicating the direction of correction needed to correct the detected displayed position to be at the expected position.

[0060] Using the vector field, local or global distortion information (e.g., in-plane translation, collective scaling, collective rotation, average pixel warping, or dioptric error, as described below) can be extracted. For example, a distortion graph may be generated from the determined vector field. The distortion graph can be used to analyze the distribution of pixel position error values (e.g., vector amplitude) across the generated vector field. The distortion graph may be a histogram showing the frequency of pixel position error (e.g., plotting the pixel position error amplitude against the frequency at which the error amplitude appears within the vector field). Other types of graphs may be used to analyze other attributes of the vector field (e.g., distortion direction).

[0061] Spatial errors can, in a broad sense, be divided into in-plane and out-of-plane spatial errors. In-plane spatial errors refer to spatial errors along a specific depth plane (e.g., the xy-plane according to the coordinate system illustrated in FIG. 6) at a specific depth (measured on the z-axis). A vector field (e.g., as illustrated in FIG. 8) can be used to derive one or more metrics for different categories of spatial errors. These metrics can be defined respectively for each layer (e.g., for each individual display layer corresponding to a specific combination of color and depth (e.g., red - 3 diopter display layer, green - 1 diopter display layer, etc.)) or for each display (e.g., to quantify the overall fidelity of the display in terms of concise parameters).

[0062] (In-plane spatial error) In some embodiments, the in-plane spatial error can be divided into a plurality of different components, each corresponding to a different type of error. These components can include translational error, rotational error, scaling error, or non-linear spatial error. These error components can be corrected respectively individually or sequentially.

[0063] (In-plane translational error) FIG. 9A illustrates an exemplary in-plane (xy) translational spatial error (also referred to as xy eccentricity). The xy translational error refers to the x- and / or y-pixel offset of the center of the displayed image of the display layer from its expected position, and is intended to indicate mechanical or display alignment. In FIG. 9A, the expected image position 900 (shown as a red rectangle in this embodiment) is translated to the displayed image position 900a (shown as a green shape with a non-linear edge). The xy translational error identifies the center position 902 of the displayed image 900a and the center position 904 of the expected image 900, and is corrected by making one or more offsets (along the determined translation vector 901) such that the displayed center position 902 is aligned with the expected center position 904 (through mechanical alignment of the display, software correction of the display image, or a combination of both). One or more metrics for the measured xy translational spatial error can include the translational error measured for each layer that measures the layer center relative to the expected or reference position (e.g., the optical axis of the display), or the maximum translational offset measured for each display that indicates the maximum translation between any two display layers to quantify the overall translational alignment.

[0064] (Collective rotation error) Figure 9B illustrates an exemplary collective rotation spatial error. Collective rotation refers to the overall rotation angle of the displayed image around its center with respect to the expected position of the image. Spatial distortion may not always be fully describable by simple affine rotation, but the collective rotation measurement can be used to provide the rotation angle at which the pixel position error (between the displayed image position and the expected image position) is minimized. The collective rotation metric is intended to inform mechanical or display alignment. As illustrated in Figure 9B, the collective rotation can be corrected by rotating the displayed image 906 around the center point 908 by the specified rotation amount 907 to the position 910 corresponding to the expected position (through mechanical alignment of the display, through software correction of the displayed image, or both). The reported metric can include the rotation error measured per layer, which indicates the measured orientation relative to the expected or reference orientation (e.g., relative to the horizontal axis of the display), and the maximum rotation offset measured per display, which indicates the maximum rotation error between any two display layers to quantify the overall rotational alignment.

[0065] (Collective scaling error) FIG. 9C illustrates an example of the collective scaling space error. Collective scaling indicates the overall scaling factor of the displayed image around its center relative to the expected image. The spatial distortion may not be fully described by simple affine scaling, but the collective scaling measurement can indicate the scaling factor at which the pixel position error is minimized. The collective scaling metric is intended to inform optical design or display alignment. As shown in FIG. 9C, the collective scaling space error can be corrected by scaling the size of the displayed image 912 by the specified scaling amount 913 to match that of the expected image 914. The reported metrics for collective scaling measure the image scaling relative to the expected or reference scaling (e.g., with reference to a physical target in a calibrated setting), the scaling error measured for each layer, and the maximum scaling offset measured for each display, which indicates the maximum scaling between any two display layers to quantify the overall scale alignment.

[0066] FIG. 9D illustrates another example of the collective scaling space error. The displayed image 916 appears smaller compared to the expected image 918. To correct the scaling error, the displayed image 916 is scaled upward by the scaling amount 917 to match the size of the expected image 918.

[0067] (Pixel warping error) Figure 9E illustrates an example of the remaining spatial error after correction for XY translation, rotation, and scaling. The remaining error (also referred to as pixel warping or spatial mapping) represents the average residual Euclidean pixel position error after XY translation, rotation, and scaling have been removed from the overall spatial distortion profile (e.g., as shown in FIGS. 9A - 9D), provides a measure of the non - linear or non - affine warping characteristics of the display system, and may be used to inform display design and quality control. The metrics reported for pixel warping include the mean pixel warping (MPW) measured for each layer, which indicates the average residual Euclidean pixel position error after XY translation, rotation, and scaling have been removed with reference to a perfect grid, and the maximum mean pixel warping (maximum MPW) measured for each display, which indicates the maximum value of the MPW between the layers of the display for quantifying the overall warping. In some embodiments, the remaining pixel warping is corrected through spatial mapping performed using a processing module (e.g., module 224 or 228) to align the displayed image 920 with the expected image 922.

[0068] (Out - of - plane spatial error) Digital light field display systems such as those illustrated in FIGS. 4 - 6 are capable of generating depth planes that appear to the viewer at different depths (z - direction). (See, e.g., FIG. 3). In some embodiments, the depth planes correspond to flat planes that appear to be located at different distances from the viewer. As is common in optics, rather than referring to the distance of the depth plane from the display, diopters (m -1)The reciprocal distance measured in units can be used to reference different depth planes. For example, the display may have two depth planes positioned at depths of 3 diopters (1 / 3 m) and 1 diopter (1 m). Due to imperfections within the display system, the diopter profile across the depth planes may not be as expected. For example, an image displayed on a depth layer may have a diopter profile with incorrect distances or varying focus across the FOV of the display.

[0069] Out-of-plane spatial error (also referred to as diopter error) is a measure of the diopter (depth) error of a depth plane and is intended to indicate errors in optics, mechanics, and waveguide alignment or design. The metrics reported for diopter error may include the diopter error measured per layer, which indicates the amount of error between the expected depth and the measured depth of a depth plane, and the maximum diopter error, which indicates the maximum depth error between depth planes.

[0070] Figure 10A illustrates an example of multiple depth planes intended to be viewed at different depths. In the example shown, three different depth planes are shown, but the display system may contain more or fewer depth planes. Additionally, each depth plane may correspond to multiple waveguide layers (e.g., RGB color layers).

[0071] Figures 10B - 10D illustrate examples of types of out-of-plane spatial error that can occur when viewing the projected depth planes shown in Figure 10A. For example, the projected depth plane may be shifted to different depths so that it appears at a depth greater or less than expected (Figure 10B). The depth plane may be misaligned so as to exhibit a bulk rotation from the expected depth (Figure 10C). The depth plane may exhibit a non-uniform profile characteristic of grid imperfections (Figure 10D). The depth plane may exhibit a combination of the errors illustrated in Figures 10B - 10D.

[0072] Figure 10E illustrates another example of out-of-plane spatial error. The projected depth plane 1002 is misaligned with the expected depth plane 1004. In the illustrated example, the misalignment comprises a depth plane rotation. To correct the out-of-plane spatial error, a rotation axis 1006 is identified and the projected depth plane 1002 is rotated about the identified rotation axis 1006 such that the projected depth plane 1002 substantially aligns with the expected depth plane 1004. The rotation axis 1006 is illustrated as parallel to the axis of the expected depth plane 1004 (e.g., the vertical axis), but it should be understood that the rotation axis can be in any direction.

[0073] The dioptric error is distinct from the in-plane spatial error, which is associated with in-plane distortion, but the dioptric error can potentially affect the in-plane spatial error, for example, by introducing a field-dependent spatial distortion due to an incorrect assumption of pixel depth. For example, for a defective depth plane with regions at unexpected depths, the pixels are non-uniformly shifted relative to the viewer position, introducing variable image warping.

[0074] In some embodiments, the error correction techniques described herein for in-plane spatial errors (e.g., xy eccentricity, convergence scaling, convergence rotation, and spatial mapping) can be extended up to three dimensions. For example, eccentricity can be performed in three dimensions by identifying the location of the center point of the plane displayed on the xyz coordinate system and shifting the plane (e.g., along the x, y, and z axes) such that the center point aligns with the expected location.

[0075] (Quantification of Spatial Error Based on Distortion Vector Field) As described herein with reference to FIG. 8, a multi-dimensional (e.g., 2D or 3D) distortion vector field can be generated by measuring the displacement of image features from an expected position to a displayed position. The distortion vector field can be calculated for each layer of a multi-layer display (e.g., a display comprising a stacked waveguide assembly 405). The distortion vector field can be used to capture and characterize the distortion of the light field projected by the display. For example, a vector analysis operation can be performed on the distortion vector field to determine a certain spatial error. A light field measurement system can calculate such a vector operation as part of the analysis of an image obtained by a measurement camera (e.g., a digital camera or a light field camera) with respect to a calibration pattern (e.g., a grid pattern) projected by the display. Such vector analysis techniques are not limited to light field displays and can be applied to any multi-dimensional measurement or calibration of any type of display.

[0076] Given a multi-dimensional distortion vector field, the curl of the vector field can be calculated to determine local rotation. The average of the curl over a region within the FOV of the display provides a measure of the collective rotation error within the region. In a discrete depth plane implementation of a light field display, the calculation of the curl of the distortion vector field can provide information regarding in-plane or out-of-plane rotation of the layer.

[0077] The divergence of the distortion vector field can be calculated to determine scaling error. In an implementation having multiple layers (e.g., RGB color layers) for generating a full-color image at each depth, this scaling error can be used to provide information regarding scaling calibration.

[0078] Vector integral type theorems (e.g., Stokes' theorem or divergence theorem (Gauss' theorem)) are applied to a distorted vector field, and the curl and divergence of the vector field over a region within the FOV of the display can be calculated (e.g., to find a set rotation or set scaling of the region). The Euclidean average of the vectors within the distorted vector field can be calculated to obtain information about the non-affinity of the spatial transformation introduced by the distortion.

[0079] (Quantification of color error) Color error occurs when the color value of a visible pixel is different from the expected color value. To evaluate the color error, a calibration image may be projected using the display system. The calibration image may be the same calibration image used for spatial error correction or a different calibration image. For example, the calibration image can comprise a solid image of a specific color, such as red, at a specific luminance level (e.g., maximum brightness). The output from projecting the calibration image can be captured using an image capture device (e.g., one or more cameras). FIG. 11 illustrates an example of a captured image of a projected calibration image. The calibration image may have a luminance level that is constant throughout the image, but the luminance of the displayed calibration image varies across the FOV of the display due to the presence of color error. For example, a region 1102 of the captured image may be at a high luminance level, while another region 1104 exhibits a lower luminance level, causing dark regions or bands to appear across the display. In some embodiments, the calibration image can comprise a colored calibration pattern rather than being colorless.

[0080] In some embodiments of the display, the observed luminance topology may be wavelength - dependent. For example, luminance variations may differ for red, green, and blue, causing the projected image to appear in colors other than expected (indicating an imbalance between the red, green, and blue components). For example, a projected white calibration image may appear as a purple with a lower green luminance level than the red and blue ones. Additionally, luminance variations may also be based on the observer's location (e.g., when the camera is moved, the dark band in 1102 may appear to move to different locations within the FOV). This phenomenon leads to difficulties in maintaining color uniformity and white balance across the FOV (especially since luminance or color balance may depend on the observer's location), and ultimately can affect the color accuracy of the displayed content.

[0081] Each display layer within the display system is associated with color characteristics, measured colors, and luminance characteristics that measure brightness or intensity. Thus, color errors can, in a broad sense, be divided into luminance flatness errors and color uniformity errors.

[0082] (Luminance flatness) The luminance flatness metric can be used to quantify the amount of variation in luminance exposed by a single display layer. Generally, in a stacked waveguide assembly, different display layers may potentially have different luminance variations across the field of view due to each display layer being generated by a different waveguide within the stack (see, for example, waveguide assembly 405 in FIG. 4).

[0083] To measure the luminance flatness of a display layer, luminance values (also referred to as intensity values) can be determined for some or all of the pixels of the captured image. This disclosure mainly refers to the luminance values of pixels, but in other implementations, the luminance values can be determined for regions comprising a plurality of pixels (e.g., an N×M grid of pixels) instead of individual pixels. In some embodiments, each determined luminance value can be assigned a luminance bin that comprises a range of one or more luminance values. For example, for an 8-bit color display system, 256 bins corresponding to 8-bit colors can be used.

[0084] From the determined luminance values, several luminance flatness metrics can be calculated by a measurement system. For example, the mode, which indicates the most common pixel luminance value across the displayed field, can be calculated. From the mode, the half-population pixel range (HPPR), which indicates the number of luminance ranges or bins adjacent to the mode that encompass 50% of the pixel population, can be determined. A small HPPR indicates that the luminance across the display layer is substantially uniform. The luminance value can also be referred to as an intensity value. For the purposes of this application, the terms "luminance" and "intensity" can be used synonymously.

[0085] FIG. 12A illustrates an intensity histogram that can be generated from the captured image of the projected calibration image (e.g., as illustrated in FIG. 11). The intensity histogram plots the luminance values against the frequency with which they appear in the captured image (e.g., the number of pixels having the luminance value). The mode is indicated by the luminance value having the highest occurrence number within the image (e.g., at location 1202).

[0086] FIG. 12B illustrates an intensity profile generated from a captured image of a projected calibration image. In the illustrated intensity profile, the mode occurs at luminance value 1204 (which has a value of 236 in this example). From the mode, a deviation range centered on the mode 1204, shown as the range between luminance values 1206 and 1208, is determined to encompass 50% of the pixel population of the image. The HPPR is determined based on the calculated deviation range (e.g., the difference between luminance values 1206 and 1208).

[0087] For an ideal display layer, the intensity values would be uniform across the field for a given input illumination (e.g., HPPR = 0). Deviations from this ideal behavior would manifest as a distribution of pixel intensity values away from the mode. HPPR measurements attempt to measure the distribution away from the mode. A substantially uniform luminance can have a small HPPR, e.g., small compared to the range of the mode or a possible luminance value (e.g., 255 for an 8-bit color). For example, a substantially uniform (e.g., flat) luminance display can have an HPPR ratio of less than about 10%, less than about 5%, less than about 1%, or less than about 0.1% of the total color range.

[0088] The HPPR can be considered as a variation of the interquartile range, which measures the distribution away from the median instead of the mode. The median of the pixel intensity values may not have a direct relationship to the desired flat-intensity response of the display layer. FIG. 13 illustrates exemplary intensity histograms 1302, 1304 that illustrate the differences (μ) between the mode, median, and mean values. The medians of the two distributions 1302, 1304 are the same in this example. The two distributions 1302, 1304 have standard deviations σ of 0.8 and 2, respectively. As schematically illustrated in FIG. 13, when the intensity distribution of the image is close to normal (e.g., intensity distribution 1302), the mode, median, and mean can all be very similar. On the other hand, when the intensity distribution is not close to the normal distribution (e.g., intensity distribution 1304), the mode, median, and mean of the intensity distribution can be substantially different from each other.

[0089] For each display layer of the display, luminance flattening attempts to reduce luminance variations across the displayed field of view. Typically, since the luminance intensity of a pixel cannot be increased beyond its maximum value, luminance flattening is generally an overall luminance reduction step, where the pixel luminance is compressed within a layer-specific profile such that the luminance of the layer is as flat as possible.

[0090] For example, luminance flattening can be performed such that the pixel luminance has a maximum value at the luminance value of the pixel with the lowest luminance value and reduces the luminance of the display layer to a substantially minimum luminance. Alternatively, the pixel luminance can be configured to have a maximum value at a selected luminance value that is above the luminance value of the pixel with the lowest luminance value. This may still result in pixels having luminance values below the selected value and luminance non-uniformity remaining, so the overall luminance may not be reduced to the minimum value. In some embodiments, reducing the luminance value for a pixel or group of pixels includes identifying a value for reducing the luminance value of the pixel or group of pixels. In other embodiments, reducing the luminance value for a pixel or group of pixels includes identifying a scaling factor for scaling the luminance value of the pixel or group of pixels down to a minimum luminance value or a threshold luminance value.

[0091] In some embodiments, if the initial luminance flatness of the display layer is good (e.g., HPPR is below a threshold), the luminance value can be reduced to the minimum to provide a flat luminance field. On the other hand, if the luminance flatness is poor (e.g., HPPR exceeds the threshold) or the minimum luminance value is low (e.g., does not reach a minimum threshold), a selected maximum luminance value can be selected. Luminance flattening can be performed within a software module (e.g., processing modules 224, 228).

[0092] The level at which the luminance is reduced when performing luminance flattening may vary for each display layer. However, different luminance levels for different layers within the same color cluster (e.g., RGB layer cluster) can lead to a loss of white balance, which can be addressed by correcting the color uniformity of the display.

[0093] (Color uniformity) Color generally refers to the color components of a display that are independent of luminance. As described above, the display layers within a display system may include a red display layer, a green display layer, and a blue display layer, but it should be understood that in other implementations, other numbers, types, or colors of display layers, or combinations of display layers, may be used. In the following examples, RGB color layers will be described for illustrative purposes, but this is not a limitation regarding methods for color balance (which can be applied to any set of display colors).

[0094] When the luminance variations of the corresponding red, green, and blue display layers are the same, the color is maintained across the display. On the other hand, when the luminance variations across the corresponding red, green, and blue display layers are different, the color of the displayed image will be different from what is expected. For example, for a white-calibrated image, if the red and blue layers have a higher luminance than the green layer, the area of the white-calibrated image may appear in a purple color. These deviations from the intended white can be referred to as off-gray scale.

[0095] The color uniformity metric can be used to capture the degree of off-gray scale of an image. The metric may include an average color error that indicates, respectively, an average over the FOV of the deviation of red, green, and blue from their corresponding averages for red, green, and blue. The smaller the average color error, the closer the image will appear to the gray scale. The average color error may be normalized to a dimensionless value by dividing by the average color or a range of colors that might be considered (e.g., 255 for 8-bit colors). In various implementations, the display can be considered to have achieved color uniformity if the average color error is less than 10%, less than 5%, less than 1%, or some other threshold.

[0096] FIG. 14A illustrates an example of a red-green-blue (RGB) intensity map generated from a captured image of a projected test image. The red and blue layers 1402 and 1404 generally have similar luminance to each other, and both the red and blue layers 1402 and 1404 have a much higher luminance than the green layer 1406. As a result, the projection of the white test image will have regions that appear to be purple (red + blue, see, e.g., FIG. 11B).

[0097] FIG. 14B illustrates a plot 1408 that maps the maximum color non-uniformity error. The average luminance 1410 can be determined as the average luminance value of the red, green, and blue layers. The "average + maximum error" surface 1412 indicates the maximum luminance values of the red, green, and blue layers, while the "average - maximum error" surface 1414 indicates the minimum luminance values of the red, green, and blue layers.

[0098] FIG. 15 illustrates an RGB intensity map for a display system with red, green, and blue layers having different intensities across the displayed field of view as illustrated in FIG. 14A after color correction. As described below and as illustrated in plot 1500, in this example, the maximum R and B luminance values have been reduced to levels of lower G luminance values in most of the display to provide color uniformity.

[0099] As shown in FIG. 14A, prior to color correction, the luminance of the red and blue layers is much higher than that of the green layer over most of the FOV, which can result in large areas of the captured image of the white calibration image appearing purple. In this embodiment, during color correction for each point in the depth plane, the minimum luminance value of the color layers (e.g., red, green, and blue) associated with the depth plane is identified, and the luminance value for each color layer is set to the minimum luminance value for that point. For example, as shown in FIG. 15, the color luminance of the red and blue layers 1502 and 1504 is reduced to match that of the green layer 1506 (e.g., by comparing the RGB intensity map of FIG. 14A with the RGB intensity map of FIG. 15). As a result, the luminance of the red and blue layers is corrected such that they match the intensity of the green layer and reduce the off-gray scale amount of the projected image.

[0100] (Image Correction Process) Image calibration refers to the evaluation of the characteristics of a display device related to previously defined image quality metrics (see, e.g., the description with reference to FIGS. 7-15). Image correction refers to the correction actions performed to improve image quality. The image quality metric informs the correction actions being taken to attempt to improve or optimize the display device image quality metric. Thus, image correction is closely tied to each of the image quality metrics.

[0101] FIG. 16 is a flowchart of an example of a process 1600 for performing image correction on a display system. In block 1602, the camera used to capture the projected image (e.g., camera 1806 of the measurement system 1800 described below) is calibrated. Camera calibration includes the evaluation of the accuracy characteristics of the camera when capturing and representing actual visual / display information. To ensure that any measured metric from the captured image is due to the display system and not an error associated with the camera, the camera used for image correction should be fully calibrated before image correction is attempted.

[0102] In some embodiments, camera calibration includes the step of performing at least one of flat field correction (e.g., ensuring that the intensity response of the camera is uniform across its FOV), lens distortion correction (e.g., identifying and compensating for lens distortion), or pixel scaling (e.g., identifying the relationship between the pixel size for the camera's image capture and the pixel size of the display system). In some implementations, a display / camera pixel mapping can be applied to perform the conversion between display pixel values and camera pixel values. The display / camera pixel mapping can be based on a first global non-linear gamma function that maps display color pixel values to a first intermediate color space, a local pixel-dependent binding function that maps the first intermediate color space to a second intermediate color space, and a second global non-linear gamma function that maps the second intermediate color space to pixel intensities within the camera color space. Details of an exemplary display / camera pixel mapping are described below with reference to FIG. 21.

[0103] In block 1604, spatial error correction can be performed on the display system. Spatial error correction can include the step of capturing one or more images of the projected light field using a calibrated camera, which can be used to generate a vector field indicative of the distortion between the displayed image location and the expected image location. In some embodiments, an individual vector field is generated for each display layer. Using the generated vector field, one or more spatial corrections can be performed, which can include XY decentering (block 1604a), convergence rotation (block 1604b), convergence scaling (1604c), or spatial mapping (block 1604d). In some embodiments, these corrections are each performed for each layer.

[0104] XY eccentricity may refer to the translational spatial error of the center of the displayed image of the display layer with respect to the expected image position. Performing XY eccentricity can include identifying the center point of the displayed image and shifting the image along the determined translation vector so that the center point corresponds to the expected center position. An example of XY eccentricity correction is described with reference to FIG. 9A.

[0105] Convergence rotation may refer to the overall rotational error between the displayed image and the expected position. Performing convergence rotation can include identifying the center point of the displayed image and rotating the image about the identified center point by the specified amount of rotation (e.g., to a position where the pixel position error with respect to the expected image position is minimized). An example of convergence rotation correction is described with reference to FIG. 9B.

[0106] Convergence scaling may refer to the overall scaling error between the displayed image and the expected image. Performing convergence scaling can include identifying the center point of the displayed image and scaling the image about the identified center point by the specified factor (e.g., the factor that minimizes the pixel position error with respect to the expected image position). Examples of convergence scaling are described with reference to FIGS. 9C and 9D.

[0107] XY eccentricity, convergence rotation, and convergence scaling can be used to correct linear or affine spatial errors, but the displayed image of the display layer may also contain additional non-linear or non-affine spatial errors. Spatial mapping can be performed to correct any remaining errors (e.g., non-linear or non-affine errors) that remain after XY eccentricity, convergence rotation, and convergence scaling corrections have been performed. Spatial mapping may also be referred to as pixel warping, and an example is described with reference to FIG. 9E.

[0108] In some embodiments, the spatial error can be divided into an in-plane spatial error and an out-of-plane spatial error (sometimes also referred to as a dioptric error). For example, the display layer may first be corrected for the in-plane spatial error before being corrected for the out-of-plane spatial error, or vice versa. Alternatively, both the in-plane spatial error and the out-of-plane spatial error can be corrected together.

[0109] In block 1606, color error correction can be performed on the display system. The color error correction may comprise luminance flattening (block 1606a) or color balance (block 1606b). In some embodiments, the luminance flattening is performed per layer, while the color balance is performed per color cluster (e.g., per RGB cluster).

[0110] Luminance flattening may refer to reducing the luminance variation across the display layer. In some embodiments, luminance flattening includes reducing the luminance of all pixels within the displayed FOV to a minimum luminance value. Alternatively, all pixels within the displayed FOV having a luminance above a maximum value or a threshold may have their luminance reduced to the maximum value / threshold, while pixels with a luminance below the maximum value / threshold may remain unchanged. In some embodiments, the luminance values may be scaled based on the distance between the luminance and a threshold luminance value. Examples of luminance flattening are described with reference to FIGS. 12A and 12B.

[0111] Color balancing may include steps to reduce an off-gray scale effect caused by inconsistencies in intensity between different color layers within a color cluster (e.g., an RGB cluster). Color balancing can be performed by reducing the luminance of the color layers at each location within the depth plane and matching it to that of the color layer within the color cluster having the lowest luminance at that location. For example, for each pixel within the FOV, the luminance for the red, green, and blue layers at each location is set to the minimum value of the three color layers at that location. In some embodiments, luminance above a threshold luminance value is reduced to either the threshold luminance value or the greater of the minimum luminance value within the color cluster at that location. In some embodiments, the luminance may be scaled based on the distance between the luminance and the threshold luminance value. An example of color balancing is described with reference to FIGS. 14A - 15.

[0112] In some implementations, image calibration (to quantify image quality metrics) is performed for each display system during the manufacturing process. Information associated with the image quality metrics and corrections that can be used to improve or optimize the display system can be stored in a non - transitory memory associated with the display system (e.g., data module 224 or data repository 232). During use of the display system, the image correction information is applied to the display so that appropriate corrections can be made to provide an improved or optimized image that reduces or eliminates image errors within the display for the user of the display system. For example, local or remote processing modules 224, 228 can use the image correction information to provide an improved image to the user on a real - time basis. Details of an exemplary calibration process are described below with reference to FIGS. 27 and 28.

[0113] (Examples of Depth Plane Measurement) Embodiments of the display system described herein are capable of generating a light field (see, e.g., the description with reference to FIGS. 1-6). Thus, a virtual object placed at a certain depth will create a (digitized) light field that will appear in focus at the intended depth, such that an actual (physical) object at a distance from the wearer of the display will generate a light field that will impinge on the eye. This enables convergence-divergence motion-accommodation matching and a more realistic mixed reality display.

[0114] Content creators may place virtual objects at a certain depth from the viewer within the rendering engine due to imperfections in the generated light field (e.g., due to imperfections within the waveguides of the waveguide assembly 405), but the virtual objects may appear in focus at a depth different from what was intended. This can result in convergence-divergence motion-accommodation mismatches. In some cases, different portions of the virtual object may appear in focus at different depths. These depth mismatches may correspond to types of out-of-plane spatial errors such as those illustrated in FIGS. 10A-10E.

[0115] Accordingly, the present disclosure describes embodiments of a measurement system that can measure the quality of a light field generated by a display. Some such measurement systems can map the topology and quality of a light field generated by a display and provide information leading to an assessment of the quality of the light field generated by the display. Some such measurement systems can capture a vector light field (e.g., direction and amplitude) generated by a display and enable analysis of focus and depth imperfections within the display. Spatial and color calibration techniques for light field displays have been developed to utilize information generated by the measurement systems described herein. Embodiments of the measurement systems described herein have particular applications for light field displays (e.g., embodiments of display systems 80, 100), but this is not limiting, and other embodiments of the measurement systems can be used to measure light originating from any type of display. Embodiments of the measurement systems can be used to determine a 3D distortion field that can be used to derive useful spatial calibration information for a display. The measurement system can also be used for binocular calibration and calibration between monocular RGB and depth planes.

[0116] FIG. 17A illustrates an example of an object 1702 as viewed by an eye 304 having a normal light field. Object 1702 may correspond to an actual or virtual object generated using a light field having substantially no imperfections. The light rays 1706 associated with points on object 1702 appear to diverge from a single point and result in a point on object 1702 that appears to be in focus at a distance 1708 from eye 304.

[0117] FIG. 17B illustrates an example of an object 1710 as viewed with an incomplete light field. The object 1710 may correspond to a virtual object such as a virtual object generated using a display system (e.g., display system 400 as illustrated in FIGS. 4 and 6). Due to the imperfection within the generated light field, e.g., due to the imperfection within waveguides 420, 422, 424, 426, 428, 604, the light rays 1712, which are intended to correspond to specific points on the object 1710, may appear to diverge from different points or may exhibit a divergence different from that intended. As a result, the object 1710 may appear out of focus at the distance 1708. Additionally, different portions of the object 1710 may appear to be in focus at different depths or distances.

[0118] A measurement system can be used to measure the quality of a light field generated by a display. FIG. 18 illustrates an example of a measurement system 1800 for measuring the light field quality of a display 1802. The display 1802 generates a light field having light rays 1804 directed towards a camera 1806. The display device 1802 may correspond to a stacked waveguide assembly (e.g., stacked waveguide assembly 405 as illustrated in FIG. 4). The light rays 1804 are illustrated as being substantially parallel, but this is for illustration purposes, and the light rays 1804 may be projected in different directions (e.g., diverging) to convey different depths of one or more virtual objects represented within the light field. Additionally, the light rays 1804 may be non-parallel due to imperfections within the display 1802 (e.g., see FIG. 17B).

[0119] In some embodiments, camera 1806 can be used to capture at least a portion of the generated light field, for example, to measure the perceived depth of a virtual object represented within the light field. Camera 1806 can be configured to focus on a particular depth or distance (hereinafter also referred to as "depth of focus"). In some embodiments, this may be done using a lens with a small depth of field (DOF). For example, the DOF may be less than a Z-distance (e.g., less than the distance between the peak of depth map 1924 shown in FIG. 19C and the intended depth of focus 1922) where imperfections within the display typically deviate the depth of focus from the intended depth of focus. In other examples, the DOF may be less than a certain multiple of the distance between the camera and the display, and the coefficient may be less than about 0.1, less than about 0.01, less than about 0.001, etc. Camera 1806 may be configurable to capture a specific portion of the light field or the entire light field. Camera 1806 may be configured to capture a portion of the light field associated with a particular virtual object to be displayed using the light field. Camera 1806 may be positioned such that it can capture an image substantially similar to what would be perceived by eye 304. Camera 1806 and display 1802 can be movable relative to each other to map the light field. For example, the relative movement can be parallel to display 1802 (e.g., the X-direction shown in FIG. 18 or the Y-direction (not shown) perpendicular to X and Z as shown in FIG. 18) or perpendicular to display 1802 (e.g., the Z-direction shown in FIG. 18). In other implementations, scanning optics (not shown) can be used to scan camera 1806 and display 1802 relative to each other. In some embodiments, camera 1806 may be used to capture a portion of the generated light field to determine a distortion map (such as that shown in FIG. 8, etc.) that can be used to identify spatial errors (e.g., in-plane spatial errors as illustrated in FIGS. 9A-9E or out-of-plane spatial errors as illustrated in FIGS. 10A-10E) within the projected image.Additionally, camera 1806 may be used to identify luminance or color errors within the generated light field (e.g., as illustrated in FIGS. 11 - 15).

[0120] In some embodiments, camera 1806 is movable so as to be oriented in different directions. For example, although camera 1806 is illustrated as facing orthogonally to display 1802, camera 1806 may also face at a different angle to display 1802 and may be rotated (e.g., along the Y - rotation axis or the X - rotation axis) so as to enable camera 1806 to measure the light field generated by display 1802 in different directions or orientations.

[0121] In various embodiments, camera 1806 can be a digital camera, e.g., a short - focus digital camera. In other embodiments, camera 1806 can be a light - field camera.

[0122] Camera 1806 can be connected to a controller 1808 that can be used to control the depth of focus of camera 1806, the field of view of camera 1806, the exposure time, the relative movement between camera 1806 and display 1802, and the like. In some embodiments, controller 1808 may correspond to controller 450 as illustrated in FIG. 4. Controller 1808 can comprise a hardware processor and a non - transitory data storage device.

[0123] FIG. 19A is a schematic diagram of an example of an image 1900 that can be captured by a camera (e.g., camera 1806) focused on a particular depth of focus. The image 1900 may contain one or more in-focus regions 1902 and one or more out-of-focus regions 904. Since the camera 1806 can be configured to focus at different depths of focus, the regions of the image that are in focus or out of focus can vary. For example, if the camera is changed to focus at different depths of focus, the region 1902 may appear out of focus while portions of the region 1904 may come into focus. By capturing multiple images of the light field across a plurality of different depths of focus, the perceived depth for various regions of the light field can be determined. For example, each pixel of an image captured by the camera may be associated with a particular depth of focus corresponding to the depth of focus at which the portion of the light field corresponding to the pixel is in focus. A depth map or graph may be constructed to map the regions of the generated light field to their perceived depth. Additionally, the depth map or graph may also define the depth of focus intended to be projected by a display, thereby enabling a comparison between the intended depth of focus of a virtual object displayed in the light field and the actual measured depth of focus.

[0124] FIG. 19B is an example of a depth graph that schematically illustrates depth of focus measurement that may be performed by an embodiment of the measurement system 1800. Graph 1910 plots the measured depth of focus 1912 of the generated light field along a line across the light field emitted from display 1802 (e.g., along the horizontal X-axis of the light field as shown in FIG. 18). In some embodiments, graph 1910 may be generated by sweeping the depth of focus of camera 1806 over a plurality of different depths of focus. For example, camera 1806 may be focused at depth of focus 1914 (illustrated by the horizontal dashed line). In a perfect display, the light field generated by the display would be such that the actually measured depth of the virtual object would be exactly the intended depth, but in an actual display, the two may differ due to imperfections within the display. Thus, any region of the light field (e.g., region 1916) with a measured depth of focus approximating depth of focus 1914 may be perceived as being substantially in focus, while a region of the light field (e.g., region 1918) with a measured depth of focus significantly different from depth of focus 1914 may be perceived as being out of focus.

[0125] FIG. 19C illustrates an example of a depth map that may be generated based on one or more captured images. Depth map 1920 contains the intended depth position 1922 (illustrated as a horizontal plane in FIG. 19C) at which the image generated by display 1802 should be in focus, and the measured depth map 1924 indicating the depth of focus (Z) at which the image is actually in focus. Comparison between the intended depth of focus 1922 and the measured depth of focus 1924 enables identification and quantification of the imperfection of the light field generated by display 1802 across the field of view (FOV) of the display.

[0126] For example, if the intended depth of focus is Z0 for light to be focused at the horizontal position (X0, Y0), and the measured depth of focus at that position is Z, then (Z - Z0) is a measure of the focus imperfection of the display at the position (X0, Y0). In some implementations, the actual horizontal position (X, Y) at which the light rays are focused can be measured. In some such implementations, the vector measurement of the actual focus position relative to the intended focus position (X, Y, Z) - (X0, Y0, Z0) can be used to characterize the imperfection within the light field generated by the display. This vector measurement of the display imperfection provides a 3D representation of both in-plane and out-of-plane (e.g., dioptric) errors. In some embodiments, only the in-plane errors are measured (and calibrated) using the 2D vector error measurement (X, Y) - (X0, Y0). In some cases, the focus error can be determined for each pixel of the display. However, due to the large number of pixels (e.g., millions of pixels) in many displays, the focus error data can be determined only for a portion of the display or a group of pixels that sample the display (e.g., 10×10 or 100×100 samples across the display). The moiré pattern need not be square and can be designed to conform to the pixel structure of the display.

[0127] FIG. 20 is a flowchart of an example of a process 2001 for measuring the quality of a virtual target pattern generated using a light field display. The process 2001 can be performed by a measurement system 1800, e.g., a controller 1808. In some implementations, the virtual target pattern is a checkerboard pattern with an array of alternating light and dark regions. The checkerboard pattern may be used to sample a portion of the display (e.g., a 10×10 or 100×100, or other size checkerboard), or may have a size corresponding to the number of pixels in each dimension of the display. In other cases, per-pixel data can be obtained by successively turning on and off a group of one (or more) pixels and acquiring an image of the pixels that are turned on. The checkerboard pattern (or sequence of turning pixels on / off) may include a random probabilistic sequence of light and dark regions or a geometric pattern of light and dark regions or any other type of calibration pattern. Examples of checkerboard patterns and pixel on-off sequences are described below with reference to FIGS. 22-23B. In block 2002, an initial depth of focus may be set. In some embodiments, this may include steps of configuring a depth of focus lens on a camera. The initial depth of focus may correspond to any depth that can be represented by the virtual target pattern. For example, the initial depth may correspond to a minimum or maximum depth associated with the virtual target pattern.

[0128] In block 2004, an image of the virtual target pattern is captured at the selected depth of focus. In some embodiments, the image may include a focused portion and an out-of-focus portion. In some embodiments, the extent of the image may be focused on a particular virtual object associated with the virtual target pattern. In other embodiments, the image may correspond to the entire light field comprising a plurality of virtual objects. The image may include per-pixel depth of focus information across the virtual target pattern.

[0129] In block 2006, it is determined whether there is an additional depth of field at which an image should be captured. If it is determined that there is an additional depth of field, in block 2008, a new depth of field may be selected. In some embodiments, the number of depths of field may be at least partially based on the number of different depths that can be displayed by the display system (e.g., the number of depth planes 306 as illustrated in FIG. 3 or the number of waveguides within the waveguide assembly illustrated in FIG. 4). In some embodiments, when an image is focused on a particular virtual object, the range of the depth of field may be based on one or more depths associated with the virtual object (e.g., the minimum depth and the maximum depth associated with the virtual object).

[0130] If it is determined that there is no additional depth of field at which an image should be captured, in block 2010, the captured image of the virtual target pattern can be analyzed to identify the depth, Z, or lateral position (X, Y) at which different regions of the target pattern are actually in focus. For example, each captured image of the virtual target pattern corresponding to a particular depth of field may contain portions that are in focus and portions that are out of focus. In some embodiments, each image may be divided into one or more regions corresponding to regions of the light field. An autofocus technique may be used to determine the depth at which each region is in focus. In some embodiments, each region may correspond to a pixel.

[0131] In block 2012, the depth map may be created based at least in part on the measured depth of focus (or lateral position). The depth map may comprise any type of data structure or visualization that maps light field locations to depths of focus. For example, the depth map may comprise depth information for one or more than one pixel of the captured image (e.g., the depth of focus along the Z-axis or the depth of focus along the Z-axis combined with measurements of lateral focus positions (X and / or Y positions)). In some embodiments, the pixels may correspond to a pixel cloud associated with a target virtual object. Thus, the depth map may define the actual perceived depth of the virtual object when viewed through the display optics.

[0132] In block 2014, the depth map may be compared to one or more desired depths of focus, which correspond to the depth at which one or more virtual objects are intended to be displayed. Imperfections and / or deviations within the light field may be identified by examining the difference between the actual perceived depth of the virtual object and the desired depth of focus at which the virtual object is intended to appear.

[0133] In block 2006, error correction may be performed based at least in part on a comparison between the depth map and the desired depth of focus. Error correction can compensate for imperfections within the light field display or the content of the image projected from the display.

[0134] Process 2001 is repeated for each waveguide within the waveguide assembly 405 of the light field display and can map the respective imperfections of the waveguides. In some cases, there may be multiple waveguides corresponding to multiple depth planes and multiple waveguides corresponding to multiple colors (e.g., red (R), green (G), and blue (B)). For example, for some displays, there are three color planes for each depth plane, and thus a waveguide assembly with two depth planes can have 2×3 = 6 waveguides. Camera 1806 can be a camera sensitive to multiple colors or a combination of cameras each sensitive to a subset of colors. The depth of focus information obtained by measurement system 1800 can be used to determine the spatial distribution of the focus error and the distribution of the color (chromatic) imperfections of the display.

[0135] In some embodiments, instead of capturing multiple images at multiple different depths of focus (e.g., using a digital camera with a swept focus), the light field camera can be used to capture the light field generated by display 1802. The captured light field can be analyzed for focus and / or depth imperfections. By analyzing the vectors of the light rays within the captured light field, the depth of focus for various regions can be determined. The identified depth of focus may then be compared to one or more intended depths of focus and appropriate error correction may be performed (as in block 2016). For example, the vector measurement of the actual focus position (X, Y, Z) relative to the intended focus position (X0, Y0, Z0) can be determined as vector error = (X, Y, Z) - (X0, Y0, Z0) and can be used to characterize the imperfections within the light field generated by the display.

[0136] (Exemplary method for chromatically balancing a display) As described above, some implementations of full-color displays generate a trichromatic response on the viewer's retina by combining the red (R), green (G), and blue (B) wavelengths of light projected from the display. An ideal display has spatially uniform luminance with respect to these three color layers. However, actual displays may have some degree of luminance variation across the field of view due to hardware imperfections. If this variation is different for different color layers, it results in color non-uniformity across the field of view (FOV) of the display (e.g., as illustrated in FIG. 11). This disclosure describes embodiments of a method that attempts to correct color variations and make the color uniform across the FOV. For example, the intensity of individual color layers (e.g., R, G, and B) of the display can be adjusted so that the white point of the display can be substantially uniform across the FOV.

[0137] In some implementations, the light field measurement system described herein can be used to characterize the color balance of a display. For example, a digital color camera can capture an image of the display (e.g., using a measurement system 1800 as illustrated in FIG. 18) from which the color response of the display can be determined for some or all of the pixels of the display. In many displays, there are three color layers (e.g., R, G, and B), however, the method is not limited to RGB or three-color displays. The method can be applied to any number of color layers (e.g., 2, 3, 4, 5, 6, or more) and any color options (e.g., cyan, magenta, yellow, black).

[0138] Examples of measured color balance for a particular implementation of an RGB display are shown in FIGS. 14A (before color calibration) and 15 (after color calibration). FIGS. 14A and 15 include plots (1400, 1500, respectively) of the distribution of R, G, and B intensities (vertical axis) across the pixels (horizontal axis) of the display. FIG. 14B includes a plot 1408 of the maximum color imbalance (vertical axis) with respect to the pixels (horizontal axis) of the display, showing the average and average ± maximum error prior to color correction.

[0139] As described above, FIG. 14A shows that an uncalibrated display has substantial color non-uniformity across the pixels of the display. The red and blue responses are approximately the same, and the R and B intensities peak towards the right of plot 1400. The green response is generally smaller than the R or B response and decreases towards the right of plot 1400. FIG. 15 shows that after application of the color calibration described below, the calibrated display has a more uniform color response across the pixels of the display.

[0140] Embodiments of the color balance systems and methods described herein provide techniques for adjusting the intensities of at least some of the color layers in a multi-color display such that the white point of the display is substantially uniform across the FOV of the display. In various implementations, the display can be a light field display. For example, the display can have the ability to present color images in multiple depth planes to a viewer. Embodiments of the color balance systems and methods can be applied to chromatically balance displays 208 (FIG. 2), display system 400 (FIGS. 4-6), and display 2500 (FIGS. 25A, 25B, 26).

[0141] The human eye does not perceive light levels in a linear fashion. For example, compared to an ideal linear display, the human eye is more sensitive to changes in darker color tones than to similar changes in brighter color tones that allow the human visual system to operate over a wide range of brightness levels. Real-world displays also cannot provide a precisely linear brightness response. Additionally, digital images are often encoded to represent color tone levels that are more perceptually uniform. Human visual perception, display output, and image encoding are generally modeled to follow an approximate power-law relationship with respect to brightness or color levels. For example, the output level is proportional to the gamma power of the input level, i.e., V out ∝V in γ where. This non-linear power-law behavior is generally referred to as gamma correction, gamma encoding, or simply gamma.

[0142] In one embodiment, when the luminance flatness of the individual color layers within the display is substantially uniform across the FOV of the display, color balancing can include scaling the intensities of the individual color layers to achieve a uniform color balance across the display. The display can have suitable luminance flatness when the variation in luminance across the FOV of the display is less than 1%, less than 5%, or less than 10% in various embodiments. Due to the gamma response of the display and human visual perception, this simple scaling can have certain disadvantages in some cases.

[0143] When the color layers of the display do not have substantial luminance flatness, color balancing can involve more than simply scaling the intensities of the individual color layers. For example, color balancing can attempt to independently balance the white point at each pixel of the display (or across a group of pixels). In some such implementations, color balance across the FOV of the display can be achieved without also performing flatness of luminance across the FOV. Luminance flatness can be performed in addition to or as an alternative to color balancing.

[0144] The goal of chromatically balancing a display is for a human viewer of the display to perceive a uniform color balance across the FOV of the display. To measure and adjust the color balance of the display, a calibration camera (not a human eye) is used to record an image of the display output. The camera represents the human perception of the display output, and it can be assumed that if the camera image of the display is chromatically balanced, then the perception of the human viewer of the display will also be chromatically balanced.

[0145] In some implementations, the following model is used for the conversion between pixel values related to the color layers of the display and pixel values related to the colors measured by the calibration camera. In the following example, three color layers are assumed to be R, G, and B. However, this is for illustrative purposes only and not a limitation. In other cases, any number and hue of color layers can be used in conjunction with embodiments of the color balance technique. Further, appropriate scaling between the pixel size of the display and the camera can be considered prior to the application of the model.

Number

[0146] In Equation (1), [Rd, Gd, Bd] represents the intensity values for the RGB image sent to the display. In many cases (e.g., standard RGB or sRGB), the intensity values are from 0 to 255. Gamma1{} represents the first non-linear gamma function (with exponent γ1) that maps the display color levels to the intermediate color representation [R1 G1 B1]. Coupling() represents a function that maps the color values [R1 G1 B1] to a second intermediate color representation [R2 G2 B2]. The coupling() function can be a linear function, e.g., a 3×3 matrix (in the case of three color layers). In other implementations, the coupling() function can be non-linear. Gamma2{} represents the second non-linear gamma function (with exponent γ2) that maps the second intermediate color representation [R2 G2 B2] to the pixel intensities [Rc Gc Bc] aligned by the calibrated camera.

[0147] In some implementations, the first and second gamma functions are global functions across the FOV of the display (e.g., the exponents γ1 and γ2 are constant across the FOV). Coupling() can be a local (pixel-dependent) function that varies pixel-by-pixel across the FOV. The pixel-by-pixel color mapping provided by the coupling() function enables pixel-by-pixel color balance.

[0148] To determine the functions Gamma1{}, Gamma2{}, and Coupling(), one or more images of a series of the display can be captured by a camera and analyzed by an analysis system programmed to perform an iterative optimization algorithm (e.g., hill climbing, local search, simplex method, genetic algorithm, etc.) to find a suitable fit for the gamma and coupling functions that provides a reasonable color balance for the display. The analysis system may use feedback during the iterative process by capturing additional images of the display as the analysis system searches for a suitable fit for the gamma and coupling functions. For example, the functions Gamma1{}, Gamma2{}, and Coupling() can be determined by iteratively adjusting these functions to improve or optimize the color balance of the camera images across the FOV of the display. The functions can be iteratively adjusted until the white point of the camera images acquired during the iterative process is substantially uniform across the FOV of the display. In various implementations, a substantially uniform white point distribution is associated with a variation in the white point across the FOV that is less than 10%, less than 5%, or less than 1% of the white point value within the measured color system. For example, a color space provided by the International Commission on Illumination (CIE) may be used. In some implementations, a substantially uniform white point distribution may be associated with a variation in the white point that is less than a threshold amount based on just noticeable differences (JND) in the color space. In some implementations, the gamma transfer functions Gamma1{} and Gamma2{} are first calculated iteratively, and then once the gamma functions (e.g., exponents γ1 and γ2) are calculated, the Coupling() function is calculated.

[0149] The production process for calibrating a display in a manufacturing environment can automatically characterize the display as it is transported along the production line. For example, at suitable points in the production process, the calibration cameras and analysis systems described herein can perform iterative analysis to determine the gamma transfer function and coupling function for a particular display and store the resulting gamma and coupling functions in memory associated with the display. The display can then automatically perform color balancing.

[0150] Once the gamma transfer function, Gamma1{} and Gamma2{}, and the Coupling() function are known for a particular display during its use, appropriate display pixel values [Rd Gd Bd] can be input into Equation (1) to achieve a chromatically balanced output. For example, the gamma exponent and Coupling() function determined for a particular display can be stored in memory accessible to the display, accessed, and used to transform the input image pixel color values to provide a chromatically balanced output from the display. In some implementations, the local processing and data module 224 of the wearable display system 200 can store the gamma conversion and coupling functions, and the processing module can use Equation (1) to output a real-time chromatically balanced image (Figure 2). In other implementations, the controller 450 of the display system 400 can perform color balancing based on Equation (1) and the stored gamma and coupling functions (Figure 4). In yet other implementations, the dynamic calibration processor 2610 of the dynamic calibration system 2600 can perform color balancing for the display 2500 using Equation (1) and the stored gamma and coupling functions, as will be described below (Figure 26).

[0151] Embodiments of method 2700 or process flow 2805 for dynamically calibrating a display based on eye tracking, each described with reference to FIGS. 27 and 28, which are described in more detail below, can perform color balance and other error correction / calibration functions. For example, the calibration accessed at block 2720 of method 2700 can include a gamma and a binding function, and at block 2730, the color imperfection of the display can be corrected by use of equation (1) and the accessed gamma and binding function. As another example, block 2880 of process flow 2805 can access the gamma and binding functions and apply them during calibration.

[0152] FIG. 21 is a flowchart illustrating an example of a method 2150 for calibrating a display. The display can be a light field display. The display can be display 208 (FIG. 2), display system 400 (FIGS. 4 - 6), and display 2500 (FIGS. 25A, 25B, 26). Method 2150 can be performed by an analysis system (including an analysis program executed by computer hardware such as a camera and measurement system 1800 shown in FIG. 18) as part of a production line of a manufacturing process for the display (e.g., as part of process 2805 described with reference to FIG. 28). Method 2150 can be performed as part of camera calibration, as described with reference to block 1602 of process 1600 described with reference to FIG. 16. In some implementations, method 2700 applies equation (1) to determine an appropriate transformation between the display and a camera (assumed to represent the visual perception of a human viewer of the display). At block 2160, an image of the display is acquired by the camera. At block 2170, global transformation parameters for the transformation between the display and the camera are determined. The global transformation parameters can include parameters that do not vary across the FOV of the display (e.g., parameters that are not pixel - dependent). For example, the global transformation parameters can include Gamma1{} and Gamma2{} functions. In some cases, method 2150 may return to block 2160 and acquire one or more additional images as part of an iterative feedback process for determining the global transformation parameters. After a suitable fit to the global transformation parameters is obtained, method 2150 moves to block 2180, where local (e.g., pixel - dependent) transformation parameters are fitted to the camera image. For example, the local transformation parameters can include a Coupling() function (e.g., the value of this function at pixel locations across the FOV of the display). In some cases, method 2150 may return to block 2160 and acquire one or more additional images as part of an iterative feedback process for determining the local transformation parameters.In some implementations, after obtaining the additional image at block 2160, instead of proceeding to block 2170 since the global transformation parameters were previously determined, method 2150 may return to block 2180 and continue to adapt the local transformation parameters. After a suitable adaptation to the local transformation parameters has been made to the camera image, method 2150 proceeds to block 2190, where the local and global transformation parameters are stored in a memory associated with the display (e.g., local data module 71). As described above, in block 2720 of method 2700 for dynamically calibrating the display, the local and global transformation parameters can be accessed as part of the calibration for the display, and in block 2730, the local and global transformation parameters and equation (1) are applied to generate a chromatically balanced image from the display.

[0153] Although described with respect to the case of color balance for a display, the present system and method are not so limited and can be applied to correct other color (or spatial) imperfections of the display (e.g., any of the color or spatial imperfections described above). For example, as described above, the display may exhibit luminance flatness variations, and embodiments of the disclosed analysis techniques can determine a luminance flatness calibration that corrects for luminance flatness imperfections. Additionally, or alternatively, the display may exhibit spatial imperfections including in-plane translation, rotation, scaling, or warping errors and out-of-plane (e.g., depth of focus) errors. Embodiments of the disclosed analysis techniques can determine a calibration for some or all of such spatial errors.

[0154] (Example of Display Calibration Using a Calibration Pattern) The imperfection within the display can manifest by spatially or chromatically distorting the virtual object projected by the display. To correct these distortions, the display may be calibrated by first measuring the distortion and then performing any necessary error correction (e.g., using the measurement system 1800 illustrated in FIG. 18). Display calibration can involve using the display to project a calibration pattern, such as a checkerboard pattern (e.g., as illustrated in FIG. 7), and capturing the resulting image with a camera. The captured image can then be processed to determine the distortion at the feature point locations of the calibration pattern by quantifying the error between the expected position and the measured position of the pattern feature points. For displays with separate color layers (e.g., red (R), green (G), and blue (B) layers), this calibration can also correct for color alignment and image quality.

[0155] FIG. 22 illustrates an exemplary calibration system 2200 using a calibration pattern. The display 2202 can be configured to project a calibration pattern 2204 as a generated light field 2206 that can be captured using an imaging device such as a camera 2208. In some embodiments, the display 2202 comprises a stacked waveguide assembly (e.g., as illustrated in FIGS. 4 or 6) or other type of light field display. In some embodiments, the camera 2208 (or the display 2202) is configured to be movable such that the system 2200 will be able to capture images of the light field 706 from different lateral positions, depths, or angles. In some embodiments, the calibration system 2200 may be similar to the measurement system 1800 of FIG. 18. For example, the display 2202, light field 2206, and camera 2208 may correspond to the display 1802, light field 1804, and camera 1806 of the measurement system 1800.

[0156] In this embodiment, the calibration pattern 2204 comprises a grating pattern, and different regions have different (e.g., alternating) optical characteristics such as, for example, luminance (e.g., light or dark color), color, hue, saturation, color, etc. The grating pattern can be a regular pattern (e.g., as shown in FIG. 22) or an irregular pattern. The calibration pattern 2204 contains a plurality of feature points that can be used to measure the amount of distortion in an image captured by the camera 2208. For example, the feature points of the grating pattern include points on the boundary and corners between the grids of the grating or points at the center of the grating. The calibration pattern 2204 can be the same size as or smaller than the display 2202. The smaller calibration pattern can be offset across the display 2202, and the camera 2208 may capture multiple images of the calibration pattern 2204 as it is offset across the display when the system 2200 measures the distortion of the display 2202. In some implementations, the calibration pattern 2204 can be probabilistically sampled according to a mathematically optimized sequence.

[0157] Due to errors within the display 2202 (e.g., imperfections in one or more waveguides or lenses), the light field 2206 can contain imperfections that distort the appearance of virtual objects or patterns within the light field. This can result in a deviation between the expected focal position (lateral or depth) of the feature points on the calibration pattern 2204 and their actual measured position in the image captured by the camera 2208. By comparing the actual measured positions of the feature points of the calibration pattern 2204 with their expected positions, the deviation caused by the distortion can be identified and measured. In some implementations, the calibration pattern includes color information so that the color error of the display 2202 can be quantified by the system 2200. In some embodiments, a distortion map may be generated for use in error correction of the spatial or color error of the display 2202 (e.g., as shown in FIG. 8).

[0158] In some implementations, each cell 2304 within the calibration pattern 2204 may correspond to a single pixel of the display 2202 that enables a direct measurement of the per-pixel display imperfection. In other implementations, each cell 2304 corresponds to a plurality of pixels (e.g., an N×M grid of pixels, where at least one of N or M is greater than 1). In some such implementations, the coarse quality of the calibration pattern means that distortion information can be obtained at sample points and interpolated to obtain per-pixel distortion information. For example, in the grid pattern shown in FIG. 23A, distortion information may be measured with respect to pattern locations corresponding to feature points 2302 such as points on the boundaries, corners, or centers of the grid. Distortion information regarding other points within the grid regions 2304 of the pattern can be inferred or interpolated with the measured distortion values associated with neighboring feature points 2302.

[0159] The moiré projection capture procedure identifies feature points (e.g., the edges of the grid) and quantifies the error between the expected and measured positions for distortion calibration. The feature points may be of low density compared to the number of pixels within the display. For example, a high-resolution display may have millions of pixels (e.g., 2.1 million pixels for a 1920×1080 pixel resolution), while the number of grids 804 within the calibration pattern may be substantially fewer (e.g., for 50×50, 100×100, 500×500 patterns). Thus, embodiments of the system 2200 that use a single projection capture approach result in sampled measurements that can be interpolated to estimate per-pixel distortion.

[0160] To obtain accurate per-pixel distortion information for a display, embodiments of system 2200 can automate the task of obtaining distortion information by implementing different or offset calibration patterns. Different calibration patterns can be projected, or the same pattern can be gradually offset so that the entire pixel space of display 2202 is measured. Automated image projection and capture or different offset calibration patterns enable a pixel-precise mapping of the distortion of display 2202.

[0161] By automatically repeating grid pattern projection capture (but using, for example, a calibration pattern offset by 1 pixel at a time), system 2200 can obtain improved per-pixel distortion information. For example, camera 2208 can obtain an image of the pattern each time the pattern is offset. Using each repeated image capture, the feature points of the projected calibration pattern correspond to different sets of pixels. The offset of the calibration pattern can be repeated until a high-density sampling of the distortion field of the display is obtained. For example, the grid pattern can be projected and offset through several positions corresponding to the pixels of the grid of the grid pattern, enabling distortion information to be measured for each pixel of the display. In other implementations, the offset can be different from 1 pixel, for example, 2, 3, 4, 5, 8, 16, or more pixels. The offset can be different for different directions of the display, for example, the x-offset need not be the same as the y-offset.

[0162] It should be understood that the present disclosure mainly refers to a grid pattern for the purposes of the examples, although other types of patterns may also be used. For example, other geometric patterns can be used, random probabilistic patterns can be used, or any other type of calibration or test pattern can be used. In some embodiments, a calibration pattern is used in which only a single pixel within the display is turned on at a time. FIG. 23B illustrates an exemplary single pixel calibration pattern in which only a single pixel 2306 is turned on. From the captured image of each resulting frame, the pixel-by-pixel transfer function from the display device to the viewer scene can be quantified. After each image capture, the location of the pixel 2306 being displayed may be offset by a set distance (e.g., a single pixel) across the display (e.g., in the direction indicated by arrow 2308). By automatically sweeping through each pixel of the display, a complete quantification of the quality of the display device can be obtained. In other implementations, the offset of the illuminated pixel can be a different number of pixels, e.g., 2, 3, 4, 5, 8, 16, or more than that, the offset can be different for different lateral directions on the display, or multiple pixels (not a single pixel as shown in FIG. 23B) can be illuminated in each image capture.

[0163] FIG. 24 is a flowchart of an exemplary process 2400 for performing automated display calibration. Process 2400 can be performed, for example, as part of processes 2700 and 2805 described with reference to FIGS. 27 and 28. In block 2402, a calibration pattern is projected by the display. The calibration pattern may comprise any pattern having one or more feature points that can be generated by the display. In some embodiments, the calibration pattern comprises a grid pattern. In other embodiments, other types of calibration patterns, such as a single pixel pattern, may be used.

[0164] In block 2404, an image of the calibration pattern to be displayed is captured using a camera or other type of image capture device. If an error or imperfection exists within the light field generated by the display, a portion of the displayed calibration pattern may be distorted, and one or more feature points within the calibration pattern may appear in locations different from those expected. The brightness or color of the image may differ from that expected from the calibration pattern.

[0165] In block 2406, a distortion corresponding to the error between the expected location of a feature point of the calibration pattern and the captured location of the feature point is determined. For example, for a single-pixel calibration pattern, the distortion information can be calculated for a particular pixel location of the pattern. For a checkerboard pattern, the distortion information can be calculated for the pixels corresponding to the feature points of the checkerboard (e.g., the edges, corners, or center of the grid). In some implementations, a brightness or color error between the brightness or color of the calibration pattern and the corresponding brightness or color of the captured image of the calibration pattern is determined.

[0166] In block 2408, a determination is made as to whether any additional positions exist across the display onto which the calibration pattern is to be projected. If it is determined that additional positions exist, in block 2410, the calibration pattern may be offset and projected at the new positions, and an image of the calibration pattern may be captured (block 2404) and used to calculate the amount of distortion (block 2406). In some embodiments, the number of different positions for displaying the calibration pattern is based on the calibration pattern used. For example, for a single-pixel calibration pattern, the number of positions may correspond to the total number of pixels that can be displayed by the display. For a checkerboard pattern, the number of positions may be based on the number of pixels within each grid.

[0167] Once the calibration pattern is displayed at all desired positions, at block 912, the calculated distortions may be aggregated and used to generate a distortion map that includes distortion information for each pixel (or group of pixels) of the display. The distortion information may include spatial distortion due to focus error (e.g., in-plane error or out-of-plane error) or color error (e.g., luminance or color error). At block 2414, error correction may be performed on the display using the calculated distortion map. For example, the distortion information (e.g., the distortion map) may be stored by data modules 224, 232 of the wearable display system 200 illustrated in FIG. 2. The processing modules 224, 228 of the wearable display system 200 may use the distortion information to correct spatial or color errors within the display 208 such that the image perceived by the wearer 204 of the display system 80 is at least partially compensated.

[0168] In some embodiments, the process 2400 illustrated in FIG. 24 may be performed for a light field display. As an example, the process 2400 may be performed for each waveguide within the waveguide assembly 405 of the light field display to calibrate each of the waveguides. In some cases, there may be multiple waveguides corresponding to multiple depth planes and multiple waveguides corresponding to multiple colors (e.g., red (R), green (G), and blue (B)). For example, for some displays having three color planes per depth plane, thus, a waveguide assembly with two depth planes can have 2×3 = 6 waveguides. In addition to pixel position, color alignment and quality may also be calibrated to correct for color (chromatic) imperfections of the display. For example, the camera 2208 may be a combination of cameras sensitive to multiple colors or cameras sensitive to respective subsets of colors, and may be used to capture an image of the light field 2208 where a deviation between the captured color or luminance values of the projected pattern 2204 and the expected color or luminance values can be identified.

[0169] (Exemplary Waveguide Display) FIG. 25A is a top view schematically illustrating an embodiment of a display 2500 that includes a waveguide 2505, an internal coupling optical element 2507, a light re-dispersing element 2511, and an external coupling optical element 2509. FIG. 25B is a cross-sectional view of the display 2500 depicted in FIG. 25A along axis A-A', schematically illustrated.

[0170] The waveguide 2505 may be part of a stack of waveguides 405 in the display system 400 shown in FIG. 4. For example, the waveguide 2505 may correspond to one of waveguides 420, 422, 424, 426, 428, and the external coupling optical element 2509 may correspond to the light extraction optical elements 460, 462, 464, 466, 468 of the display system 400.

[0171] The display 2500 is configured such that incoming incident light of different wavelengths, represented by light rays 2503i1, 2503i2, and 2503i3 (solid line, dashed line, and dashed-dotted line respectively), is coupled into the waveguide 2505 by the internal coupling optical element 2507. The incoming incident light to the waveguide 2505 can be projected from an image input device (such as one of the image input devices 440, 442, 444, 446, 448 shown in FIG. 4). The internal coupling optical element 2507 can be configured to couple the wavelength of the incident light into the waveguide 2505 at an appropriate angle to assist propagation through the waveguide 2505 via total internal reflection (TIR).

[0172] The light dispersing element 2511 can be disposed within the optical path through which different wavelengths of light 2503i1, 2503i2, and 2503i3 propagate through the waveguide 2505. The light dispersing element 2511 can be configured to redirect a portion of the light from the internal coupling optical element 2507 towards the external coupling optical element 2509, thereby expanding the beam size that interacts with the light along the propagation direction. Thus, the light dispersing element 2511 can be advantageous in expanding the exit pupil of the display device 2500. In some embodiments, the light dispersing element 2511 can thus function as an orthogonal pupil expander (OPE).

[0173] The external coupling optical element 2509 can be configured to redirect the internally coupled light that is incident on the element 2509 from the x-y plane of the waveguide 2505 at an appropriate angle (e.g., in the z-direction) and efficiency to facilitate an appropriate overlay of light at different wavelengths and different depth planes such that a viewer can perceive a color image of good visual quality. The external coupling optical element 2509 can have a refractive power that provides divergence to the light exiting through the waveguide 2505 such that the image formed by the light exiting through the waveguide 2505 appears (to the viewer) to originate from a certain depth. The external coupling optical element 2509 can expand the exit pupil of the display 2500 and can be referred to as an exit pupil expander (EPE) that directs light towards the viewer's eye.

[0174] The internal coupling optical element 2507, the external coupling optical element 1009, and the light dispersion element 2511 can include a plurality of gratings such as, for example, an analog surface relief grating (ASR), a binary surface relief structure (BSR), a volume holographic optical element (VHOE), a digital surface relief structure, and / or a volume phase holographic material (e.g., a hologram recorded in a volume phase holographic material), or a switchable diffractive optical element (e.g., a polymer dispersed liquid crystal (PDLC) grating). In various embodiments, the internal coupling optical element 2507 can include an optical component including one or more optical prisms, or one or more diffractive and / or refractive elements. Various sets of diffractive or grating structures can be disposed on the waveguide by using processing methods such as injection compression molding, UV replication, or nanoimprinting of the diffractive structure.

[0175] The internal coupling optical element 2507, the external coupling optical element 1009, or the light dispersion element 2511 need not be a single element (e.g., as depicted schematically in FIGS. 25A and 25B), and each such element can include a plurality of such elements. These elements can be disposed on one (or both) of the major surfaces 2505a, 2505b of the waveguide 2505. In the embodiments shown in FIGS. 25A and 25B, the internal coupling optical element 2507, the external coupling optical element 2509, and the light dispersion element 2511 are disposed on the major surface 2505a of the waveguide 2505.

[0176] In some embodiments, one or more wavelength selective filters may be integrated with or disposed adjacent to the internal coupling optical element 2507, the external coupling optical element 2509, or the light dispersion element 2511. The display 2500 illustrated in FIG. 25A includes a wavelength selective filter 2513 integrated within or on the surface of the waveguide 2505. The wavelength selective filter can be configured to filter out a portion of the light at one or more wavelengths that can propagate along various directions within the waveguide 2505. The wavelength selective filter can be an absorptive filter such as a color ribbon absorber.

[0177] (Example of Dynamic Calibration of AR or VR Display Based on Eye Tracking) The display system can generate calibrated (spatially and / or chromatically) and improved quality images. In the case of a certain eyepiece display (for example, the stacked waveguide assembly 405 shown in FIG. 4 as used within the display 208 shown in FIG. 2 or the display 2500 described with reference to FIGS. 25A and 25B), this calibration is reasonably accurate with respect to a nominally fixed eye position (for example, the wearer looking straight through the display 208), but may not be very accurate with respect to other eye posture directions or positions. Therefore, the calibration for the display may depend on the eye position or eye direction. If calibration for only a single (for example, reference) position is used, there may be errors that are not corrected when the wearer is looking in a different direction (for example, away from the reference position).

[0178] This disclosure also describes an example of dynamic calibration for a wearable display system 400 that uses eye tracking, and the spatial and / or color calibration can change in response to changes in the eye position (or in some cases, the eye direction). One such calibration provides a feedforward calibration system that can result in the maintenance of high-quality images for a wide range of eye movements. In some implementations, the calibration is performed in real time via a hardware processor (for example, the processing modules 224, 228 of the wearable display system 200 or the controller 450 of the display system 400) without the addition of special hardware.

[0179] Calibration can compensate (or correct) for spatial errors and / or color (chromatic) errors within the field of view of the display. For example, spatial errors can include in-plane translation, rotation, scaling, or warping errors and out-of-plane (for example, depth of focus) errors. Color errors can include luminance flatness or color uniformity errors for each color that can be displayed (for example, R, G, and B).

[0180] FIG. 26 schematically illustrates an embodiment of a dynamic calibration system 2600 for a display 2500, and the calibration can be applied to correct spatial and / or color errors at a grid of reference positions (indicated by dots 2602). The dynamic calibration system 2600 can include the display 2500, an imaging system facing inward such as the eye-tracking camera 500, and a dynamic calibration processor 2610 (which reads and applies the calibration). FIG. 26 schematically illustrates another embodiment of the display 2500, including the embodiment of the optical elements described with reference to FIGS. 25A and 25B. The external coupling optical element 2509 directs light to the viewer's eye. As the viewer's eye is positioned at different positions 2602 relative to the external coupling optical element 2509, the optical calibration of the display 2500 with respect to that particular eye position (schematically shown as dots 2602 in FIG. 26) may be different. For example, the calibration when the eye is positioned over position 2602a near the center of the external coupling optical element 2509 is different from the calibration when the eye is positioned over position 2602b towards the upper left corner of the external coupling optical element 2509, and similarly may be different from the calibration with respect to any of the other exemplary positions 2602 on the optical element 2509.

[0181] As the user's eyes move relative to the display, the field of view (FOV) of the display remains substantially the same, but spatial and / or color distortion within the display can change as the eyes translate parallel to the display. Since the FOV includes the range of angles at which the image is presented to the user, calibration data (at a given position relative to the display) can account for substantially all orientations or viewing angles of the eyes. For example, if the user orients their vision at different angles (while maintaining the same position relative to the display), the user may simply view different portions of the image with the same overall distortion. Thus, at any given position, as the orientation of the eyes changes (e.g., the direction of the eye line of sight changes), the visual field of the eyes generally remains within the FOV of the display, and the same calibration (for that given eye position) can be used for substantially all eye orientations. Accordingly, certain embodiments of the calibration system utilize position-dependent calibration, which, additionally, is not orientation-dependent.

[0182] Note that dots 2602, 2602a, 2602b are for reference only and do not form part of the external coupling optical element 2509 or the display 2500. Further, although nine positions 2602 within a 3×3 grid are schematically illustrated in FIG. 26, this is for illustrative purposes only, and it should be understood that the number (or arrangement) of positions for calibration of the display 2500 may differ from that shown in FIG. 26. For example, in various implementations, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 16, 20, 25, 100, 256, or more calibration positions are used. The calibration positions can be arranged in a 2×2, 3×3, 4×4, 5×5, 6×6, 7×7, 9×9, or other dimensional grid or other pattern or arrangement of positions.

[0183] Calibration for one or more positions on display 2500 can be determined using a light field measurement system that measures errors within a calibration pattern (e.g., gridlines) projected from the display. The calibration can depend on the position across the display where the display is viewed. For example, the measurement system can sweep an eye proxy camera relative to the display to simulate a range of positions with respect to the user's eye (e.g., by relatively translating the camera and the display). As the camera is swept relative to the display at each sample point 2602, the measurement system can build a calibration (correction) value, thereby resulting in a set of calibration vs. eye proxy positions. Calibration for a particular display may be stored as a look-up table (LUT) (or other efficient data structure) by data modules 224, 228 of wearable display system 200. In other implementations, an analytical model can be fit to calibration data obtained from the measurement system, and the fit analytical model can be stored by wearable display system 200. Other modeling or data reference methods can also be used to store the calibration. As described above, the calibration can include spatial and / or color correction generated for each calibration position of the display (e.g., a 3×3 grid of calibration positions for exemplary display 2500 shown in FIG. 26). Note that in various implementations, to capture the calibration, the display is swept (translated) relative to a fixed camera, the camera is swept (translated) relative to a fixed display, or both the camera and the display are swept (translated) relative to each other.

[0184] In implementations where the field of view (FOV) of the eye proxy camera exceeds the FOV of the display, placing calibration cameras at several discrete positions relative to the display (e.g., across the positions indicated by dot 2602) and taking one or more calibration images provides sufficient information about the imperfection of the display to determine calibration for each discrete position. In some such implementations, the camera may be able to capture the entire FOV of the display and may not need to vary the orientation (e.g., the direction it is facing) of the camera at each of the positions 2602. In other implementations, the orientation of the calibration camera (at each position 2602) may be varied to acquire additional images and map the FOV of the display (e.g., when the FOV of the camera is less than the FOV of the display).

[0185] The calibration positions can represent the eye positions relative to the display 2500. For example, the wearer of the display 2500 will typically position the display such that the wearer's eye (in the x-y plane) is near the approximate center of the external coupling optical element 2509, e.g., such that the wearer's eye is positioned across the positions 2602a. Calibration for the positions 2602a (near the center of the optical element 2509) thus corresponds to light propagating substantially perpendicular to the display 2500 (e.g., substantially along the z-direction) and can be applied by the dynamic calibration processor 2610. If the wearer's eye moves upward and to the left across the positions 2602b (near the upper left corner of the optical element 2509), calibration for the positions 2602b can be applied by the processor 2510. The eye tracking camera 500 can image the eye (e.g., in real time), and the dynamic calibration processor 2510 can use the eye tracking data to determine the position of the eye, select the appropriate calibration (based on the determined eye position), and apply the calibration to the display. In some implementations, the eye position is determined from the corneal position and the line of sight direction. Additionally, in other embodiments, the eye orientation (e.g., the line of sight direction) may be determined and orientation-dependent calibration may be used.

[0186] An embodiment of the wearable display system 200 can include an embodiment of the dynamic calibration system 2600 schematically illustrated in FIG. 8. For example, an eye tracking camera 500 (described with reference to FIG. 4) can be attached to the frame of the wearable display system 200 and can dynamically measure the eye pose (e.g., eye position or eye direction) of the wearer. Images from the camera 500 are used by the dynamic calibration processor 2610 to determine the eye pose of the wearer in real time or near real time. When the system being dynamically calibrated is in operation, the eye tracking camera can inform the dynamic calibration processor 2610 of the wearer's current eye pose in real time or near real time. The dynamic calibration processor 2610 can fetch and apply an appropriate calibration (e.g., an appropriate calibration LUT stored in data modules 224, 228) based on the measured eye pose (e.g., position or orientation). If the wearer is not looking directly at the stored calibration position or the wearer's eye is not positioned directly above the calibration position, the dynamic calibration processor can interpolate (or extrapolate) between calibrations for nearby calibration positions (e.g., including at least the calibration position closest to the wearer's eye pose) and determine an appropriate calibration to apply for the wearer's current eye pose. Thus, the display system 200 (with the dynamic calibration system 2600) can correct for imperfections (spatial or color) in the display and thereby provide the wearer with a good quality color image. As described herein, in some cases, the calibration depends on the eye position relative to the display rather than the eye orientation (e.g., line of sight direction), but this is not limiting.

[0187] The dynamic calibration processor 2610 can be implemented as software stored in a memory (e.g., data modules 224, 228), and the software instructions can be executed by one or both of the processing modules 224, 228 or by the controller 450. Thus, continuous adjustment of the calibration can result in high quality images over a wide range of input movements of the wearer's eyes.

[0188] In some implementations, the calibration is stored at a reduced number of calibration positions (e.g., a 2×2 or 3×3 grid), reducing the amount of data storage. As described above, the dynamic calibration processor can interpolate or extrapolate to determine calibrations for eye poses that are not at the directly stored calibration positions.

[0189] In some embodiments, the wearable display system 200 uses a single eye-tracking camera to measure the pose of the wearer's single eye, and the dynamic calibration processor 2610 infers the pose of the wearer's other eye with respect to the display system 200 (since both eyes are typically oriented in the same direction). In other embodiments, the wearable display system 200 uses two eye-tracking cameras (one per eye) to independently measure the pose of each eye. In some embodiments, separate calibrations are stored within the wearable system for each display (often, there are two displays, one in front of each of the wearer's eyes, and thus two calibrations are stored). In other embodiments, a single calibration (e.g., an average calibration) is stored and used for all of the displays within the wearable system 200.

[0190] An eye-tracking camera (or other type of inward-facing imaging system) can image the area around the user's eyes on the face. The area around the eyes can include the eyes and the area around the eyes. For example, the area around the eyes can include the eyes (such as eye sockets, etc.) and the area around the eyes. The area around the eyes may include, for example, eyebrows, a part of the nose, cheeks, and the forehead. The area around the eyes can have various characteristics, such as the shape of the eyebrows, the inner corners of the eyes, the characteristics of the eyelids, etc. In some implementations, one or more of these characteristics may be represented by key points, point clouds, or other types of mathematical expressions. The wearable device can identify these characteristics in the image and use these characteristics to determine the relative position between the wearable display system and the user's face. In one embodiment, the wearable display system 200 may calculate the relative position separately for each eye. For example, when the wearable device has one or two eye cameras configured to image one eye of the user respectively, the wearable device may calculate one relative position between the left eye and the wearable display system and another relative position between the right eye and the wearable display system. The wearable device can also track the relative positions regarding individual eyes separately. Since the relative position between the left eye and the wearable display system may be different from the relative position between the right eye and the wearable display system (such as when the wearable system is tilted to one side), the adjustment for the rendering location of the virtual object may be different for the left-eye display and the right-eye display.

[0191] A wearable display system can calculate and track periorbital features using neural networks or visual key point techniques such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoint (FREAK), etc. In some embodiments, certain face features may be tracked using detectors specifically designed for those particular face features. For example, periorbital features such as the inner canthus, nose features, mouth corners, etc. may be separately identified and tracked using various algorithms. Separately tracking one or more of these periorbital features can be advantageous because they are subject to substantial movement while the user is expressing themselves or during speech. The detectors associated with these periorbital features may take into account the range of mobility. As an example, some face features may be more likely to move in one direction and be stable in other directions (e.g., eyebrows tend to move up and down but not side to side). The wearable system can statistically analyze the movement of face features. These statistics may be used to determine the likelihood that a face feature will move in a certain direction. In some embodiments, one or more face features may be removed or untracked. For example, the wearable display system may ignore eye movement when tracking the position of the periorbital region.

[0192] The wearable display system can also identify and track facial features using sequential Bayesian estimators (e.g., Kalman filters, extended Kalman filters, etc.) and visual simultaneous localization and mapping (vSLAM) techniques such as bundle adjustment. In some embodiments, the wearable device may be configured to enable depth perception. For example, the wearable system can construct a high-density map that encodes at least a portion of the face from data acquired by one or more cameras. Instead of a keypoint map, the high-density map may comprise face patches or regions whose 3D shape is measured. The patches or regions may be used to calculate the location of the HMD relative to the user's face using techniques such as the iterative closest point algorithm or similar algorithms.

[0193] In some implementations, the images acquired by the eye cameras may be low-resolution images because the wearable display system 200 does not need high-quality images to track periorbital features. Additionally, or alternatively, the resolution of the images obtained from the eye imagers may be downsampled relative to their original resolution or the resolution used for other purposes (e.g., eye tracking).

[0194] The wearable display system 200 can use various techniques to analyze the images obtained by one or both eye cameras and determine the relative position between the display of the display system and the user. The relative position between the display and the user's eyes may be the normal rest position of the display system 200 relative to the user's face. The normal rest position of the display system 200 may be determined during the initial phase of the wearable system. For example, when the user first uses the wearable system, the wearable system may construct a face model (e.g., a map of the user's face) and determine the normal rest position of the display relative to the user's eyes based on the face model.

[0195] While the user is using the wearable system 200, the wearable system can track the relative position between the display and the user using various techniques. For example, the wearable device can identify and track visual fiducials associated with periorbital features. The wearable system can also match the identified face region in the acquired image to a high-density map of the user's face and calculate the location of the display relative to the face.

[0196] Thus, various eye-tracking or face-imaging techniques can be used to determine (statically or dynamically) the relative position between the user's eyes and the display of the display system. The display system 200 can then select and apply appropriate spatial and / or color calibration based at least in part on the determined relative eye position, as further described herein.

[0197] Figure 27 is a flowchart illustrating an exemplary method 2700 for dynamically calibrating a display based on eye tracking. Method 2700 can be performed by a dynamic calibration system 2600. At block 2710, the user's eyes are tracked to determine the user's eye position relative to the display. For example, the camera 500 of the display system 2600 can determine the user's eye position. One or both eyes can be tracked. At block 2720, a calibration based on the determined eye position is accessed. At block 2730, the calibration is applied to the display to correct for spatial and / or color imperfections within the display. For example, the dynamic calibration processor 2610 can apply corrections so that the desired light beams are output by the display and can adjust the properties of the light input into the waveguide of the display. In some cases, the light can be input with slightly different colors or positions or orientations and adjusted for display imperfections. For example, one or more of the RGB color values within the input image to be projected by the display are corrected via corresponding RGB calibrations (based on the user's eye position), and the corrected RGB values can be sent to the display for projection. The net effect of the imperfect display projecting the corrected RGB values is, at least in part, to generate a projected image that corrects for the imperfections (spatial and / or color) of the display. In other cases, the actively controlled diffractive optical elements within the waveguide assembly can be adjusted by the dynamic calibration processor so that light beams are projected from the display that at least in part correct for the imperfections within the display. In some implementations, method 2700 is performed in real time as a feedback loop such that when the eye tracking camera 500 monitors the user's eyes and a change in eye position is detected, a new calibration (for the new eye position) is used to calibrate the display. In some cases, a new calibration is applied when the change in eye position exceeds a threshold (e.g., a percentage of the spacing between the grid of calibration positions). Some such implementations can advantageously provide a continuously calibrated display for the user to view.In some implementations, method 2700 may be performed at any time (e.g., when the user positions the display across the user's eyes) or periodically (e.g., to correct for accidental slippage between the display and the user's eyes).

[0198] FIG. 28 is a process flow diagram 2805 schematically illustrating an example of the interaction of a factory calibration system and a dynamic calibration system associated with a particular display. In this example, an eye proxy camera calibration system 2810 is used in a factory (manufacturing) setting to determine position-dependent calibration for the display being manufactured. In block 2820, the process analyzes one or more calibration images for each particular display being manufactured and generates a calibration for each eye proxy position. In block 2830, the calibration is stored in a memory associated with the particular display such that each display has access to the calibration that is customized for that particular display during the manufacturing process. For example, the calibration may be stored as a look-up table (LUT) in the data module 224 of display 208 or in the remote data repository 232. This portion of process flow 2805 may be performed once for each display during manufacturing to provide a calibration customized for each display.

[0199] In this embodiment, each display system (e.g., an embodiment of the wearable display system 200) can perform real-time calibration using the calibration stored in block 2830. For example, the eye tracking system 2840 of the display (which may include the eye tracking camera 500) can determine the position of the cornea of the eye and the direction of the line of sight of the eye, and may also determine the position of the eye. In block 2850, the display system (e.g., via the dynamic calibration processor 2610) may fetch the appropriate calibration from the memory based on the determined eye position. In block 2860, the calibration is applied to the display (e.g., via the dynamic calibration processor 2610) to correct the spatial and / or color errors of the display. In block 2870, the wearer can view the image projected by the calibrated display. As the position of the wearer's eyes relative to the display changes, the process flow within the display system may update the calibration, for example, in real time.

[0200] Although an embodiment of the dynamic calibration system 2600 has been described in the context of a display within a wearable display system, this is not limiting, and a dynamic calibration system (e.g., an eye tracking camera and a dynamic calibration processor) can be used for any display (wearable or non-wearable), and its calibration is only good by approximating the nominal viewing position (e.g., perpendicular to the center of the display). For example, the dynamic calibration system can be used for flat panel displays, liquid crystal displays, light emitting diode displays, microelectromechanical system (MEMS) displays, etc.

[0201] (Additional aspects for performing image correction) In a first aspect, a computer-implemented method for performing image correction on a display is disclosed. The method is under the control of a display calibration system comprising computer hardware and a camera, and includes steps of calibrating the camera, capturing, using the camera, an image of a light field projected by the display, wherein the light field is associated with a display layer of the display, generating a vector field comprising vectors corresponding to deviations between the projected positions and the expected positions of points of the display layer, at least partially based on the captured image, performing at least one of decentering correction, convergence rotation correction, convergence scaling correction, or spatial mapping for the display using the generated vector field, determining a plurality of luminance values corresponding to a plurality of points on the display layer, at least partially based on the captured image, and performing at least one of luminance flattening or color balance for the display using the determined plurality of luminance values.

[0202] In a second aspect, the step of performing decentering correction includes identifying a center point of the projected display layer and determining a translation vector corresponding to a translation error between the identified center point and an expected center point position, according to the computer-implemented method described in aspect 1.

[0203] In a third aspect, the step of performing convergence rotation includes identifying a center point of the projected display layer and determining a rotation amount corresponding to a rotation of the projected display layer about the center point such that a pixel error amount between the projected position and the expected position is minimized, according to the computer-implemented method described in aspect 1 or aspect 2.

[0204] In the fourth aspect, the step of performing collective scaling includes the step of identifying the center point of the projected display layer and the step of determining the scaling amount, where the scaling amount corresponds to the scaling of the projected display layer centered on the center point such that the pixel error amount between the projected position and the expected position is minimized. The computer-implemented method according to any one of aspects 1-3.

[0205] In the fifth aspect, the step of performing spatial mapping includes the step of identifying a non-linear transformation and the step of aligning the projected position and the expected position of the display layer. The computer-implemented method according to any one of aspects 1-4.

[0206] In the sixth aspect, the step of performing brightness flattening includes the step of determining the minimum brightness value of a plurality of brightness values and the step of reducing all the brightness values of the plurality of brightness values to the minimum brightness value. The computer-implemented method according to any one of aspects 1-5.

[0207] In the seventh aspect, the step of performing brightness flattening includes the step of determining a threshold brightness value and the step of reducing all the brightness values of the plurality of brightness values exceeding the threshold brightness value to the threshold brightness value. The computer-implemented method according to any one of aspects 1-5.

[0208] In the eighth aspect, the step of performing color balance includes the step of identifying a color cluster associated with the display layer, where the color cluster includes at least one additional display layer, and for each point among a plurality of points on the display layer, comparing the brightness value corresponding to the point on the display layer with the brightness value corresponding to the point on the additional display layer, and reducing each brightness value of the plurality of brightness values to the minimum brightness value associated with its corresponding point. The computer-implemented method according to any one of aspects 1-7.

[0209] In a ninth aspect, the step of performing collective rotation correction is the computer-implemented method according to any one of aspects 1-8, including the step of calculating the curl of the vector field.

[0210] In a tenth aspect, the step of performing collective scaling correction is the computer-implemented method according to any one of aspects 1-9, including the step of calculating the divergence of the vector field.

[0211] In an eleventh aspect, the display is the computer-implemented method according to any one of aspects 1-10, comprising a light field display.

[0212] In a twelfth aspect, the light field display is the computer-implemented method according to aspect 11, comprising a stacked waveguide assembly.

[0213] In a thirteenth aspect, the stacked waveguide assembly is the computer-implemented method according to aspect 12, each comprising two or more waveguides corresponding to two or more depth planes.

[0214] In a fourteenth aspect, each depth plane is the computer-implemented method according to aspect 13, associated with a red display layer, a green display layer, and a blue display layer.

[0215] In a fifteenth aspect, a method for calibrating a display is disclosed. The method is under the control of a display calibration system comprising computer hardware, and includes accessing an image of a calibration pattern projected by the display, determining a spatial distortion between the expected position of a calibration point in the projected light field and the actual displayed position in the image, analyzing the spatial distortion and determining a spatial calibration for the display, and storing the spatial calibration in a non-transitory memory associated with the display.

[0216] In the 16th aspect, spatial calibration is the method described in aspect 15 that corrects one or more of the in-plane spatial error or the out-of-plane spatial error.

[0217] In the 17th aspect, spatial calibration is the method described in aspect 15 or 16 that corrects one or more of the translational error, rotational error, scaling error, or pixel warping.

[0218] In the 18th aspect, the method further includes the steps of determining color distortion from an image, analyzing the color distortion to determine color calibration for a display, and storing the color calibration in a non-transitory memory associated with the display, as described in any one of aspects 15 - 17.

[0219] In the 19th aspect, color calibration is the method described in aspect 18 that corrects the luminance flatness or color uniformity of a display.

[0220] (Additional aspects of the optical measurement system) In the 20th aspect, an optical measurement system for measuring the imperfection in a light field generated by a display is disclosed. The optical measurement system includes a display configured to project a target light field with a virtual object having an intended focal position, a camera configured to obtain an image of the target light field, access to one or more images corresponding to a portion of the light field, analyze the one or more images, identify a measured focal position corresponding to the position where the virtual object is in focus, and determine the imperfection in the light field using executable instructions programmed based at least in part on a comparison of the measured focal position and the intended focal position, and a processor.

[0221] In the 21st aspect, the display is the optical measurement system described in aspect 20 that includes a light field display.

[0222] In the 22nd aspect, the display outputs light and is configured to project a virtual object onto a specific depth plane, and includes a stack of waveguides, and is the optical measurement system described in aspect 20 or aspect 21.

[0223] In the 23rd aspect, the camera has a small depth of focus and includes a digital camera, and is the optical measurement system described in any one of aspects 20 - 22.

[0224] In the 24th aspect, the camera has a focus, and the system is configured to sweep the focus of the camera over a range of foci to obtain one or more images, and is the optical measurement system described in aspect 23.

[0225] In the 25th aspect, the camera includes a light field camera, and is the optical measurement system described in any one of aspects 20 - 22.

[0226] In the 26th aspect, the virtual object includes a grating pattern, a geometric pattern, or a stochastic pattern, and is the optical measurement system described in any one of aspects 20 - 25.

[0227] In the 27th aspect, the display includes a plurality of pixels, and the target light field corresponds to a subset less than all of the illuminated pixels, and is the optical measurement system described in any one of aspects 20 - 26.

[0228] In the 28th aspect, the measured focus position includes a depth of focus, and is the optical measurement system described in any one of aspects 20 - 27.

[0229] In the 29th aspect, the measured focus position further includes a lateral focus position, and is the optical measurement system described in aspect 28.

[0230] In the 30th aspect, the determined imperfection is at least partially based on an error vector between the intended focus position and the measured focus position, and is the optical measurement system described in aspect 29.

[0231] In a 31st aspect, the determined imperfection is an optical measurement system according to any one of aspects 20 - 30, having a spatial imperfection.

[0232] In a 32nd aspect, the determined imperfection is an optical measurement system according to any one of aspects 20 - 31, having a color imperfection.

[0233] In a 33rd aspect, the processor is further programmed to determine error correction for the display, at least in part, based on the determined imperfection, in an optical measurement system according to any one of aspects 20 - 32.

[0234] In a 34th aspect, a method for measuring imperfection in a light field is disclosed, the method including accessing one or more images corresponding to a portion of the light field projected by a display, the portion of the light field having an intended focal position; analyzing the one or more images to identify a measured focal position corresponding to a position where the portion of the light field is in focus; and determining the imperfection in the light field, at least in part, based on a comparison of the measured focal position and the intended focal position.

[0235] In a 35th aspect, the method according to aspect 34, including sweeping the focus of a camera to obtain one or more images.

[0236] In a 36th aspect, the method according to aspect 34, including using a light field camera to obtain one or more images.

[0237] In a 37th aspect, the method according to any one of aspects 34 - 36, further including projecting a light field image having a moiré pattern.

[0238] In a 38th aspect, the method according to any one of aspects 34 - 37 further includes, at least partially, a step of determining error correction for a light field based on a determined imperfection degree.

[0239] (Additional aspects of display calibration) In a 39th aspect, a calibration system for a display is provided. The calibration system includes a camera configured to acquire an image of the display, and a hardware processor communicating with the camera, the hardware processor being programmed to receive the image of the display, determine calibration for the display, and store the calibration in a memory associated with the display.

[0240] In a 40th aspect, the calibration is the calibration system according to aspect 39, which includes spatial calibration for correcting spatial imperfection in the display.

[0241] In a 41st aspect, the calibration is the calibration system according to aspect 39, which includes color calibration for correcting color imperfection in the display.

[0242] In a 42nd aspect, the display includes a plurality of pixels in a field of view. For determining the calibration, the hardware processor is programmed to determine global transformation parameters independent of the pixels in the field of view of the display and determine local transformation parameters dependent on the pixels in the field of view of the display, according to any one of aspects 39 - 41.

[0243] In a 43rd aspect, the global transformation parameters include one or more non - linear gamma corrections, according to the calibration system of aspect 42.

[0244] In a 44th aspect, the local transformation includes a linear function, according to the calibration system of aspect 42 or aspect 43.

[0245] In the 45th aspect, for calibrating, the hardware processor is programmed to iteratively obtain calibration using feedback from an image acquired by a camera, for the calibration system according to any one of aspects 39 - 44.

[0246] In the 46th aspect, the calibration includes color calibration, the display has multiple color levels that can provide a white point, and for calibrating, the hardware processor is programmed to adjust the intensity of the color levels such that the white point is substantially uniform across the field of view of the display, for the calibration system according to any one of aspects 39 - 45.

[0247] In the 47th aspect, for calibrating, the hardware processor obtains a first gamma correction that maps the color levels sent to the display to a first intermediate color representation, obtains a pixel - dependent combination function that maps the first intermediate color representation to a second intermediate color representation, and obtains a second gamma correction that maps the second intermediate color representation to the color levels aligned by the camera, for the calibration system according to aspect 46.

[0248] In the 48th aspect, the hardware processor is programmed to obtain the first gamma correction and the second gamma correction prior to obtaining the pixel - dependent combination function, for the calibration system according to aspect 47.

[0249] In the 49th aspect, the display includes a light - field display, for the calibration system according to any one of aspects 39 - 48.

[0250] In the 50th aspect, the display includes a stackable waveguide assembly with multiple waveguides, for the calibration system according to any one of aspects 39 - 49.

[0251] In the 51st aspect, the display is configured for a wearable display system, for the calibration system according to any one of aspects 39 - 50.

[0252] In a 52nd aspect, a method for calibrating a display is provided. The method is under the control of a dynamic calibration system performed by computer hardware, and includes steps of accessing calibration for the display, and determining, at least in part based on the accessed calibration, a correction for application to the display to at least in part correct imperfections within the display, and applying the correction to the display.

[0253] In a 53rd aspect, the step of accessing calibration is the method described in aspect 52, including color calibration.

[0254] In a 54th aspect, the display comprises a plurality of pixels within the field of view, and the color calibration is the method described in aspect 53, comprising a plurality of pixel-independent non-linear gamma corrections and a pixel-dependent coupling function.

[0255] In a 55th aspect, the display comprises a light field display, and the method is as described in any one of aspects 52 - 54.

[0256] In a 56th aspect, a head-mounted display is provided, comprising a display, a memory configured to store calibration, and a hardware processor in communication with the non-transitory memory and programmed to perform the method as described in any one of aspects 14 - 17.

[0257] (Additional aspects of calibration patterns) In a 57th aspect, an optical system for calibrating a light field generated by a display, the optical system comprising a calibration pattern containing feature points, configured to project a target light field, a display, a camera configured to obtain an image of the target light field, and for each of a plurality of positions, causing the calibration pattern to be projected onto a position within the plurality of positions on the display, causing the camera to obtain an image of the projected calibration pattern, calculating a distortion of the feature points, the distortion corresponding to an error between an expected location of the feature points and a measured location of the feature points or an error between an expected luminance or color of the calibration pattern and a measured luminance or color of the calibration pattern, and programmed with executable instructions to shift a calibration pattern to be displayed to the next position in response to determination of the next position within the plurality of positions, and a processor.

[0258] In a 58th aspect, the calibration pattern comprises a grid pattern, the optical system according to aspect 57.

[0259] In a 59th aspect, the number of the plurality of positions corresponds to the number of pixels within the grid of the grid pattern, the optical system according to aspect 57.

[0260] In a 60th aspect, the calibration pattern comprises a single pixel pattern, the optical system according to aspect 57.

[0261] In a 61st aspect, the number of the plurality of positions corresponds to the number of pixels to be displayed, the optical system according to aspect 60.

[0262] In a 62nd aspect, the processor is further programmed to generate a distortion map, at least in part, based on the calculated distortions corresponding to the plurality of positions, the optical system according to any one of aspects 57 - 61.

[0263] In a 63rd aspect, the processor is further programmed, at least in part, to determine error correction for a display based on computed distortions corresponding to a plurality of positions, for the optical system according to any one of aspects 57 - 62.

[0264] In a 64th aspect, the display comprises separate red, green, and blue layers, for the optical system according to any one of aspects 57 - 63.

[0265] In a 65th aspect, the display comprises a light field display, for the optical system according to any one of aspects 57 - 64.

[0266] In a 66th aspect, the light field display comprises a stacked waveguide assembly, for the optical system according to aspect 65.

[0267] In a 67th aspect, the stacked waveguide assembly each comprises two or more waveguides corresponding to two or more depth planes, for the optical system according to aspect 66.

[0268] In a 68th aspect, the computed distortion further comprises luminance distortion or color distortion, for the optical system according to any one of aspects 57 - 67.

[0269] In a 69th aspect, a method for calibrating a light field generated by a display is provided. The method includes projecting a calibration pattern to positions within a plurality of positions on the display for each of the plurality of positions, obtaining an image of the projected calibration pattern by a camera, calculating a distortion of feature points, wherein the distortion corresponds to an error between an expected location of the feature points and a measured location of the feature points or an error between an expected luminance or color of the feature points and a measured luminance or color of the feature points, and shifting the calibration pattern to be displayed to a next position in response to determination of a next position within the plurality of positions.

[0270] In the 70th aspect, the calibration pattern is the method described in aspect 69, which is a grid pattern.

[0271] In the 71st aspect, the number of multiple positions corresponds to the number of pixels within the grid of the grid pattern, which is the method described in aspect 70.

[0272] In the 72nd aspect, the calibration pattern comprises a single pixel pattern, a probabilistic pattern, or a geometric pattern, which is the method described in aspect 69.

[0273] In the 73rd aspect, the number of multiple positions corresponds to the number of pixels to be displayed, which is the method described in aspect 72.

[0274] In the 74th aspect, the method according to any one of aspects 69 - 73 further includes the step of generating a distortion map based at least in part on the calculated distortion corresponding to multiple positions.

[0275] In the 75th aspect, the method according to any one of aspects 69 - 74 further includes the step of determining error correction for the display based at least in part on the calculated distortion corresponding to multiple positions.

[0276] In the 76th aspect, the display is the optical system according to any one of aspects 69 - 75, which comprises separate red, green, and blue layers.

[0277] In the 77th aspect, the display is the optical system according to any one of aspects 69 - 76, which comprises a light field display.

[0278] In the 78th aspect, the light field display is the optical system according to aspect 77, which comprises a stacked waveguide assembly.

[0279] In the 79th aspect, the stacked waveguide assembly comprises two or more waveguides respectively corresponding to two or more depth planes, which is the optical system described in aspect 78.

[0280] In the 80th aspect, the calculated distortion further includes a luminance distortion or a color distortion, and is the optical system according to any one of aspects 69-79.

[0281] (Additional aspects of performing dynamic calibration) In the 81st aspect, a display system is provided. The display system includes an eye-tracking camera, a display, and a non-transitory data storage device configured to store a plurality of calibrations for the display, wherein each calibration in the plurality of calibrations is associated with a calibration position for the display, and a hardware processor that communicates with the eye-tracking camera, the display, and the non-transitory data storage device, the hardware processor being configured to determine an eye position with respect to the display for a user of the display, access one or more than one of the plurality of calibrations at least partially based on the determined eye position, determine a correction for application to the display to correct an imperfection in the display at least partially based on one or more than one of the plurality of calibrations, and be programmed to apply the correction to the display.

[0282] In the 82nd aspect, the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more, and is the display system according to aspect 81.

[0283] In the 83rd aspect, the calibration positions are distributed across the display within a grid, and is the display system according to aspect 81 or aspect 82.

[0284] In the 84th aspect, the grid includes a 2×2, 3×3, 5×5, or 9×9 grid, and is the display system according to aspect 83.

[0285] In the 85th aspect, a display system according to any one of aspects 81 - 84, wherein one or more than one of a plurality of calibrations comprises a calibration associated with the calibration position closest to the eye position.

[0286] In the 86th aspect, a display system according to any one of aspects 81 - 85, wherein, to determine a correction, a hardware processor is programmed to interpolate or extrapolate between one or more than one of a plurality of calibrations.

[0287] In the 87th aspect, a display system according to any one of aspects 81 - 86, wherein each calibration in a plurality of calibrations corrects a spatial imperfection of a display, a color imperfection of the display, or both the spatial imperfection and the color imperfection.

[0288] In the 88th aspect, a display system according to any one of aspects 81 - 87, wherein the display comprises a light field display.

[0289] In the 89th aspect, a display system according to any one of aspects 81 - 88, wherein the display comprises a stackable waveguide assembly comprising a plurality of waveguides.

[0290] In the 90th aspect, a display system according to any one of aspects 81 - 89, wherein the display is configured as a wearable display system.

[0291] In the 91st aspect, a head-mounted display comprising a display system according to any one of aspects 81 - 90 is provided.

[0292] In a 92nd aspect, a method for calibrating a display is provided. The method is under the control of a dynamic calibration system performed by computer hardware and includes steps of determining an eye position of a user of the display, accessing a calibration for the display at least partially based on the determined eye position, wherein the calibration is associated with a calibration position in the vicinity of the determined eye position, determining a correction for application to the display to at least partially correct an imperfection within the display at least partially based on the accessed calibration, and applying the correction to the display.

[0293] In a 93rd aspect, the step of accessing a calibration includes the step of selecting one or more calibrations from a plurality of calibrations, each calibration being associated with a different calibration position for the display, the method according to aspect 92.

[0294] In a 94th aspect, the calibration positions are arranged within a grid across the display, the method according to aspect 93.

[0295] In a 95th aspect, the calibration corrects a spatial imperfection of the display, a color imperfection of the display, or both the spatial imperfection and the color imperfection, any one of aspects 92 - 94. The method according to any one of the preceding claims.

[0296] In a 96th aspect, the step of determining a correction includes the step of interpolating or extrapolating between one or more calibrations associated with a calibration position in the vicinity of the eye pose, any one of aspects 92 - 95.

[0297] In a 97th aspect, the display comprises a light field display, any one of aspects 92 - 96.

[0298] On the 98th aspect, a head-mounted display is provided, which includes an eye-tracking system and a hardware processor programmed to perform the method according to any one of aspects 92-97.

[0299] (Additional aspects of the optical measurement system) On the 99th aspect, an optical measurement system is provided for measuring the imperfection in the light field generated by the display. The optical measurement system includes a display configured to project a target light field having a virtual object with an intended focal position, a camera configured to obtain an image of the target light field, access to one or more images corresponding to a part of the light field, analyze the one or more images, identify the measured focal position corresponding to the position where the virtual object is in focus, and determine the imperfection in the light field using executable instructions, at least partially based on a comparison between the measured focal position and the intended focal position, and a hardware processor programmed to do so.

[0300] On the 100th aspect, the display of the optical measurement system according to aspect 99 includes a stack of waveguides configured to output light and project a virtual object onto at least one depth plane.

[0301] On the 101st aspect, the camera of the optical measurement system according to any one of aspects 99-100 includes a digital camera having a small depth of focus.

[0302] On the 102nd aspect, the camera of the optical measurement system according to aspect 101 has a focus, and the system is configured to sweep the focus of the camera over a range of foci to obtain one or more images.

[0303] On the 103rd aspect, the camera of the optical measurement system according to any one of aspects 99-102 includes a light field camera.

[0304] In the 104th aspect, the virtual object is the optical measurement system according to any one of aspects 99 - 103, comprising a lattice pattern, a geometric pattern, or a stochastic pattern.

[0305] In the 105th aspect, the display comprises a plurality of pixels, and the target light field corresponds to a subset less than all of the illuminated pixels, which is the optical measurement system according to any one of aspects 99 - 104.

[0306] In the 106th aspect, the measured focal position includes the depth of focus, which is the optical measurement system according to any one of aspects 99 - 105.

[0307] In the 107th aspect, the measured focal position further includes a lateral focal position, which is the optical measurement system according to aspect 106.

[0308] In the 108th aspect, the determined imperfection is at least partially based on the error vector between the intended focal position and the measured focal position, which is the optical measurement system according to any one of aspects 99 - 107.

[0309] In the 109th aspect, the hardware processor is further programmed to determine error correction for the display at least partially based on the determined imperfection, which is the optical measurement system according to any one of aspects 99 - 108.

[0310] In the 110th aspect, the hardware processor is further programmed to apply a display / camera pixel mapping to convert the pixel values of the display into the pixel values of the camera, which is the optical measurement system according to any one of aspects 99 - 109.

[0311] In the 111th aspect, the display / camera pixel mapping includes a first gamma correction that maps the color levels of the display to a first intermediate color representation, a pixel-dependent binding function that maps the first intermediate color representation to a second intermediate color representation, and a second gamma correction that maps the second intermediate color representation to the color levels aligned by the camera, for the optical measurement system described in aspect 110.

[0312] In the 112th aspect, the determined imperfection includes a spatial imperfection, for the optical measurement system described in any of aspects 99 - 111.

[0313] In the 113th aspect, the spatial imperfection includes one or more of in-plane translation, rotation, scaling, or warping errors, or out-of-plane or depth-of-field errors, for the optical measurement system described in aspect 112.

[0314] In the 114th aspect, the determined imperfection includes a color imperfection, for the optical measurement system described in any of aspects 99 - 113.

[0315] In the 115th aspect, the color imperfection includes one or more of luminance flatness or color uniformity errors associated with the colors that can be displayed by the display, for the optical measurement system described in aspect 114.

[0316] On the 116th aspect, an optical measurement system for performing image correction on a display is provided. The system includes a camera configured to capture an image of a light field projected by the display, where the light field is associated with a display layer of the display, and at least partially based on the image captured by the camera, a vector field is generated, the vector field comprising vectors corresponding to the deviation between the projected position and the expected position of points on the display layer, and at least partially based on the vector field, at least one of decentering correction, convergence rotation correction, convergence scaling correction, or spatial mapping is calculated for the display, and at least partially based on the image captured by the camera, luminance values corresponding to a plurality of points on the display layer are calculated, and at least partially based on the determined luminance values, executable instructions are used to program a hardware processor to calculate luminance flattening correction or color balance correction for the display.

[0317] On the 117th aspect, the display layer of the display comprises the optical measurement system described in aspect 116, which includes a color layer or a depth layer.

[0318] On the 118th aspect, the camera comprises a light field camera or a digital camera having a small depth of focus, for the optical measurement system described in any of aspects 116 - 117.

[0319] On the 119th aspect, for calculating decentering correction, the hardware processor is programmed to determine a translation vector corresponding to the translation error between the identified center point of the projected display layer and the expected center point position, for the optical measurement system described in any of aspects 116 - 118.

[0320] On the 120th side, in order to calculate the convergence rotation correction, the hardware processor is programmed to determine the amount of rotation corresponding to the rotation of the projected display layer centered on the center point so that the amount of pixel error between the projected position and the expected position is reduced or minimized, for the optical measurement system described in any of sides 116 - 119.

[0321] On the 121st side, in order to calculate the convergence rotation correction, the hardware processor is programmed to calculate the curl of the vector field, for the optical measurement system described in any of sides 116 - 120.

[0322] On the 122nd side, in order to calculate the convergence scaling correction, the hardware processor is programmed to determine the amount of scaling corresponding to the scaling of the projected display layer centered on the center point so that the amount of pixel error between the projected position and the expected position is reduced or minimized, for the optical measurement system described in any of sides 116 - 121.

[0323] On the 123rd side, in order to calculate the convergence scaling correction, the hardware processor is programmed to calculate the divergence of the vector field, for the optical measurement system described in any of sides 116 - 122.

[0324] On the 124th side, in order to calculate the spatial mapping, the hardware processor is programmed to determine the non - linear transformation for aligning the projected position and the expected position of the display layer, for the optical measurement system described in any of sides 116 - 123.

[0325] On the 125th side, in order to calculate the luminance flattening correction, the hardware processor is programmed to determine the threshold luminance value and calculate the amount by which each luminance value above the threshold luminance value is reduced to the threshold luminance value, for the optical measurement system described in any of sides 116 - 124.

[0326] On the 126th aspect, in order to calculate color balance correction, the hardware processor identifies color clusters associated with the display layer, where the color clusters constitute at least one additional display layer, and for each point of the display layer, compares the luminance value corresponding to the point on the display layer with the luminance value corresponding to the point on the additional display layer, and is programmed to calculate the amount by which each luminance value is reduced to the lowest luminance value associated with its corresponding point, the optical measurement system according to any of aspects 116 - 125.

[0327] (Additional aspect of dynamic display calibration) On the 127th aspect, a display system is provided. The display system includes an eye-tracking camera, a display, and a non-transitory data storage device configured to store a plurality of calibrations for the display, where each calibration in the plurality of calibrations is associated with a calibration position for the display, a non-transitory data storage device, and a hardware processor that communicates with the eye-tracking camera, the display, and the non-transitory data storage device, and based on information from the eye-tracking camera, determines the eye position of the user of the display relative to the display, accesses one or more than one of the plurality of calibrations at least partially based on the determined eye position, calculates a correction for application to the display to correct at least partially an imperfection within the display at least partially based on one or more than one of the plurality of calibrations, and is programmed to apply the correction to the display.

[0328] On the 128th aspect, the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more than that, the display system according to aspect 127.

[0329] On the 129th aspect, the calibration positions are distributed across the display within a grid, the display system according to any of aspects 127 - 128.

[0330] On the 130th aspect, the grid is the display system described in aspect 129, which includes a 2×2, 3×3, 5×5, or 9×9 grid.

[0331] On the 131st aspect, one or more than one of the plurality of calibrations is the display system described in any of aspects 127 - 130, which includes a calibration associated with the calibration position closest to the eye position.

[0332] On the 132nd aspect, in order to calculate the correction, the hardware processor is programmed to interpolate or extrapolate between one or more than one of the plurality of calibrations based at least in part on the calibration positions and the determined eye positions of one or more than one of the plurality of calibrations, for the display system described in any of aspects 127 - 131.

[0333] On the 133rd aspect, the display includes a first display associated with the user's first eye and a second display associated with the user's second eye, and the hardware processor is programmed to determine the user's eye position for the first display and apply the determined eye position to calculate the correction for the second display, for the display system described in any of aspects 127 - 132.

[0334] On the 134th aspect, the display includes a first display associated with the user's first eye and a second display associated with the user's second eye, and at least some of the plurality of calibrations represent an average calibration for the first display and the second display, for the display system described in any of aspects 127 - 133.

[0335] On the 135th aspect, the display includes a light field display, for the display system described in any of aspects 127 - 134.

[0336] On the 136th aspect, the display comprises a display system according to any one of aspects 127 - 135, comprising a stackable waveguide assembly with a plurality of waveguides.

[0337] On the 137th aspect, the display comprises a display system according to any one of aspects 127 - 136, configured as a head - mounted wearable display system.

[0338] On the 138th aspect, each calibration in a plurality of calibrations corrects for spatial non - perfection of the display, color non - perfection of the display, or both spatial and color non - perfection, according to any one of claims 127 - 137 of the display system.

[0339] On the 139th aspect, the spatial non - perfection comprises one or more of in - plane translation, rotation, scaling, or warping errors, or out - of - plane or depth - of - focus errors, of the display system according to aspect 138.

[0340] On the 140th aspect, the color non - perfection comprises one or more of luminance flatness or color uniformity errors associated with the colors that can be displayed by the display, of the display system according to aspect 138.

[0341] On the 141st aspect, a method for calibrating a display is provided. The method is under the control of a dynamic calibration system performed by computer hardware, and includes the steps of determining the eye position of the user of the display, accessing a calibration for the display at least in part based on the determined eye position, where the calibration is selected based on the associated calibration position and the determined eye position, calculating a correction for application to the display to correct for non - perfection in the display at least in part based on the accessed calibration, and applying the correction to the display.

[0342] In the 142nd aspect, the step of accessing calibration includes the step of selecting one or more calibrations from a plurality of calibrations, and each calibration is associated with a different calibration position for the display, as described in aspect 141.

[0343] In the 143rd aspect, the calibration positions are arranged within a grid across the display, as described in aspect 142.

[0344] In the 144th aspect, the step of calculating a correction includes the step of interpolating or extrapolating between one or more of the plurality of calibrations based on the associated calibration positions and the determined eye positions of one or more of the plurality of calibrations, as described in any of aspects 142 - 143.

[0345] In the 145th aspect, the method further includes the step of accessing an image of the user's eye on the display and determining the eye position at least partially based on the image of the eye, as described in any of aspects 141 - 144.

[0346] In the 146th aspect, the step of calculating a calibration includes the step of calibrating the spatial non - uniformity of the display, the color non - uniformity of the display, or both the spatial and color non - uniformities, as described in any one of aspects 141 - 145.

[0347] In a 147th aspect, there is provided a wearable display system including an imaging system facing inward, a display, and a non-transitory data storage device configured to store a plurality of calibrations for the display, wherein each calibration in the plurality of calibrations is associated with a calibration position for the display, a non-transitory data storage device, and a hardware processor that communicates with the imaging system facing inward, the display, and the non-transitory data storage device, the hardware processor determining an eye position of a user of the display with respect to the display using the imaging system facing inward, and calculating a correction for application to the display to at least partially correct at least one or more of spatial non-uniformity or color non-uniformity within the display based at least in part on the determined eye position and one or more of the plurality of calibrations, and being programmed to apply the correction to the display.

[0348] In a 148th aspect, there is provided the wearable display system according to aspect 147, wherein the hardware processor is programmed to apply the correction via a feedback loop that monitors a change in the eye position.

[0349] In a 149th aspect, there is provided the wearable display system according to any one of aspects 147-148, wherein the hardware processor is programmed to determine a change in the eye position with respect to a previous eye position and, if the change exceeds a threshold, calculate the correction.

[0350] In a 150th aspect, there is provided the wearable display system according to any one of aspects 147-149, wherein the spatial non-uniformity includes one or more of in-plane translation, rotation, scaling, or warping error, or out-of-plane or depth-of-focus error.

[0351] On the 151st side, the color imperfection comprises one or more of the luminance flatness or color uniformity errors associated with the colors that can be displayed by the display, and the wearable display system according to any one of aspects 147-150.

[0352] (Conclusion) The processes, methods, and algorithms described herein and / or depicted in the accompanying figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being fully or partially automated. For example, a computing system can include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc. programmed with specific computer instructions. The code modules can be installed in a dynamic link library that is compiled and linked into an executable program, or can be written in an interpreted programming language. In some implementations, certain operations and methods can be performed by circuits specific to a given function.

[0353] Furthermore, the functional implementations of the present disclosure are sufficiently mathematically, computationally, or technically complex that a special-purpose hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results in substantially real time. For example, a video can include many frames, each frame can have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0354] A code module or any type of data can be stored on any type of non-transitory computer-readable medium such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. The methods and modules (or data) can also be transmitted as data signals generated on various computer-readable transmission media including wireless-based and wired / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal), and can take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0355] Any process, block, state, step, or functionality in a flow diagram described herein and / or depicted in the accompanying figures is to be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems or code modules can implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in other suitable sequences, e.g., sequentially, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the implementations described herein is for purposes of illustration and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

[0356] The present process, method, and system can be implemented in a network (or distributed) computing environment. The network environment can include an enterprise-wide computer network, an intranet, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), a cloud computing network, a cloud source computing network, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0357] The systems and methods of the present disclosure each have several innovative aspects, none of which alone contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the disclosure, principles, and novel features disclosed herein.

[0358] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, the various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable sub-combination. Further, a feature may be described above as acting in a certain combination and may further be claimed as such, but one or more features from the claimed combination can in some cases be deleted from the combination, and the claimed combination can be directed to a sub-combination or variation of a sub-combination. No single feature or group of features is necessary or essential in every embodiment.

[0359] In particular, conditional statements used herein such as "can", "could", "might", "may", "e.g.", and equivalents, generally convey that while one embodiment includes certain features, elements, and / or steps, other embodiments do not, unless specifically stated otherwise or understood otherwise within the context in which they are used. Thus, such conditional statements are not generally intended to imply that the features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, regardless of the author's input or prompting. The terms "comprising", "including", "having", and equivalents are synonyms and are used inclusively in a non-limiting manner, without excluding additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a", "an", and "the" as used in this application and the appended claims should be construed to mean "one or more than one" or "at least one" unless otherwise defined.

[0360] As used herein, the phrase referring to a list of items "at least one of" refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as "at least one of X, Y, and Z" are generally understood in a context such that, unless specifically stated otherwise, they are used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such connective phrases are generally not intended to suggest that an embodiment requires the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0361] Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that such operations need not be performed in the particular order shown, or in a sequential order, or that all illustrated operations be performed, to achieve a desired result. Further, the drawings may schematically depict one or more exemplary processes in the form of flowcharts. However, other operations not depicted may also be incorporated into the exemplary methods and processes schematically illustrated. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the illustrated operations. Additionally, operations may be rearranged or re-ordered in other implementations. In some situations, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the program components and systems described generally may be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve a desired result.

Claims

1. A head mounted display system, comprising: The head mounted display system comprises: a wearable display configured to be positioned in front of an eye of a wearer of the head mounted display system, the wearable display configured to present virtual image content to the eye of the wearer, the wearable display having a plurality of depth planes associated with the wearable display, the wearable display configured to present different virtual images at different depth planes, the wearable display comprising a plurality of waveguides including a first waveguide and a second waveguide configured to output a first and second color, respectively; a non-transitory data storage device configured to store calibration parameters associated with the first and second waveguides of the wearable display, the calibration parameters including color calibration parameters; and a hardware processor in communication with the wearable display and the non-transitory data storage device; Equipped with The hardware processor includes: accessing the color calibration parameters associated with the first and second waveguides of the wearable display; applying a correction to the wearable display based on the color calibration parameters; and A head mounted display system that is programmed to:

2. A head-mounted display system as described in claim 1, wherein the wearable display includes a light field display.

3. A head-mounted display system as described in claim 1, wherein the plurality of waveguides are configured to form an image in the wearer's eye by directing light to the wearer's eye.

4. A head-mounted display system as described in claim 3, wherein different depth planes among the plurality of depth planes are associated with different waveguides among the plurality of waveguides.

5. A head-mounted display system as described in claim 1, wherein the correction is configured to correct color non-uniformity.

6. A head-mounted display system as described in claim 1, wherein the correction is configured to provide color balance.

7. A head-mounted display system as described in claim 1, wherein the correction includes balancing the white point.

8. A head-mounted display system as described in claim 1, wherein the hardware processor is further programmed to correct luminance flattening.

9. A head-mounted display system as described in claim 1, wherein the calibration parameters further include spatial calibration parameters, the spatial calibration parameters being associated with a vector field including vectors corresponding to deviations between expected positions and projected positions of points of at least one of the virtual images in at least one of the multiple depth planes.

10. A head-mounted display system as described in claim 9, wherein the spatial calibration parameter is associated with the curl or divergence of the vector field.

11. A head-mounted display system as described in claim 9, wherein the vector field includes a three-dimensional (3D) vector field corresponding to in-plane and out-of-plane deviations between expected and projected positions of points of the virtual image.

12. A head-mounted display system as described in claim 1, wherein the correction further corrects at least one of in-plane translation, rotation, scaling, warping error, or out-of-plane focal depth error for at least one depth plane of the multiple depth planes of the wearable display.

13. The head-mounted display system of claim 1, wherein the wearable display comprises a first display associated with a first eye of the wearer and a second display associated with a second eye of the wearer, and the calibration parameters include a first calibration parameter for the first display and a second calibration parameter for the second display.

14. The head-mounted display system of claim 1, wherein the color calibration parameters are configured to correct color imperfections of the wearable display, including color mismatch between the first waveguide and the second waveguide.

15. A head-mounted display system as described in claim 1, wherein the color calibration parameters include a luminance adjustment for at least one depth plane of the plurality of depth planes.

16. The head-mounted display system of claim 1, wherein the calibration parameters include a plurality of calibration parameters, different calibration parameters of the plurality of calibration parameters being associated with corresponding calibration positions relative to the wearable display.

17. A head-mounted display system as described in claim 1, wherein the hardware processor is programmed to apply the correction via a feedback loop that monitors changes in eye position.

18. A head-mounted display system as described in claim 1, wherein the hardware processor is programmed to determine a change in eye position relative to a previous eye position and calculate the correction if the change exceeds a threshold.

19. A head-mounted display system as described in claim 1, wherein the calibration parameters further include spatial calibration parameters.

Images