Dynamic display calibration based on eye tracking

Abstract

To provide dynamic display calibration based on preferred eye tracking. [Solution] An embodiment of a light field measurement system for use with a display is disclosed. The light field measurement may capture images of a projected light field and use the captured images to determine focal depths (or lateral focal positions) for various regions of the light field. The determined focal depths (or lateral positions) may then be compared to the intended focal depths (or lateral positions) to quantify imperfections in the display. Based on the measured imperfections, appropriate error corrections may be applied to the light field to correct for the measured imperfections. The display can be an optical display element in a head-mounted display, for example, an optical display element capable of generating multiple depth planes or a light field display.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application is filed in accordance with U.S. Application No. 62 / 250,925, filed Nov. 4, 2015, entitled LIGHT FIELD DISPLAY METROLOGY; U.S. Application No. 62 / 278,779, filed Jan. 14, 2016, entitled LIGHT FIELD ERROR CORRECTION; U.S. Application No. 62 / 250,934, filed Nov. 4, 2015, entitled AUTOMATED CALIBRATION IMAGE PROJECTION AND CAPTURE FOR DISPLAY CALIBRATION; U.S. Application No. 62 / 278,824, filed Jan. 14, 2016, entitled DYNAMIC CALIBRATION OF A DISPLAY BASED ON EYE-TRACKING; and U.S. Application No. 62 / 278,794, filed Jan. 14, 2016, entitled CHROMATIC BALANCING A DISPLAY This application claims the benefit of priority to HAVING VARYING CHROMATICITY ACROSS A FIELD OF VIEW, all of which are incorporated herein by reference in their entireties.

[0002] (Field) The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and more particularly to a metrology system for measuring and calibrating optical properties of the imaging and visualization systems. The present disclosure also relates to dynamic calibration of virtual reality and augmented reality imaging and visualization systems based on eye tracking. [Background technology]

[0003] (background) Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences in which digitally reproduced images or portions thereof are presented to a user in a manner that appears or can be perceived as real. Virtual reality, or "VR", scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, augmented reality, or "AR", scenarios typically involve the presentation of digital or virtual image information as an extension of the visualization of the real world around the user, or mixed reality, or "MR", relates to the merging of real and virtual worlds to generate new environments in which physical and virtual objects coexist and interact in real time. In conclusion, the human visual perception system is highly complex, making it difficult to generate VR, AR, or MR technologies that promote a comfortable and natural-feeling rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0004] (summary) An embodiment of the imaging system includes a projection device for projecting an image toward an eye of a viewer, the image comprising a light field representing light from a virtual object, the virtual object being configured to be projected as if it were located at one or more intended depths of focus, and a light field measurement device for measuring imperfections in the light field. The light field measurement device may be configured to capture one or more images corresponding to a portion of the light field, analyze the one or more captured images, identify one or more perceived depths of focus corresponding to a depth at which the portion of the light field is focused, create a depth map based at least in part on the identified depths of focus, and compare the created depth map to the one or more intended depths of focus. The system can generate calibrations for spatial and / or chromatic imperfections that can be used to dynamically calibrate the wearable display system.

[0005] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this Summary nor the following Detailed Description purport to define or limit the scope of the inventive subject matter. The present specification also provides, for example, the following items: (Item 1) 1. A display system comprising: An eye-tracking camera; A display and a non-transitory data storage device configured to store a plurality of calibrations for the display, each calibration in the plurality of calibrations being associated with a calibration position relative to the display; a hardware processor in communication with the eye tracking camera, the display, and the non-transitory data storage device, determining an eye position of a user of the display relative to the display based on information from the eye tracking camera; accessing one or more of the plurality of calibrations based at least in part on the determined eye position; and calculating a correction to apply to the display to correct for imperfections in the display based at least in part on one or more of the plurality of calibrations; applying said correction to said display; a hardware processor programmed to A display system comprising: (Item 2) 2. The display system of claim 1, wherein the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more. (Item 3) 2. The display system of claim 1, wherein the calibration positions are distributed across the display in a grid. (Item 4) 4. The display system of claim 3, wherein the grid comprises a 2x2, 3x3, 5x5, or 9x9 grid. (Item 5) 2. The display system of claim 1, wherein one or more of the plurality of calibrations comprises a calibration associated with a calibration position closest to the eye position. (Item 6) 2. The display system of claim 1, wherein to calculate the correction, the hardware processor is programmed to interpolate or extrapolate between one or more of the plurality of calibrations based, at least in part, on the calibration position of the one or more of the plurality of calibrations and the determined eye position. (Item 7) 2. The display system of claim 1, wherein the display comprises a first display associated with a first eye of the user and a second display associated with a second eye of the user, and the hardware processor is programmed to determine an eye position of the user relative to the first display and apply the determined eye position to calculate a correction for the second display. (Item 8) 2. The display system of claim 1, wherein the display comprises a first display associated with a first eye of the user and a second display associated with a second eye of the user, and at least some of the plurality of calibrations represent average calibrations for the first display and the second display. (Item 9) Item 1. The display system of item 1, wherein the display comprises a light field display. (Item 10) Item 10. The display system of item 1, wherein the display comprises a stackable waveguide assembly comprising a plurality of waveguides. (Item 11) 2. The display system of item 1, wherein the display is configured as a head-mounted wearable display system. (Item 12) 12. A display system according to any one of items 1-11, wherein each calibration in the plurality of calibrations corrects spatial imperfections of the display, chromatic imperfections of the display, or both the spatial imperfections and the chromatic imperfections. (Item 13) Item 13. The display system of item 12, wherein the spatial imperfections comprise one or more of in-plane translation, rotation, scaling, or warping errors, or out-of-plane or depth of focus errors. (Item 14) Item 13. The display system of item 12, wherein the chromatic imperfections comprise one or more of luminance flatness or chromatic uniformity errors associated with colors displayable by the display. (Item 15) 1. A method for calibrating a display, comprising: Under the control of a dynamic calibration system implemented by computer hardware, determining an eye position of a user of the display; accessing a calibration for the display based at least in part on the determined eye position, the calibration being selected based on an associated calibration position and the determined eye position; calculating, based at least in part on the accessed calibration, a correction to apply to the display to correct for imperfections in the display; applying the correction to the display; A method comprising: (Item 16) Item 16. The method of item 15, wherein the step of accessing a calibration includes selecting one or more calibrations from a plurality of calibrations, each calibration being associated with a different calibration position relative to the display. (Item 17) Item 17. The method of item 16, wherein the calibration positions are arranged in a grid across the display. (Item 18) Item 17. The method of item 16, wherein calculating the correction comprises interpolating or extrapolating between one or more of the plurality of calibrations based on associated calibration positions of the one or more of the plurality of calibrations and the determined eye position. (Item 19) 20. The method of claim 15, further comprising the step of accessing an image of a user's eye on the display and determining the eye position based, at least in part, on the image of the eye. (Item 20) 20. The method according to any one of items 15-19, wherein the step of calculating the calibration comprises the step of calibrating spatial imperfections of the display, chromatic imperfections of the display, or both the spatial imperfections and the chromatic imperfections. (Item 21) A wearable display system, comprising: an inward-facing imaging system; A display and a non-transitory data storage device configured to store a plurality of calibrations for the display, each calibration in the plurality of calibrations being associated with a calibration position relative to the display; a hardware processor in communication with the inwardly facing imaging system, the display, and the non-transitory data storage device, determining an eye position of a user of the display relative to the display using the inwardly facing imaging system; calculating a correction to apply to the display to correct for one or more of spatial imperfections in the display or chromatic imperfections in the display based at least in part on the determined eye position and one or more of the plurality of calibrations; applying said correction to said display; a hardware processor programmed to A wearable display system comprising: (Item 22) 22. The wearable display system of claim 21, wherein the hardware processor is programmed to apply the correction via a feedback loop that monitors changes in the eye position. (Item 23) 22. The wearable display system of claim 21, wherein the hardware processor is programmed to determine a change in the eye position relative to a previous eye position and calculate the correction if the change exceeds a threshold. (Item 24) 24. The wearable display system of any one of claims 21-23, wherein the spatial imperfections comprise one or more of in-plane translation, rotation, scaling, or warping errors, or out-of-plane or depth of focus errors. (Item 25) 24. The wearable display system of any one of claims 21-23, wherein the chromatic imperfections comprise one or more of a luminance flatness or chromatic uniformity error associated with colors displayable by the display. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 depicts an illustration of an augmented reality scenario with a virtual reality object and a real reality object viewed by a person. [Diagram 2] FIG. 2 diagrammatically illustrates an embodiment of a wearable display system. [Diagram 3] FIG. 3 diagrammatically illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. [Figure 4] FIG. 4 diagrammatically illustrates an embodiment of a waveguide stack for outputting image information to a user. [Diagram 5] FIG. 5 shows an exemplary output beam that may be output by a waveguide. [Figure 6] FIG. 6 is a schematic diagram showing an optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem for use in generating a multifocal volumetric display, image, or light field. [Figure 7] FIG. 7 illustrates example distortions that may occur when projecting a calibration pattern. [Figure 8] FIG. 8 illustrates another example vector field that may be generated to visualize the distortion between the displayed imaging location and the expected image location. [Figure 9A] FIG. 9A illustrates an example XY translation spatial error. [Figure 9B] FIG. 9B illustrates an example aggregate rotational spatial error. [Figure 9C] 9C and 9D illustrate an example of ensemble scaling spatial error. [Figure 9D] 9C and 9D illustrate an example of ensemble scaling spatial error. [Figure 9E] FIG. 9E illustrates an example of the remaining spatial error after XY translation, rotation, and scaling corrections have been made. [Figure 10A] FIG. 10A illustrates an example of multiple depth planes that are intended to be viewed at different depths. [Figure 10B] 10B-10E illustrate example out-of-plane spatial errors that may occur when viewing a projected depth plane. [Figure 10C] 10B-10E illustrate example out-of-plane spatial errors that may occur when viewing a projected depth plane. [Figure 10D] 10B-10E illustrate example out-of-plane spatial errors that may occur when viewing a projected depth plane. [Figure 10E] 10B-10E illustrate example out-of-plane spatial errors that may occur when viewing a projected depth plane. [Figure 11] FIG. 11 illustrates a captured image of the projected test image. [Figure 12A] FIG. 12A illustrates an intensity histogram that may be generated from a captured image of a projected test image. [Figure 12B] FIG. 12B illustrates an intensity profile generated from a captured image of the projected test image. [Figure 13]FIG. 13 illustrates an exemplary intensity histogram illustrating the difference between the mode, median, and mean. [Figure 14A] FIG. 14A illustrates a red-green-blue (RGB) intensity map generated from a captured image of a projected test image. [Figure 14B] FIG. 14B illustrates a plot mapping an example of maximum color imbalance error. [Figure 15] FIG. 15 illustrates an RGB intensity map for an example display with red, green, and blue color layers after chromatic correction. [Figure 16] FIG. 16 is a flow chart of an embodiment of a process for providing image correction to a display system. [Figure 17] 17A and 17B illustrate examples of objects viewed using a normal light field and an incomplete light field. [Figure 18] FIG. 18 diagrammatically illustrates an example of a light field metrology system for measuring the depth of focus of a projected light field, in accordance with some embodiments. [Figure 19A] FIG. 19A is a schematic diagram of an example of an image that may be captured by a camera focused on a particular focal depth. [Figure 19B] 19B and 19C illustrate examples of a depth graph and a depth map. [Figure 19C] 19B and 19C illustrate examples of a depth graph and a depth map. [Figure 20] FIG. 20 is a flow chart of an embodiment of a process for measuring the quality of a virtual target pattern produced by a light field display. [Figure 21] FIG. 21 is a flow chart illustrating an embodiment of a method for calibrating a display. [Figure 22] FIG. 22 diagrammatically illustrates an embodiment of a system that uses a calibration pattern to calibrate a display. [Figure 23A]FIG. 23A illustrates an exemplary checkerboard calibration pattern. [Figure 23B] FIG. 23B illustrates an exemplary single pixel calibration pattern. [Figure 24] FIG. 24 is a flowchart of an exemplary process for calibrating a projected light field. [Figure 25A] FIG. 25A is a top view diagrammatically illustrating an example of a display including a waveguide, an in-coupling optical element, a light redistribution element, and an out-coupling optical element. [Figure 25B] FIG. 25B is a cross-sectional view of the display depicted in FIG. 7A along axis AA'. [Figure 26] FIG. 26 diagrammatically illustrates an example of a dynamic calibration system for a display, in which calibration can be applied to correct spatial and / or chromatic errors in a grid of reference positions (indicated by dots). [Figure 27] FIG. 27 is a flow chart illustrating an exemplary method for dynamically calibrating a display based on eye tracking. [Figure 28] FIG. 28 is a process flow diagram that diagrammatically illustrates an example of the interaction of a factory calibration system and a dynamic calibration system associated with a particular display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure.

[0008] (overview) For a three-dimensional (3D) display to provide a true sensation of depth, and more specifically, a simulated sensation of surface depth, it is desirable to generate, for each point in the field of view of the display, an accommodation response that corresponds to its virtual depth. If the accommodation in response to a display point does not correspond to the virtual depth of that point as determined by the binocular depth cues of convergence and stereopsis, the human eye may suffer from accommodation conflicts, resulting in unstable imaging, deleterious eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0009] VR and AR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing a slightly different presentation of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus different image features of a scene located on different depth planes and / or based on observing different image features on different depth planes out of focus. As discussed elsewhere herein, such depth cues provide a realistic perception of depth.

[0010] (3D display) Figure 1 depicts an illustration of an augmented reality scenario with certain virtual reality objects and certain actual reality objects viewed by a person. Figure 1 depicts an augmented reality scene 100 in which a user of the AR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the AR technology also perceives that they "see" a robotic figure 130 standing on the real-world platform 120 and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0011] In order for a three-dimensional (3D) display to produce a true depth sensation, and more specifically a simulated sensation of surface depth, it is desirable to generate, for each point in the field of view of the display, an accommodation response that corresponds to its virtual depth. If the accommodation response to a display point does not correspond to the virtual depth of that point as determined by the binocular depth cues of convergence and stereopsis, the human eye may experience accommodation conflicts, resulting in unstable imaging, deleterious eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0012] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different depth planes and / or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0013] FIG. 2 illustrates an example of a wearable display system 200 that may be used to present a VR, AR, or MR experience to a display system wearer or viewer 204. The display system 200 includes a display 208 and various mechanical and electronic modules and systems to support the functionality of the display 208. The display 208 may be coupled to a frame 212 that is wearable by a display system user, wearer, or viewer 204 and configured to position the display 208 in front of the eye of the wearer 204. The display 208 may be a light field display. In some embodiments, a speaker 216 is coupled to the frame 212 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). The display 208 is operably coupled 220 to a local data processing module 224, which may be mounted in a variety of configurations, such as fixedly attached to the frame 212, by wired leads or a wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 204 (e.g., in a backpack configuration, in a belt-connected configuration).

[0014] The local processing and data module 224 may comprise a hardware processor and non-transitory digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in the processing, caching, and storage of data. The data may include (a) data captured from sensors (e.g., which may be operatively coupled to the frame 212 or otherwise attached to the user 204), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes, and / or (b) data acquired and / or processed using the remote processing module 228 and / or remote data repository 232, possibly for passing through the display 208 after such processing or reading. The local processing and data module 224 may be operatively coupled to the remote processing module 228 and the remote data repository 232 by communications links 236 and / or 240, such as via wired or wireless communication links, such that these remote modules 228, 232 are available as resources to the local processing and data module 224. In addition, the remote processing module 228 and the remote data repository 232 may be operatively coupled to each other.

[0015] In some embodiments, the remote processing module 228 may comprise one or more processors configured to analyze and process data and / or image information, such as video information captured by an image capture device. The video data may be stored locally within the local processing and data module 224 and / or the remote data repository 232. In some embodiments, the remote data repository 232 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 224, allowing for fully autonomous use from the remote modules.

[0016] The human visual system is complex and difficult to provide a realistic perception of depth. Without being limited by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. Vergence of the two eyes relative to one another (i.e., the rotation of the pupils toward or away from one another to converge and fixate the gaze of the eyes on an object) is closely coupled to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, changing the focus of the eye's lenses, or accommodating the eyes and changing focus from one object to another at a different distance, will automatically produce a corresponding change in vergence at the same distance, under a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a corresponding change in accommodation under normal conditions. Display systems that provide a better match between accommodation and convergence-divergence may produce a more realistic or comfortable simulation of three-dimensional images.

[0017] FIG. 3 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 3, objects at various distances from the eyes 302 and 304 on the z-axis are accommodated by the eyes 302 and 304 such that the objects are in focus. The eyes 302 and 304 assume particular accommodated states that focus objects at different distances along the z-axis. As a result, a particular accommodated state may be said to be associated with a particular one of the depth planes 306 with an associated focal length such that an object or part of an object at a particular depth plane is in focus when the eye is in an accommodated state relative to that depth plane. In some embodiments, a three-dimensional image may be simulated by providing different presentations of images for each eye 302 and 304, and by providing different presentations of images corresponding to each of the depth planes. Although shown as separate for clarity of illustration, it should be understood that the fields of view of the eyes 302 and 304 may overlap, for example, as the distance along the z-axis increases. In addition, while shown to be flat for ease of illustration, it should be understood that the contour of the depth plane may be curved in physical space such that all features within the depth plane are in focus with the eye in a particular accommodation state. Without being limited by theory, it is believed that the human eye may typically interpret a finite number of depth planes to provide depth perception. As a result, a highly realistic simulation of perceived depth may be achieved by providing the eye with a different representation of an image corresponding to each of these limited number of depth planes.

[0018] (Waveguide Stack Assembly) 4 illustrates an example of a waveguide stack for outputting image information to a user. The display system 400 includes a stack of waveguides or a stacked waveguide assembly 405 that can be utilized to provide a three-dimensional perception to the eye 410 or the brain using multiple waveguides 420, 422, 424, 426, 428. In some embodiments, the display system 400 may correspond to the system 200 of FIG. 2, of which FIG. 4 diagrammatically illustrates some portions of the system 200 in greater detail. For example, in some embodiments, the waveguide assembly 405 may be integrated into the display 208 of FIG. 2.

[0019] 4, the waveguide assembly 405 may also include a number of features 430, 432, 434, 436 between the waveguides. In some embodiments, the features 430, 432, 434, 436 may be lenses. In some embodiments, the features 430, 432, 434, 436 may not be lenses. Rather, they may be spacers (e.g., cladding layers and / or structures to form air gaps).

[0020] The waveguides 420, 422, 424, 426, 428 and / or the multiple lenses 430, 432, 434, 436 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image injection devices 440, 442, 444, 446, 448 may be utilized to inject image information into the waveguides 420, 422, 424, 426, 428, which may be configured to distribute the incident light across each individual waveguide, respectively, for output toward the eye 410. Light exits the output surfaces of the image injection devices 440, 442, 444, 446, 448 and is injected into the corresponding input edges of the waveguides 420, 422, 424, 426, 428. In some embodiments, a single beam of light (e.g., a collimated beam) may be launched into each waveguide and output an entire field of cloned collimated beams that are directed towards the eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0021] In some embodiments, image input devices 440, 442, 444, 446, 442 are each discrete displays that generate image information for input into each corresponding waveguide 420, 422, 424, 426, 428. In some other embodiments, image input devices 440, 442, 446, 446, 448 are outputs of a single multiplexed display that may pipe image information to each of image input devices 440, 442, 444, 446, 448, for example, via one or more optical conduits (such as fiber optic cables).

[0022] A controller 450 controls the operation of the stacked waveguide assembly 405 and the image input devices 440, 442, 444, 446, 448. In some embodiments, the controller 450 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to the waveguides 420, 422, 424, 426, 428. In some embodiments, the controller 450 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 450 may be part of the processing modules 224 and / or 228 (illustrated in FIG. 2 ) in some embodiments. In some embodiments, the controller may communicate with an inwardly facing imaging system 452 (e.g., a digital camera), an outwardly facing imaging system 454 (e.g., a digital camera), and / or a user input device 466. An inwardly facing imaging system 452 (e.g., a digital camera) can be used to capture images of the eye 410, for example, to determine the size and / or orientation of the pupil of the eye 410. An outwardly facing imaging system 454 can be used to image a portion of the world 456. A user can input commands into the controller 450 and interact with the display system 400 via a user input device 466.

[0023] The waveguides 420, 422, 424, 426, 428 may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 420, 422, 424, 426, 428 may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, the waveguides 420, 422, 424, 426, 428 may each include light extraction optical elements 460, 462, 464, 466, 468 configured to extract light from the waveguides by redirecting the light to propagate within each individual waveguide and outputting image information from the waveguides to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical elements may also be referred to as out-coupling optical elements. The extracted beam of light is output by the waveguide where the light propagating within the waveguide strikes the light redirecting element. The light extraction optical elements (460, 462, 464, 466, 468) may be, for example, reflective and / or diffractive optical features. Although shown disposed on the bottom major surfaces of the waveguides 420, 422, 424, 426, 428 for ease of explanation and clarity of drawing, in some embodiments the light extraction optical elements 460, 462, 464, 466, 468 may be disposed on the top and / or bottom major surfaces and / or directly within the volume of the waveguides 420, 422, 424, 426, 428. In some embodiments the light extraction optical elements 460, 462, 464, 466, 468 may be formed within a layer of material that is attached to a transparent substrate and forms the waveguides 420, 422, 424, 426, 428. In some other embodiments, the waveguides 420, 422, 424, 426, 428 may be a monolithic piece of material and the light extraction optical elements 460, 462, 464, 466, 468 may be formed on and / or within that piece of material.

[0024] Continuing with reference to FIG. 4, as discussed herein, each waveguide 420, 422, 424, 426, 428 is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 420 closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 420. The collimated light may represent an optical infinity focal plane. The next waveguide 422 may be configured to send collimated light that passes through a first lens 430 (e.g., a negative lens) before it can reach the eye 410. The first lens 430 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets the light emerging from that next waveguide 422 as emerging from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third waveguide 424 passes its output light through both a first lens 430 and a second lens 432 before reaching the eye 410. The combined refractive power of the first and second lenses 430 and 432 may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 424 as emerging from a second focal plane even closer inward toward the person from optical infinity where the light from the next waveguide 422 was.

[0025] Other waveguide layers (e.g., waveguides 426, 428) and lenses (e.g., lenses 434, 436) are similarly configured, with the highest waveguide 428 in the stack sending its output through all of the lenses between it and the eye for an aggregate focal power that represents the focal plane closest to the person. To compensate for the stack of lenses 430, 432, 434, 436 when viewing / interpreting light originating from the world 456 on the other side of the stacked waveguide assembly 405, a compensating lens layer 438 may be placed on top of the stack to compensate for the aggregate power of the lens stacks 430, 432, 434, 436 below. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. The light extraction optical elements 460, 462, 464, 466, 468 of the waveguides 420, 422, 424, 426, 428 and the focusing sides of the lenses 430, 432, 434, 436 may both be static (e.g., not dynamic or electroactive). In some alternative embodiments, either or both may be dynamic using electroactive features.

[0026] Continuing with reference to FIG. 4, the light extraction optical elements 460, 462, 464, 466, 468 may be configured to both redirect light from its respective waveguide and output this light with the appropriate amount of divergence or collimation for the particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes may have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 460, 462, 464, 466, 468 may be volume or surface features that may be configured to output light at specific angles. For example, the light extraction optical elements 460, 462, 464, 466, 468 may be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated by reference in its entirety. In some embodiments, features 430, 432, 434, 436 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers and / or structures to form an air gap).

[0027] In some embodiments, the light extraction optical elements 460, 462, 464, 466, 468 are diffractive features, i.e., "diffractive optical elements" (also referred to herein as "DOEs"), that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency, such that only a portion of the light of the beam is deflected toward the eye 410 with each intersection point of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related exit beams that exit the waveguide at multiple locations, resulting in a very uniform exit emission pattern toward the eye 410 for this particular collimated beam bouncing within the waveguide.

[0028] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which the microdroplets form a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match that of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0029] In some embodiments, the number and distribution of depth planes and / or depths of field may be dynamically varied based on the pupil size and / or orientation of the viewer's eye. In some embodiments, an inwardly facing imaging system 452 (e.g., a digital camera) may be used to capture images of the eye 410 and determine the pupil size and / or orientation of the eye 410. In some embodiments, the inwardly facing imaging system 452 may be mounted to the frame 212 (as illustrated in FIG. 2) and may be in electrical communication with the processing modules 224 and / or 228, which may process image information from the inwardly facing imaging system 452 and, for example, determine the pupil diameter and / or orientation of the user's eye 204.

[0030] In some embodiments, an inwardly facing imaging system 452 (e.g., a digital camera) can observe user movements, such as eye movement and facial movement. The inwardly facing imaging system 452 may be used to capture images of the eye 410 and determine the size and / or orientation of the pupil of the eye 410. The inwardly facing imaging system 452 can be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). Images obtained by the inwardly facing imaging system 452 may be analyzed to determine the user's eye pose and / or mood, which may be used by the display system 400 to determine audio or visual content to be presented to the user. The display system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an inertial measurement unit (IMU), accelerometer, gyroscope, etc. Head pose may be used alone or in combination with eye pose to interact with support tracking and / or to present audio content.

[0031] In some embodiments, one camera may be utilized per eye to determine the pupil size and / or orientation of each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adjusted to that eye. In some embodiments, at least one camera may be utilized per eye to independently determine the pupil size and / or eye pose of each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adjusted to that eye. In some other embodiments, the pupil diameter and / or orientation of only one eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the viewer 204.

[0032] For example, the depth of field may vary inversely with the viewer's pupil size. As a result, as the size of the viewer's eye pupil decreases, the depth of field increases so that one plane that is indistinguishable because its location exceeds the eye's focal depth may become distinguishable and appear more focused with a corresponding increase in the depth of field with a reduction in pupil size and a corresponding increase in the depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a reduced pupil size. For example, the viewer may not be able to clearly perceive the details of both the first and second depth planes at one pupil size without adjusting the eye's accommodation from one depth plane to the other. However, these two depth planes may be sufficient to simultaneously focus the user at another pupil size without changing accommodation.

[0033] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size and / or orientation, or in response to receiving an electrical signal indicative of a particular pupil size and / or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 450 may be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments in which the DOE for a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0034] In some embodiments, it may be desirable to have the exit beam meet a condition that it has a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, the exit beam size may be varied using a variable aperture.

[0035] The display system 400 may include an outward-facing imaging system 454 (e.g., a digital camera) that images a portion of the world 456. This portion of the world 456 may be referred to as the field of view (FOV), and the imaging system 454 is sometimes referred to as an FOV camera. The entire area available for viewing or imaging by the viewer 204 may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the display system 400. In some implementations of the display system 400, the FOR may include substantially all of the solid angle around the user 204 of the display system 400 (in front, behind, above, below, or to the side of the user) because the user 204 may move his or her head and eyes to view objects surrounding the user. Images obtained from the outward-facing imaging system 454 can be used to track gestures (e.g., hand or finger gestures) performed by the user, detect objects in the world 456 in front of the user, etc.

[0036] The display system 400 may include a user input device 466 through which a user may input commands into the controller 450 and interact with the display system 400. For example, the user input device 466 may include a track pad, a touch screen, a joystick, a multiple degree of freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a tactile device, a totem (e.g., functioning as a virtual user input device), and the like. In some cases, a user may use a finger (e.g., a thumb) to press or swipe on a touch-sensitive input device to provide input to the display system 400 (e.g., to provide user input to a user interface provided by the display system 400). The user input device 466 may be held by the user's hand during use of the display system 400. The user input device 466 may communicate with the display system 400 via wired or wireless communication.

[0037] 5 shows an example of an exit beam output by a waveguide. Although one waveguide is shown, it should be understood that other waveguides in the waveguide assembly 405 may function similarly and that the waveguide assembly 405 includes multiple waveguides. Light 505 is launched into the waveguide 420 at an input edge 510 of the waveguide 420 and propagates within the waveguide 420 by TIR. At the point where the light 505 impinges on the DOE 460, a portion of the light exits the waveguide as an exit beam 515. Although the exit beams 515 are shown as approximately parallel, they may also be redirected to propagate to the eye 410 at an angle (e.g., forming a diverging exit beam) depending on the depth plane associated with the waveguide 420. It should be understood that a nearly collimated exit beam may refer to a waveguide, with light extraction optics that outcouples light and forms an image that appears to be set on the depth plane at a long distance (e.g., optical infinity) from the eye 410. Other waveguides or other sets of light extraction optics may output a more divergent exit beam pattern that would require the eye 410 to accommodate to closer distances and focus on the retina, and would be interpreted by the brain as light from a distance closer to the eye 410 than optical infinity.

[0038] FIG. 6 illustrates another embodiment of a display system 400 including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The display system 400 can be used to generate multifocal stereoscopic images, images, or light fields. The display system 400 can include one or more primary planar waveguides 604 (only one is shown in FIG. 6 ) and one or more DOEs 608 associated with each of at least some of the primary waveguides 604. The planar waveguides 604 can be similar to the waveguides 420, 422, 424, 426, 428 discussed with reference to FIG. 4 . The optical system may employ a dispersive waveguide device to relay light along a first axis (vertical or Y-axis in the view of FIG. 6 ) and expand the effective exit pupil of the light along the first axis (e.g., Y-axis). The distributed waveguide device may include, for example, a dispersive planar waveguide 612 and at least one DOE 616 (illustrated by a double dashed line) associated with the dispersive planar waveguide 612. The dispersive planar waveguide 612 may be similar or the same as the primary planar waveguide 604 with a different orientation therefrom, at least in some respects. Similarly, the at least one DOE 616 may be similar or the same as the DOE 608, at least in some respects. For example, the dispersive planar waveguide 612 and / or the DOE 616 may be made of the same material as the primary planar waveguide 604 and / or the DOE 608, respectively. The optical system shown in FIG. 6 can be integrated into the wearable display system 200 shown in FIG. 2.

[0039] The relayed, exit pupil expanded light is optically coupled from the dispersive waveguide device into one or more primary planar waveguides 604. The primary planar waveguide 662 preferably relays the light along a second axis (e.g., the horizontal or X-axis in the diagram of FIG. 6) that is orthogonal to the first axis. It should be noted that the second axis can be a non-orthogonal axis to the first axis. The primary planar waveguide 604 expands the effective exit path of the light along its second axis (e.g., the X-axis). For example, the dispersive planar waveguide 612 can relay and expand the light along the vertical or Y-axis and pass the light to the primary planar waveguide 604, which relays and expands the light along the horizontal or X-axis.

[0040] The display system 400 may include one or more colored light sources (e.g., red, green, and blue laser light) 620 that may be optically coupled into a proximal end of a single mode optical fiber 624. The distal end of the optical fiber 624 may be threaded or received through a hollow tube 628 of piezoelectric material. The distal end protrudes from the tube 628 as an unfixed flexible cantilever 632. The piezoelectric tube 628 may be associated with four quadrant electrodes (not shown). The electrodes may be, for example, plated on the outside, outer surface or outer periphery, or diameter of the tube 628. A core electrode (not shown) is also located in the core, center, inner periphery, or inner diameter of the tube 628.

[0041] For example, drive electronics 636, electrically coupled via wires 640, drive opposing pairs of electrodes to bend the piezoelectric tube 628 in two axes independently. The protruding distal tip of the optical fiber 624 has a mechanical resonant mode. The frequency of the resonance may depend on the diameter, length, and material properties of the optical fiber 624. By oscillating the piezoelectric tube 628 near the first mechanical resonant mode of the fiber cantilever 632, the fiber cantilever 632 may be caused to oscillate and sweep through a large deflection.

[0042] By stimulating resonant vibrations in two axes, the tip of the fiber cantilever 632 is scanned in two axial directions within an area filling a two-dimensional (2-D) scan. By modulating the intensity of the light source 620 synchronously with the scanning of the fiber cantilever 632, the light emitted from the fiber cantilever 632 forms an image. A description of such a setup is provided in U.S. Patent Publication No. 2014 / 0003762, which is incorporated herein by reference in its entirety.

[0043] An optical coupler subsystem component 644 collimates the light emitted from the scanning fiber cantilever 632. The collimated light is reflected by a mirrored surface 648 into a narrow dispersive planar waveguide 612, which contains at least one diffractive optical element (DOE) 616. The collimated light propagates perpendicularly (with respect to the view of FIG. 6) along the dispersive planar waveguide 612 by total internal reflection, and in so doing repeatedly intersects with the DOE 616. The DOE 616 preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract toward the edge of the larger primary planar waveguide 604 at each point of intersection with the DOE 616, and a portion of the light to continue on its original trajectory down the length of the dispersive planar waveguide 612 via TIR.

[0044] At each point of intersection with the DOE 616, additional light is diffracted towards the entrance of the primary waveguide 612. By splitting the incoming light into multiple outcoupled sets, the exit pupil of the light is vertically expanded by the DOE 616 within the dispersive planar waveguide 612. This vertically expanded light that is outcoupled from the dispersive planar waveguide 612 enters the edge of the primary planar waveguide 604.

[0045] Light entering the primary waveguide 604 propagates horizontally (with respect to the diagram of FIG. 6) along the primary waveguide 604 via TIR. The light propagates horizontally along at least a portion of the length of the primary waveguide 604 via TIR as it intersects the DOE 608 at multiple points. The DOE 608 may be designed or configured to have a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern to produce both deflection and focusing of the light. The DOE 608 may advantageously have a low diffraction efficiency (e.g., 10%) such that only a portion of the light in the beam is deflected toward the viewer's eye at each intersection of the DOE 608, while the remainder of the light continues to propagate through the waveguide 604 via TIR.

[0046] At each point of intersection between the propagating light and the DOE 608, a portion of the light is diffracted towards an adjacent face of the primary waveguide 604, allowing the light to escape the TIR and emerge from the face of the primary waveguide 604. In some embodiments, the radially symmetric diffraction pattern of the DOE 608 additionally imparts a focal level to the diffracted light, both shaping (e.g., imparting curvature) the optical wavefronts of the individual beams and steering the beams to angles that match the designed focal level.

[0047] Thus, these different paths can couple light out of the primary planar waveguide 604 by resulting in different fill patterns at the DOE 608 multiplicity, focus level, and / or exit pupil at different angles. Different fill patterns at the exit pupil can be advantageously used to generate light field displays with multiple depth planes. Each layer or set of layers (e.g., three layers) in the waveguide assembly or stack may be employed to generate individual colors (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a first focal depth. A second set of three adjacent layers may be employed to generate red, blue, and green light, respectively, at a second focal depth. Multiple sets may be employed to generate full 3D or 4D color image light fields with various focal depths.

[0048] (Other components of the AR system) In many implementations, the AR system may include other components in addition to the wearable display system 80 (or optical system 100). The AR device may include, for example, one or more tactile devices or components. The tactile devices or components may be operable to provide tactile sensations to the user. For example, the tactile device or component may provide a tactile sensation of pressure and / or texture upon touching the virtual content (e.g., a virtual object, virtual tool, other virtual structure). The tactile sensation may replicate the feel of a physical object that the virtual object represents, or may replicate the feel of an imaginary object or character (e.g., a dragon) that the virtual content represents. In some implementations, the tactile device or component may be worn by the user (e.g., on a user-wearable glove). In some implementations, the tactile device or component may be held by the user.

[0049] An AR system may include one or more physical objects that can be manipulated by a user to enable, for example, input or interaction with the AR system. These physical objects are referred to herein as totems. Some totems may take the form of inanimate objects, e.g., pieces of metal or plastic, walls, the surface of a table. Alternatively, some totems may take the form of animate objects, e.g., the user's hand. As described herein, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem may simply provide a physical surface, and the AR system may render a user interface to appear to the user as being on one or more surfaces of the totem. For example, the AR system may render an image of a computer keyboard and trackpad to appear resident on one or more surfaces of the totem. For example, the AR system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself does not have any physical keys or trackpad or sensors, however, the AR system may detect user manipulations or interactions or touches with the rectangular plate as selections or inputs made via the virtual keyboard and / or virtual trackpad.

[0050] Examples of tactile devices and totems that can be used with the AR devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety.

[0051] (Example of Error Correction on a Display System) As described above, the display system may comprise a stacked waveguide assembly, such as that illustrated in FIGS. 4-6, having multiple display layers of substrate material with diffraction gratings for redirecting light to generate a digitized light field that impinges on the eye. In some embodiments, the waveguide assembly comprises one substrate layer per color per depth. For example, a two-depth planar RGB display can have a total of six waveguide layers. The display system can be an embodiment of a wearable display system 80.

[0052] In stacked waveguide assemblies, there is a range of potential phenomena that can introduce artifacts and cause degraded image quality. These can include persistence (multiple images), distortion, mismatch (between colors or depths), and color intensity variations across the field of view. In addition, certain types of artifacts can also occur in other types of conditions, such as when illuminating with a laser as opposed to an LED (e.g., speckle, banding, Newtonian fringes), or when the density of the outcoupled beam is below a certain amount (e.g., wavefield thinning, which can be perceived as if looking through a screen door or picket fence).

[0053] Due to imperfections in the optics of a light field display, the perfect three-dimensional grid in the rendering engine may be distorted when displayed through the optics. To identify and correct distortions between the expected image and the actual displayed image, a calibration pattern, such as a checkerboard pattern, can be projected using the display system.

[0054] FIG. 7 illustrates example distortions that may occur when a calibration pattern 702 is projected by a display system. The calibration pattern 702 can be any type of pattern suitable for performing spatial or chromatic calibration (e.g., a checkerboard pattern comprising a plurality of checkerboard squares). The calibration pattern 702 can include any type of test or calibration pattern, such as a geometric pattern or a random stochastic pattern. The projected calibration pattern 702 results in a generated light field image 704. The distortions present in the image 704 can include spatial distortions (e.g., when a visible pixel is not at an expected location in the field of view) and chromatic distortions (e.g., when the color value of a visible pixel differs from that expected). For example, the checkerboard squares of the pattern 702 may be shifted from their expected positions in the image 704 (e.g., spatial error). Additionally, instead of the checkerboard squares appearing in black and white, some checkerboard squares in the image 704 may appear in other colors, such as purple (e.g., chromatic error). Display errors can be measured using a light field metrology system, which may include a digital camera positioned to capture images of the calibration pattern projected by the display. In some embodiments, multiple images may be captured, corresponding to the calibration image shifted to different locations, to obtain finer granularity information regarding expected versus actual position. The digital camera can be configured to focus to different focal depths to determine the depth at which different regions of the displayed image (e.g., features on the displayed calibration pattern) are in focus.

[0055] Capturing multiple images at different focal depths and determining the depths of different regions of the images to be displayed, according to some embodiments, is described in more detail below in connection with Figures 17-20. Different types of calibration patterns that may be used in various embodiments are described in more detail below in connection with Figures 22-24.

[0056] (spatial error) Spatial errors can have several different manifestations. For example, spatial misalignment includes translation or rotation of display layers. Spatial errors can also involve non-linear spatial distortion that varies across the field of view (FOV) of the display's depth plane.

[0057] Spatial errors can be a manifestation of mechanical or optical imperfections in the display system. By interpreting the measured spatial errors, metrics can be derived that quantify the opto-mechanical quality of the system and suggest methods for improvement. For example, spatial errors that represent depth plane rotation may suggest that the display is mechanically rotated relative to the desired position. Scaling per color plane may suggest that the lens system is not sufficiently achromatic.

[0058] To identify spatial errors, a light field metrology system, comprising an image capture device such as a digital camera, can be used to capture one or more images projected by the display system (e.g., projection of a calibration pattern) and generate a vector field that represents the deviation of the actual displayed image from the expected image. The vector field may be a three-dimensional vector field, including in-plane deviations in the xy plane of the display and out-of-plane deviations in the z-direction (depth), or a two-dimensional vector field, including deviations only in the xy plane. In some embodiments, a vector field may be generated for each depth or color plane of the display system. In some embodiments, depth may be measured in diopters (representing the inverse of the focal length of the layer in meters).

[0059] FIG. 8 illustrates an example of a vector field that may be generated from one or more captured images that maps the deviation between an expected location of a point in a projected light field and its actual displayed location. The points in the projected light field may correspond to features in the calibration image (e.g., the center and corners of a calibration checkerboard square). Each vector in the vector field represents the distortion between an expected location in the light field and its corresponding actual location. In this example, the distortion vector field is 2D. In the illustrated vector field, the expected location of a feature is marked using a first color and marker type (e.g., "O" 802 for the expected location), while the actual displayed location of the feature is marked using a second color (e.g., "X" 804 for the detected location). Each pair of corresponding expected and displayed locations are connected by a line 806, which may include an arrow indicating the direction of correction needed to correct the detected displayed location to be at the expected location.

[0060] Using the vector field, local or global distortion information (e.g., in-plane translation, collective scaling, collective rotation, average pixel warping, or diopter error, as described below) can be extracted. For example, a distortion graph may be generated from the determined vector field. The distortion graph can be used to analyze the distribution of pixel position error values ​​(e.g., vector amplitude) across the generated vector field. The distortion graph may be a histogram showing the frequency of pixel position errors (e.g., plotting pixel position error amplitude against the frequency at which the error amplitude appears in the vector field). Other types of graphs may be used to analyze other attributes of the vector field (e.g., distortion direction).

[0061] Spatial errors can be broadly divided into in-plane and out-of-plane spatial errors. In-plane spatial errors refer to spatial errors along a particular depth plane (e.g., the xy-plane according to the coordinate system illustrated in FIG. 6) at a particular depth (measured on the z-axis). A vector field (e.g., as illustrated in FIG. 8) can be used to derive one or more metrics for different categories of spatial errors. Each of these metrics may be defined per layer (e.g., for each individual display layer corresponding to a particular combination of color and depth (e.g., a red-3 diopter display layer, a green-1 diopter display layer, etc.)) or per display (e.g., to quantify the overall fidelity of a display in concise parameters).

[0062] (In-plane spatial error) In some embodiments, the in-plane spatial error can be split into several different components, each corresponding to a different type of error. These components can include translation error, rotation error, scaling error, or non-linear spatial error. Each of these error components can be corrected individually or in succession.

[0063] (In-plane translation error) FIG. 9A illustrates an exemplary in-plane (xy) translation spatial error (also referred to as xy decentering). The xy translation error refers to an x- and / or y-pixel deviation of the center of the displayed image of the display layer from its expected location and is intended to inform mechanical or display alignment. In FIG. 9A, the expected image location 900 (shown in this example as a red rectangle) is translated to the displayed image location 900a (shown as a green shape with non-straight edges). The xy translation error can be corrected by identifying the center location 902 of the displayed image 900a and the center location 904 of the expected image 900, and making one or more deviations (along the determined translation vector 901) such that the displayed center location 902 is aligned with the expected center location 904 (through mechanical alignment of the display, software correction of the display image, or a combination of both). One or more metrics related to the measured xy translation spatial error may include a translation error measured per layer, measuring the layer center versus an expected or reference position (e.g., the optical axis of the display), or a maximum translation offset measured per display, indicating the maximum translation between any two display layers, to quantify the overall translation alignment.

[0064] (Collective rotation error) FIG. 9B illustrates an exemplary collective rotation spatial error. Collective rotation refers to the global rotation angle of the displayed image around its center relative to the image's expected location. Although spatial distortion may not always be perfectly describable by a simple affine rotation, collective rotation measurements can be used to provide a rotation angle at which pixel position error (between displayed image location versus expected image location) is minimized. Collective rotation metrics are intended to inform mechanical or display alignment. As illustrated in FIG. 9B, collective rotation can be corrected by rotating the displayed image 906 around a center point 908 by a specified rotation amount 907 to a position 910 that corresponds to the expected location (through mechanical alignment of the display, through software correction of the displayed image, or both). Reported metrics can include a measured rotation error per layer, indicating the measured orientation versus the expected or reference orientation (e.g., relative to the horizontal axis of the display), and a measured maximum rotation offset per display, indicating the maximum rotation error between any two display layers to quantify the global rotation alignment.

[0065] (ensemble scaling error) FIG. 9C illustrates an example of collective scaling spatial error. Collective scaling indicates the global scaling factor of a displayed image around its center relative to an expected image. Although spatial distortion may not be fully described by simple affine scaling, collective scaling measurements may indicate the scaling factor at which pixel position errors are minimized. Collective scaling metrics are intended to inform optical design or display alignment. As illustrated in FIG. 9C, collective scaling spatial errors may be corrected by scaling the size of a displayed image 912 by a specified scaling amount 913 to match that of an expected image 914. Reported metrics for collective scaling may include scaling errors measured per layer, which measure image scaling versus expected or reference scaling (e.g., with reference to a physical target in a calibrated setup), and maximum scaling offsets measured per display, which indicate the maximum scaling between any two display layers to quantify global scale alignment.

[0066] 9D illustrates another example of an aggregate scaling spatial error. A displayed image 916 appears smaller compared to an expected image 918. To correct for the scaling error, the displayed image 916 is scaled upward by a scaling amount 917 to match the size of the expected image 918.

[0067] (Pixel Warping Error) FIG. 9E illustrates an example of the residual spatial error after XY translation, rotation, and scaling corrections have been made. The residual error (also referred to as pixel warping or spatial mapping) indicates the average residual Euclidean pixel position error after xy translation, rotation, and scaling have been removed from the global spatial distortion profile (e.g., as illustrated in FIGS. 9A-9D), and may be used to provide a measure of the non-linear or non-affine warping characteristics of the display system and inform display design and quality control. Metrics reported for pixel warping may include mean pixel warping (MPW), measured per layer, indicating the average residual Euclidean pixel position error after xy translation, rotation, and scaling have been removed with reference to a perfect grid, and maximum mean pixel warping (maxMPW), measured per display, indicating the maximum value of MPW between layers of a display to quantify the global warping. In some embodiments, the remaining pixel warping can be corrected through spatial mapping performed using a processing module (e.g., module 224 or 228) to align the displayed image 920 with the expected image 922.

[0068] (Out-of-plane spatial error) Digital light field display systems such as those illustrated in Figures 4-6 are capable of generating depth planes that appear at different depths (z-direction) from the viewer (see, e.g., Figure 3). In some embodiments, the depth planes correspond to flat planes that appear to be located at different distances from the viewer. Rather than referring to the distance of the depth plane from the display, as is common in optics, the distance is referred to in diopters (m -1Inverse distance, measured in units of dpi, can be used to refer to the different depth planes. For example, a display may have two depth planes located at depths of 3 diopters (1 / 3 m) and 1 diopter (1 m). Due to imperfections in the display system, the diopter profile across the depth planes may not be as expected. For example, images displayed on a depth layer may have a diopter profile with incorrect distance or varying focus across the FOV of the display.

[0069] Out-of-plane spatial error (also referred to as diopter error) is a measurement of diopter (depth) error in the depth planes and is intended to account for errors in optical, mechanical, and waveguide alignment or design. Metrics reported for diopter error may include measured diopter error per layer, which indicates the amount of error between the expected and measured depths of the depth planes, and maximum diopter error, which indicates the maximum depth error between depth planes.

[0070] 10A illustrates an example of multiple depth planes intended to be viewed at different depths. In the illustrated example, three different depth planes are shown, but the display system may contain more or fewer depth planes. In addition, each depth plane may correspond to multiple waveguide layers (e.g., RGB color layers).

[0071] 10B-10D illustrate examples of the types of out-of-plane spatial errors that may occur when viewing the projected depth plane shown in FIG. 10A. For example, the projected depth plane may be shifted to a different depth, appearing to be at a greater or lesser depth than expected (FIG. 10B). The depth plane may be misaligned to exhibit a bulk rotation from the expected depth (FIG. 10C). The depth plane may exhibit a non-uniform profile characteristic of lattice imperfections (FIG. 10D). The depth plane may exhibit a combination of errors illustrated in FIGS. 10B-10D.

[0072] 10E illustrates another example of an out-of-plane spatial error. The projected depth plane 1002 is misaligned with respect to the expected depth plane 1004. In the illustrated example, the misalignment comprises a depth plane rotation. To correct the out-of-plane spatial error, a rotation axis 1006 may be identified and a rotation may be performed on the projected depth plane 1002 about the identified rotation axis 1006 such that the projected depth plane 1002 is substantially aligned with the expected depth plane 1004. Although the rotation axis 1006 is illustrated as parallel to the axis (e.g., vertical axis) of the expected depth plane 1004, it should be understood that the rotation axis may be in any orientation.

[0073] Diopter error is distinct from in-plane spatial error, which is related to in-plane distortion, but diopter error can potentially affect in-plane spatial error by introducing field-dependent spatial distortion, for example, due to incorrect assumptions of pixel depth. For example, for a defective depth plane with regions at a different depth than expected, pixels may shift non-uniformly relative to the viewer position, introducing variable image warping.

[0074] In some embodiments, the error correction techniques described herein for in-plane spatial errors (e.g., xy decentering, collective scaling, collective rotation, and spatial mapping) can be extended to three dimensions. For example, decentering may be performed in three dimensions by identifying the location of the center point of the displayed plane on the xyz coordinate system and shifting the plane (e.g., along the x, y, and z axes) so that the center point aligns with the expected location.

[0075] (Quantification of spatial error based on distortion vector fields) As described herein with reference to FIG. 8, a multi-dimensional (e.g., 2D or 3D) distortion vector field can be generated by measuring the displacement of image features from expected positions to displayed positions. The distortion vector field can be calculated for each layer of a multi-layer display (e.g., a display comprising stacked waveguide assemblies 405). The distortion vector field can be used to capture and characterize the distortion of a light field projected by the display. For example, vector analysis operations can be performed on the distortion vector field to determine certain spatial errors. A light field metrology system can calculate such vector operations as part of an analysis of an image obtained by a metrology camera (e.g., a digital camera or a light field camera) with respect to a calibration pattern (e.g., a checkerboard pattern) projected by the display. Such vector analysis techniques are not limited to light field displays and can be applied to any multi-dimensional metrology or calibration of any type of display.

[0076] Given a multi-dimensional distortion vector field, the curl of the vector field can be calculated to determine local rotation. The average of the curl over a region within the FOV of the display provides a measure of the aggregate rotation error in the region. In a discrete depth plane implementation of a light field display, calculating the curl of the distortion vector field can provide information about the in-plane or out-of-plane rotation of the layer.

[0077] The divergence of the distortion vector field can be calculated to determine the scaling error. In implementations with multiple layers (e.g., RGB color layers) to generate full color images at each depth, this scaling error can be used to provide information for scaling calibration.

[0078] Vector integral type theorems (e.g., Stokes' theorem or the divergence theorem (Gauss's theorem)) can be applied to the distorted vector field to calculate the curl and divergence of the vector field over a region within the FOV of the display (e.g., to find the aggregate rotation or aggregate scaling of the region). The Euclidean mean of the vectors in the distorted vector field can be calculated to obtain information about the non-affinity of the spatial transformation introduced by the distortion.

[0079] (Quantification of color error) Chromatic errors occur when the color values ​​of a visible pixel differ from the expected color values. To assess chromatic errors, a calibration image may be projected using the display system. The calibration image may be the same calibration image used to perform spatial error correction, or may be a different calibration image. For example, the calibration image may comprise a solid image of a particular color, such as red, at a particular brightness level (e.g., maximum brightness). The output from projecting the calibration image may be captured using an image capture device (e.g., one or more cameras). FIG. 11 illustrates an example of a captured image of a projected calibration image. The calibration image may have a brightness level that is constant throughout the image, but the brightness of the displayed calibration image varies across the field of view of the display due to the presence of chromatic errors. For example, some regions 1102 of the captured image may have a high brightness level, while other regions 1104 may exhibit a lower brightness level, causing dark areas or bands to appear across the display. In some embodiments, the calibration image may comprise a colored calibration pattern rather than being colorless.

[0080] In some embodiments of the display, the observed luminance topology may be wavelength dependent. For example, the luminance variation may be different for red, green, and blue, causing the projected image to appear in a color other than expected (indicating an imbalance between the red, green, and blue components). For example, the projected white calibration image may appear to be purple with a green luminance level lower than that of the red and blue. In addition, the luminance variation may also be based on the observer location (e.g., if the camera is moved, the dark band in 1102 may appear to move to different locations within the FOV). This phenomenon may lead to difficulties in maintaining color uniformity and white balance across the FOV (especially since the luminance or color balance may depend on the observer location), ultimately affecting the color accuracy of the displayed content.

[0081] Each display layer in a display system is associated with a chromatic characteristic, a measured color, and a luminance characteristic that measures brightness or intensity. Thus, chromatic error can be broadly divided into luminance flatness error and chromatic uniformity error.

[0082] (Brightness uniformity) The luminance flatness metric can be used to quantify the amount of variation in luminance exposed by a single display layer. In general, in a stacked waveguide assembly, different display layers can potentially have different luminance variations across the field of view due to each display layer being produced by a different waveguide in the stack (see, for example, waveguide assembly 405 in FIG. 4).

[0083] To measure luminance flatness for a display layer, luminance values ​​(also referred to as intensity values) can be determined for some or all pixels of a captured image. This disclosure primarily refers to pixel luminance values, but in other implementations, luminance values ​​can be determined for an area comprising multiple pixels (e.g., an N×M grid of pixels) instead of individual pixels. In some embodiments, each determined luminance value can be assigned an luminance bin comprising one or more ranges of luminance values. For example, for an 8-bit color display system, 256 bins corresponding to 8-bit color can be used.

[0084] From the determined luminance values, several luminance flatness metrics can be calculated by the metrology system. For example, a mode can be calculated, which indicates the most common pixel luminance value across the displayed field. From the mode, a half pixel population range (HPPR) can be determined, which indicates the number of luminance ranges or luminance bins adjacent to the mode that cover 50% of the pixel population. A small HPPR indicates that the luminance for the display layer is substantially uniform across the display. The luminance value may also be referred to as an intensity value. For purposes of this application, the terms "luminance" and "intensity" may be used interchangeably.

[0085] 12A illustrates an intensity histogram that may be generated from a captured image of a projected calibration image (e.g., as illustrated in FIG. 11). An intensity histogram plots luminance values ​​against the frequency with which they appear in the captured image (e.g., the number of pixels having that luminance value). The mode is indicated by the luminance value that has the highest number of occurrences in the image (e.g., at location 1202).

[0086] 12B illustrates an intensity profile generated from a captured image of the projected calibration image. In the illustrated intensity profile, the mode occurs at brightness value 1204 (which in this example has a value of 236). From the mode, it is determined that a deviation range centered on mode 1204, shown as the range between brightness values ​​1206 and 1208, covers 50% of the pixel population of the image. The HPPR is determined based on the calculated deviation range (e.g., the difference between brightness values ​​1206 and 1208).

[0087] For an ideal display layer, the intensity values ​​would be uniform across the field for a given input illumination (e.g., HPPR=0). Deviations from this ideal behavior would be manifested as a distribution of pixel intensity values ​​away from the mode. HPPR measurements attempt to measure the distribution away from the mode. A substantially uniform luminance may have a small HPPR, e.g., a small HPPR compared to the mode or range of possible luminance values ​​(e.g., 255 for 8-bit color). For example, a substantially uniform (e.g., flat) luminance display may have a ratio of HPPR to total color range of less than about 10%, less than about 5%, less than about 1%, or less than about 0.1%.

[0088] HPPR can be considered as the variation of the interquartile range, which measures the distribution away from the median instead of the mode. The median of pixel intensity values ​​may not have a direct relationship to the desired flat-intensity response of the display layer. FIG. 13 illustrates exemplary intensity histograms 1302, 1304 illustrating the difference (μ) between the mode, median, and mean. The medians of the two distributions 1302, 1304 are identical in this example. The two distributions 1302, 1304 have standard deviations σ of 0.8 and 2, respectively. As illustrated diagrammatically in FIG. 13, if the intensity distribution of the image is close to the normal (e.g., intensity distribution 1302), the mode, median, and mean may all be very similar. On the other hand, if the intensity distribution is not close to the normal distribution (e.g., intensity distribution 1304), the mode, median, and mean of the intensity distribution may be substantially different from each other.

[0089] For each display layer of a display, luminance flattening attempts to reduce the luminance variation across the displayed field of view. Typically, the luminance intensity of a pixel cannot be increased beyond its maximum value, so luminance flattening is generally a global luminance reduction step, where pixel luminances are compressed within a layer-specific profile such that the luminance of that layer is as flat as possible.

[0090] For example, luminance flattening can be performed such that the pixel luminance has a maximum value at the luminance value of the pixel with the lowest luminance value, reducing the luminance of the display layer to a substantially minimum luminance. Alternatively, the pixel luminance can be configured to have a maximum value at a selected luminance value that is above the luminance value of the pixel with the lowest luminance value. This may not reduce the overall luminance to a minimum value, since there may still be pixels with luminance values ​​below the selected value and luminance non-uniformity may remain. In some embodiments, reducing the luminance value for the pixel or group of pixels includes identifying a value for reducing the luminance value of the pixel or group of pixels. In other embodiments, reducing the luminance value for the pixel or group of pixels includes identifying a scaling factor for scaling the luminance value of the pixel or group of pixels downward to a minimum or threshold luminance value.

[0091] In some embodiments, if the initial luminance flatness of the display layer is good (e.g., HPPR is below a threshold), the luminance value may be reduced to a minimum to provide a flat luminance field. On the other hand, if the luminance flatness is poor (e.g., HPPR is above a threshold) or the minimum luminance value is low (e.g., does not reach the minimum threshold), a selected maximum luminance value may be chosen. Luminance flatness may be performed within a software module (e.g., processing module 224, 228).

[0092] The level to which the luminance is reduced when performing luminance flattening may vary from display layer to display layer. However, different luminance levels for different layers within the same color cluster (e.g., RGB layer cluster) may lead to a loss of white balance, which can be addressed by correcting the chromatic uniformity of the display.

[0093] (color uniformity) Color generally refers to the color components of a display independent of brightness. As described above, the display layers in the display system may comprise a red display layer, a green display layer, and a blue display layer, although it should be understood that in other implementations, other numbers, types, or colors of display layers, or combinations of display layers, may be used. In the following examples, RGB color layers will be described for illustrative purposes, but this is not a limitation on the method for color balancing, which can be applied to any set of display colors.

[0094] If the luminance variation of corresponding red, green, and blue display layers is the same, the color will be maintained across the display. On the other hand, if the luminance variation across corresponding red, green, and blue display layers is different, the color of the displayed image will be different from that expected. For example, for a white calibration image, if the red and blue layers have a higher luminance than the green layer, areas of the white calibration image may appear purple in color. These deviations from the intended white color may be referred to as off-grayscale.

[0095] A chromatic uniformity metric can be used to capture the degree of off-grayscale of an image. Metrics may include average color error, which indicates the average over the FOV of the deviation of red, green, and blue colors from their corresponding averages, respectively. The smaller the average color error, the closer the image is to the grayscale it would appear. The average color error may be normalized to a dimensionless value by dividing by the average color or range of possible colors (e.g., 255 for 8-bit color). In various implementations, a display can be considered to have chromatic uniformity achieved if the average color error is less than 10%, less than 5%, less than 1%, or some other threshold.

[0096] 14A illustrates an example of a red-green-blue (RGB) intensity map generated from a captured image of a projected test image. The red and blue layers 1402 and 1404 generally have similar brightness to one another, and both have a much higher brightness than the green layer 1406. As a result, a projection of a white test image will have areas that appear to be purple (red + blue, see, e.g., FIG. 11B).

[0097] 14B illustrates a plot 1408 that maps the maximum color imbalance error. The average luminance 1410 can be determined as the average luminance value of the red, green, and blue layers. The "average + max error" surface 1412 shows the maximum luminance value of the red, green, and blue layers, while the "average - max error" surface 1414 shows the minimum luminance value of the red, green, and blue layers.

[0098] 15 illustrates an RGB intensity map for a display system with red, green, and blue layers having different intensities across the displayed field of view as illustrated in FIG 14A after chromatic correction. As described below and illustrated in plot 1500, in this example, the maximum R and B luminance values ​​have been reduced to the level of the lower G luminance value in most of the display to provide chromatic uniformity.

[0099] As illustrated in FIG. 14A, prior to color correction, the luminance of the red and blue layers is much higher than that of the green layer over most of the FOV, which may result in large areas of the captured image of the white calibration image appearing purple. In this embodiment, during point-by-point color correction of the depth plane, the lowest luminance value of the color layers (e.g., red, green, and blue) associated with the depth plane is identified, and the luminance value for each color layer is set to the lowest luminance value for that point. For example, as illustrated in FIG. 15, the color luminance of the red and blue layers 1502 and 1504 is lowered to match that of the green layer 1506 (e.g., by comparing the RGB intensity map of FIG. 14A with the RGB intensity map of FIG. 15). As a result, the luminance of the red and blue layers is corrected so that they match the intensity of the green layer, reducing the amount of off-grayscale in the projected image.

[0100] (Image correction process) Image calibration refers to the characterization of a display device relative to previously defined image quality metrics (see, e.g., description with reference to Figures 7-15). Image correction refers to corrective actions taken to improve image quality. Image quality metrics inform corrective actions taken that attempt to improve or optimize the display device image quality metrics. Image correction is therefore tightly tied to each of the image quality metrics.

[0101] 16 is a flow chart of an embodiment of a process 1600 for performing image correction on a display system. In block 1602, a camera used to capture a projected image (e.g., camera 1806 of metrology system 1800 described below) is calibrated. Camera calibration involves characterizing the accuracy of the camera in capturing and representing actual visual / display information. To ensure that any measured metrics from the captured image are due to the display system and not errors associated with the camera, the camera used for image correction should be fully calibrated before image correction is attempted.

[0102] In some embodiments, the camera calibration includes performing at least one of flat-field correction (e.g., ensuring that the camera's intensity response is uniform across its FOV), lens distortion correction (e.g., identifying and compensating for lens distortion), or pixel scaling (e.g., identifying the relationship between the pixel size for the camera's image capture versus the pixel size of the display system). In some implementations, a display / camera pixel mapping can be applied to perform the conversion between the display pixel values ​​and the camera pixel values. The display / camera pixel mapping can be based on a first global nonlinear gamma function that maps the display color pixel values ​​to a first intermediate color space, a local pixel-dependent combination function that maps the first intermediate color space to a second intermediate color space, and a second global nonlinear gamma function that maps the second intermediate color space to pixel intensities in the camera color space. Details of an exemplary display / camera pixel mapping are described below with reference to FIG. 21.

[0103] In block 1604, spatial error correction can be performed on the display system. Spatial error correction can include capturing one or more images of the projected light field using a calibrated camera, which can be used to generate a vector field indicative of the distortion between the displayed image location and the expected image location. In some embodiments, a separate vector field is generated for each display layer. Using the generated vector field, one or more spatial corrections can be performed, which can include XY decentering (block 1604a), collective rotation (block 1604b), collective scaling (1604c), or spatial mapping (block 1604d). In some embodiments, each of these corrections is performed on a layer-by-layer basis.

[0104] XY decentering may refer to the translation spatial error of the center of the displayed image of the display layer relative to the expected image location. Performing XY decentering may include identifying a center point of the displayed image and shifting the image along a determined translation vector such that the center point corresponds to the expected center location. An example of XY decentering correction is described with reference to FIG. 9A.

[0105] A collective rotation may refer to the overall rotation error between the displayed image and the expected position. Performing a collective rotation may include identifying a center point of the displayed image and rotating the image about the identified center point by a specified rotation amount (e.g., to a position where pixel position error relative to the expected image position is minimized). An example of collective rotation correction is described with reference to FIG. 9B.

[0106] Aggregate scaling may refer to the overall scaling error between the displayed image and the expected image. Performing aggregate scaling may include identifying a center point of the displayed image and scaling the image about the identified center point by a specified factor (e.g., a factor that minimizes pixel location error relative to the expected image location). An example of aggregate scaling is described with reference to Figures 9C and 9D.

[0107] Although the xy decentering, collective rotation, and collective scaling can be used to correct linear or affine spatial errors, the displayed image of the display layer may also contain additional non-linear or non-affine spatial errors. Spatial mapping can be performed to correct any remaining errors (e.g., non-linear or non-affine errors) that remain after the xy decentering, collective rotation, and collective scaling corrections have been performed. Spatial mapping can also be referred to as pixel warping, and an example is described with reference to FIG. 9E.

[0108] In some embodiments, spatial error may be split into in-plane spatial error and out-of-plane spatial error (sometimes also referred to as diopter error). For example, the display layers may first be corrected for in-plane spatial error before being corrected for out-of-plane spatial error, or vice versa. Alternatively, the in-plane and out-of-plane spatial errors may be corrected together.

[0109] In block 1606, color error correction can be performed on the display system. The color error correction may comprise luminance flattening (block 1606a) or color balancing (block 1606b). In some embodiments, luminance flattening is performed layer by layer, while color balancing is performed color cluster by color (e.g., RGB cluster by RGB cluster).

[0110] Luminance flattening may refer to the reduction of luminance variations across a display layer. In some embodiments, luminance flattening includes reducing the luminance of all pixels in the displayed FOV to a minimum luminance value. Alternatively, all pixels in the displayed FOV that have a luminance above a maximum or threshold value may have their luminance reduced to the maximum / threshold value, while pixels with luminance below the maximum / threshold value may remain unchanged. In some embodiments, the luminance value may be scaled based on the distance between the luminance and the threshold luminance value. An example of luminance flattening is described with reference to Figures 12A and 12B.

[0111] Color balancing may include reducing off-grayscale effects caused by mismatches in intensity between different color layers in a color cluster (e.g., an RGB cluster). Color balancing can be done by lowering the luminance of a color layer at each location in the depth plane to match that of a color layer in the color cluster with the lowest luminance at that location. For example, for each pixel in the FOV, the luminances for the red, green, and blue color layers at each location are all set to the lowest value of the three color layers at that location. In some embodiments, luminances above a threshold luminance value are lowered to the greater of either the threshold luminance value or the minimum luminance value in the color cluster at that location. In some embodiments, the luminance may be scaled based on the distance between the luminance and the threshold luminance value. Examples of color balancing are described with reference to FIGS. 14A-15.

[0112] In some implementations, image calibration (to quantify image quality metrics) is performed for each display system during the manufacturing process. Information associated with the image quality metrics and corrections that can be used to improve or optimize the display system can be stored in a non-transient memory associated with the display system (e.g., data module 224 or data repository 232). During use of the display system, the image correction information can be applied to the display to make appropriate corrections such that a user of the display system is provided with an improved or optimized image that reduces or eliminates image errors in the display. For example, a local or remote processing module 224, 228 can use the image correction information to provide an improved image to a user on a real-time basis. Details of an exemplary calibration process are described below with reference to FIGS. 27 and 28.

[0113] (Example of depth plane measurement) The display system embodiments described herein are capable of generating light fields (see, e.g., description with reference to FIGS. 1-6). Thus, just as real (physical) objects at a distance from the display wearer would generate a light field impinging on the eyes, a virtual object placed at a depth would create a (digitized) light field that would make it appear in focus at the intended depth. This enables vergence-accommodation matching and more believable mixed reality displays.

[0114] Although a content creator may place a virtual object at a certain depth from the viewer in the rendering engine due to imperfections in the generated light field (e.g., due to imperfections in the waveguides of the waveguide assembly 405), the virtual object may appear focused at a different depth than intended. This may result in vergence-accommodation mismatches. In some cases, different parts of the virtual object may appear focused at different depths. These depth mismatches may correspond to types of out-of-plane spatial errors such as those illustrated in FIGS. 10A-10E.

[0115] Thus, this disclosure describes examples of metrology systems that can measure the quality of the light field generated by a display. Some such metrology systems can map the topology and quality of the light field generated by the display, providing information that can lead to an assessment of the quality of the light field generated by the display. Some such metrology systems can capture the vector light field (e.g., direction and amplitude) generated by the display, allowing analysis of focus and depth imperfections within the display. Spatial and chromatic calibration techniques for light field displays have been developed that utilize information generated by the metrology systems described herein. While embodiments of the metrology systems described herein have particular application to light field displays (e.g., embodiments of display systems 80, 100), this is not a limitation and other embodiments of the metrology systems can be used to measure light resulting from any type of display. Embodiments of the metrology systems can be used to determine 3D distortion fields, which can be used to derive useful spatial calibration information for the display. The metrology systems can also be used for binocular calibration and monocular RGB and depth inter-plane calibration.

[0116] 17A illustrates an example of an object 1702 viewed by an eye 304 with a normal light field. The object 1702 may correspond to a real object or a virtual object generated with a light field substantially free of imperfections. Light rays 1706 associated with points on the object 1702 result in points on the object 1702 appearing to diverge from a single point and appearing to focus at a distance 1708 from the eye 304.

[0117] FIG. 17B illustrates an example of an object 1710 viewed with an incomplete light field. The object 1710 may correspond to a virtual object, such as a virtual object generated using a display system (e.g., the display system 400 as illustrated in FIGS. 4 and 6). Due to imperfections in the generated light field, e.g., due to imperfections in the waveguides 420, 422, 424, 426, 428, 604, light rays 1712 intended to correspond to a particular point on the object 1710 may appear to diverge from a different point or exhibit a different divergence than otherwise intended. As a result, the object 1710 may appear out of focus in the distance 1708. In addition, different portions of the object 1710 may also appear to be in focus at different depths or distances.

[0118] A metrology system can be used to measure the quality of the light field generated by the display. FIG. 18 illustrates an example of a metrology system 1800 for measuring the light field quality of a display 1802. The display 1802 generates a light field having light rays 1804 directed towards a camera 1806. The display device 1802 may correspond to a stacked waveguide assembly (e.g., stacked waveguide assembly 405 as illustrated in FIG. 4). Although the light rays 1804 are illustrated as approximately parallel, this is for illustrative purposes and the light rays 1804 may be projected in different directions (e.g., divergent) to convey different depths of one or more virtual objects represented in the light field. In addition, the light rays 1804 may be non-parallel due to imperfections in the display 1802 (see, e.g., FIG. 17B).

[0119] In some embodiments, the camera 1806 can be used to capture at least a portion of the generated light field, for example, to measure the perceived depth of a virtual object represented in the light field. The camera 1806 can be configured to focus on a particular depth or distance (hereinafter also referred to as "depth of focus"). In some embodiments, this may be done using a lens with a small depth of focus (DOF). For example, the DOF may be less than the Z-distance (e.g., less than the distance between the peak of the depth map 1924 and the intended depth of focus 1922 shown in FIG. 19C) where imperfections in the display typically cause the depth of focus to deviate from the intended depth of focus. In other examples, the DOF may be less than a factor times the distance between the camera and the display, where the factor may be less than about 0.1, less than about 0.01, less than about 0.001, etc. The camera 1806 may be configurable to capture a specific portion of the light field or the entire light field. The camera 1806 may be configured to capture a portion of the light field associated with a particular virtual object to be displayed using the light field. The camera 1806 may be positioned such that it can capture an image substantially similar to that which would be perceived by the eye 304. The camera 1806 and the display 1802 can be movable relative to one another to map the light field. For example, the relative motion can be parallel to the display 1802 (e.g., the X-direction shown in FIG. 18 or a Y-direction (not shown) that is perpendicular to X and Z) or perpendicular to the display 1802 (e.g., the Z-direction shown in FIG. 18). In other implementations, scanning optics (not shown) can be used to relatively scan the camera 1806 and the display 1802. In some embodiments, camera 1806 may be used to capture a portion of the generated light field to determine a distortion map (such as that illustrated in FIG. 8), which may be used to identify spatial errors in the projected image (e.g., in-plane spatial errors as illustrated in FIGS. 9A-9E or out-of-plane spatial errors as illustrated in FIGS. 10A-10E).Additionally, the camera 1806 may be used to identify brightness or color errors in the generated light field (eg, as illustrated in FIGS. 11-15).

[0120] In some embodiments, the camera 1806 is movable so that it is oriented in different directions. For example, while the camera 1806 is illustrated as facing orthogonally to the display 1802, the camera 1806 may also be rotated (e.g., along a Y- or X-rotational axis) to face different angles relative to the display 1802, allowing the camera 1806 to measure the light field generated by the display 1802 in different directions or orientations.

[0121] In various embodiments, the camera 1806 can be a digital camera, such as a short focal length digital camera, hi other embodiments, the camera 1806 can be a light field camera.

[0122] The camera 1806 can be connected to a controller 1808, which can be used to control the focal depth of the camera 1806, the field of view of the camera 1806, exposure time, relative movement of the camera 1806 and the display 1802, and the like. In some embodiments, the controller 1808 may correspond to the controller 450 as illustrated in Figure 4. The controller 1808 can include a hardware processor and non-transitory data storage.

[0123] FIG. 19A is a schematic diagram of an example of an image 1900 that may be captured by a camera (e.g., camera 1806) focused on a particular focal depth. Image 1900 may contain one or more regions 1902 in focus and one or more regions 904 out of focus. Because camera 1806 can be configured to focus at different focal depths, the regions of the image that are in focus or out of focus may vary. For example, if the camera is changed to focus on a different focal depth, region 1902 may appear out of focus while portions of region 1904 may be in focus. By capturing multiple images of the light field across multiple different focal depths, the perceived depth for various regions of the light field can be determined. For example, each pixel of an image captured by a camera may be associated with a particular focal depth that corresponds to the focal depth at which the portion of the light field corresponding to the pixel is in focus. A depth map or graph may be constructed to map regions of the generated light field and their perceived depth. In addition, the depth map or graph may also define the focal depth intended to be projected by the display, thereby enabling a comparison between the intended focal depth of a virtual object displayed in the light field and the actual measured focal depth.

[0124] 19B is an example of a depth graph that diagrammatically illustrates focal depth measurements that may be made by an embodiment of the metrology system 1800. The graph 1910 plots the measured focal depth 1912 of the generated light field along a line spanning the light field emitted from the display 1802 (e.g., along the horizontal X-axis of the light field as illustrated in FIG. 18). In some embodiments, the graph 1910 may be generated by sweeping the focal depth of the camera 1806 across multiple different focal depths. For example, the camera 1806 may be focused at focal depth 1914 (illustrated by the horizontal dashed line). In a perfect display, the light field generated by the display would be such that the actual measured depth of a virtual object is exactly the intended depth, but in an actual display the two may differ due to imperfections in the display. Thus, any region of the light field with a measured depth of focus that approximates depth of focus 1914 (e.g., region 1916) may be perceived as substantially in focus, while a region of the light field with a measured depth of focus that differs significantly from depth of focus 1914 (e.g., region 1918) may be perceived as out of focus.

[0125] 19C illustrates an example of a depth map that may be generated based on one or more captured images. Depth map 1920 contains intended depth locations 1922 (illustrated as a horizontal plane in FIG. 19C) at which an image generated by display 1802 should be focused, and a measured depth map 1924 that indicates the focal depth (Z) at which the image actually focuses. A comparison between the intended focal depth 1922 and the measured focal depth 1924 allows imperfections in the light field generated by display 1802 to be identified and quantified across the display's field of view (FOV).

[0126] For example, if the intended focal depth is Z0 for light to be focused at a horizontal position (X0,Y0), and the measured focal depth at that position is Z, then (Z-Z0) is a measurement of the focus imperfection of the display at position (X0,Y0). In some implementations, the actual horizontal position (X,Y) at which the light rays are focused can be measured. In some such implementations, a vector measurement (X,Y,Z)-(X0,Y0,Z0) of the actual focus position relative to the intended focus position can be used to characterize the imperfections in the light field produced by the display. This vector measurement of the display imperfections provides a 3D representation of both in-plane and out-of-plane (e.g., diopter) errors. In some embodiments, only the in-plane error is measured (and calibrated) using the 2D vector error measurement (X,Y)-(X0,Y0). In some cases, the focus error can be determined on a pixel-by-pixel basis for the display. However, due to the large number of pixels in many displays (e.g., millions of pixels), the focus error data may be determined for only a portion of the display or a group of pixels sampling the display (e.g., 10x10 or 100x100 samples across the display). The checkerboard pattern does not need to be square and can be designed to conform to the pixel structure of the display.

[0127] FIG. 20 is a flow chart of an example of a process 2001 for measuring the quality of a virtual target pattern generated using a light field display. The process 2001 can be performed by the measurement system 1800, e.g., the controller 1808. The virtual target pattern is a checkerboard pattern, in some implementations, with an array of alternating light and dark areas. The checkerboard pattern may be used to sample a portion of the display (e.g., a 10×10 or 100×100, or other size checkerboard), or may have a size corresponding to the number of pixels in each dimension of the display. In other cases, per-pixel data can be acquired by sequentially turning on and off one (or more) groups of pixels and acquiring images of the turned-on pixels. The checkerboard pattern (or sequence of turning pixels on / off) may include a random stochastic sequence of light and dark areas or a geometric pattern of light and dark areas or any other type of calibration pattern. Examples of checkerboard patterns and pixel on-off sequences are described below with reference to FIGS. 22-23B. In block 2002, an initial depth of focus may be set. In some embodiments, this may include configuring a depth of focus lens on the camera. The initial depth of focus may correspond to any depth that may be represented in the virtual target pattern. For example, the initial depth may correspond to a minimum or maximum depth associated with the virtual target pattern.

[0128] At block 2004, an image of the virtual target pattern is captured at a selected focal depth. In some embodiments, the image may comprise in-focus and out-of-focus portions. In some embodiments, the extent of the image may be focused on a particular virtual object associated with the virtual target pattern. In other embodiments, the image may correspond to an entire light field comprising multiple virtual objects. The image may comprise per-pixel focal depth information across the virtual target pattern.

[0129] At block 2006, a determination is made whether there are additional focal depths at which images should be taken. If it is determined that there are additional focal depths, then at block 2008, a new focal depth may be selected. In some embodiments, the number of focal depths may be based, at least in part, on the number of different depths that may be displayed by the display system (e.g., the number of depth planes 306 as illustrated in FIG. 3 or the number of waveguides in a waveguide assembly illustrated in FIG. 4). In some embodiments, if an image is focused on a particular virtual object, the range of focal depths may be based on one or more depths associated with the virtual object (e.g., a minimum and maximum depth associated with the virtual object).

[0130] If it is determined that there are no further focal depths at which images should be taken, then in block 2010, the captured images of the virtual target pattern can be analyzed to identify the depth, Z or lateral positions (X, Y) at which different regions of the target pattern are actually in focus. For example, each captured image of the virtual target pattern corresponding to a particular focal depth may contain portions that are in focus and portions that are out of focus. In some embodiments, each image may be divided into one or more regions that correspond to regions of the light field. Autofocus techniques may be used to determine the depth at which each region is in focus. In some embodiments, each region may correspond to a pixel.

[0131] In block 2012, a depth map may be created based, at least in part, on the measured focal depth (or lateral position). The depth map may comprise any type of data structure or visualization that maps light field location and focal depth. For example, the depth map may comprise depth information for one or more pixels of the captured image (e.g., Z-axis focal depth or Z-axis focal depth combined with measurements of lateral focal position (X and / or Y position)). In some embodiments, the pixels may correspond to a pixel cloud associated with a target virtual object. Thus, the depth map may define the actual perceived depth of the virtual object when viewed through the display optics.

[0132] In block 2014, the depth map may be compared to one or more desired focal depths, which correspond to depths at which one or more virtual objects are intended to be displayed. By examining differences between the actual perceived depth of the virtual object and the focal depth at which the virtual object is intended to appear, imperfections and / or deviations in the light field may be identified.

[0133] In block 2006, error correction may be performed based at least in part on a comparison between the depth map and the desired depth of focus. The error correction may compensate for imperfections in the light field display or content of the image projected from the display.

[0134] The process 2001 can be repeated for each waveguide in the waveguide assembly 405 of the light field display to map the imperfections of each of the waveguides. In some cases, there may be multiple waveguides corresponding to multiple depth planes and multiple waveguides corresponding to multiple colors (e.g., red (R), green (G), and blue (B)). For example, for some displays, for each depth plane, there may be three color planes, and thus a waveguide assembly with two depth planes may have 2×3=6 waveguides. The camera 1806 can be a combination of cameras sensitive to multiple colors, or cameras each sensitive to a subset of colors. The focal depth information obtained by the measurement system 1800 can be used to determine the spatial distribution of focus errors and the distribution of chromatic (color) imperfections of the display.

[0135] In some embodiments, instead of capturing multiple images at multiple different focal depths (e.g., using a digital camera with a swept focus), a light field camera can be used to capture the light field generated by the display 1802. The captured light field can be analyzed for focus and / or depth imperfections. By analyzing the vectors of light rays in the captured light field, focal depths for various regions can be determined. The identified focal depths can then be compared to one or more intended focal depths, and appropriate error corrections can be made (as in block 2016). For example, a vector measurement of the actual focal position (X, Y, Z) relative to the intended focal position (X0, Y0, Z0) can be determined as vector error = (X, Y, Z) - (X0, Y0, Z0) and used to characterize imperfections in the light field generated by the display.

[0136] (Exemplary Methods for Chroma Balancing a Display) As mentioned above, some implementations of full-color displays generate a tristimulus response on the viewer's retina by combining the red (R), green (G), and blue (B) wavelengths of light projected from the display. An ideal display would have spatially uniform luminance for these three color layers. However, real displays may have some amount of luminance variation across the field of view due to hardware imperfections. If this variation is different for different color layers, it will result in non-uniformity in color across the field of view (FOV) of the display (e.g., as illustrated in FIG. 11). This disclosure describes examples of methods that attempt to correct for chromatic variation and make the color uniform across the FOV. For example, the intensities of the individual color layers (e.g., R, G, and B) of the display can be adjusted such that the white point of the display can be substantially uniform across the FOV.

[0137] In some implementations, the light field measurement system described herein can be used to characterize the color balance of a display. For example, a digital color camera can take an image of a display (e.g., using a measurement system 1800 as illustrated in FIG. 18) from which the color response of the display can be determined for some or all of the pixels of the display. In many displays, there are three color layers (e.g., R, G, and B), however, the method is not limited to RGB or three-color displays. The method can be applied to any number of color layers (e.g., 2, 3, 4, 5, 6, or more) and any color choices (e.g., cyan, magenta, yellow, black).

[0138] Examples of measured color balance for a particular implementation of an RGB display are shown in Figure 14A (before color calibration) and Figure 15 (after color calibration). Figures 14A and 15 include plots (1400, 1500, respectively) of the distribution of R, G, and B intensities (vertical axis) across the pixels (horizontal axis) of the display. Figure 14B includes a plot 1408 of maximum color imbalance (vertical axis) with respect to the pixels of the display (horizontal axis), showing the average and the average ± maximum error prior to color correction.

[0139] As previously discussed, Figure 14A shows that an uncalibrated display has substantial chromatic non-uniformity across the pixels of the display. The red and blue responses are nearly identical, with the R and B intensities peaking toward the right of plot 1400. The green response is generally smaller than the R or B responses and decreases toward the right of plot 1400. Figure 15 shows that, after application of the color calibration described below, the calibrated display has a more uniform chromatic response across the pixels of the display.

[0140] The embodiments of the color balancing system and method described herein provide techniques for adjusting the intensities of at least some of the color layers in a multi-color display such that the white point of the display is substantially uniform across the FOV of the display. In various implementations, the display can be a light field display. For example, the display can have the ability to present color images at multiple depth planes to a viewer. The embodiments of the color balancing system and method can be applied to chroma-balanced the display 208 (FIG. 2), the display system 400 (FIGS. 4-6), and the display 2500 (FIGS. 25A, 25B, 26).

[0141] The human eye does not perceive light levels in a linear fashion. For example, compared to an ideal linear display, the human eye is more sensitive to changes in dark tones than similar changes in light tones, which allows the human visual system to operate over a wide range of brightness levels. Real-world displays also cannot provide a precisely linear brightness response. Furthermore, digital images are often encoded to represent tone levels that are more perceptually uniform. Human visual perception, display output, and image encoding are generally modeled to follow an approximate power-law relationship for brightness or color levels. For example, the output level is proportional to the input level raised to the power of gamma, i.e., V out ∝V in γ This nonlinear power law behavior is commonly referred to as gamma correction, gamma encoding, or simply gamma.

[0142] In some embodiments, if the luminance flatness of the individual color layers in a display is approximately uniform across the FOV of the display, color balancing can include scaling the intensities of the individual color layers to achieve uniform color balance across the display. A display may have suitable luminance flatness if the variation in luminance across the FOV of the display is, in various embodiments, less than 1%, less than 5%, less than 10%. Due to the gamma response of displays and human visual perception, this simple scaling may have certain disadvantages in some cases.

[0143] If the color layers of a display do not have substantial luminance flatness, color balancing may involve more than simply scaling the intensities of individual color layers. For example, color balancing may attempt to balance the white point at each pixel (or across groups of pixels) of the display independently. In some such implementations, color balancing across the FOV of the display can be achieved without also flattening the luminance across the FOV. Luminance flattening can be performed in addition to or as an alternative to color balancing.

[0144] The goal of saturation balancing a display is for a human viewer of the display to perceive a uniform color balance across the display's FOV. To measure and adjust the color balance of a display, a calibration camera (rather than the human eye) is used to record an image of the display output. The camera represents the human perception of the display output, and it can be assumed that if the camera image of the display is saturation balanced, then the human viewer's perception of the display will also be saturation balanced.

[0145] In some implementations, the following model is used for conversion between pixel values ​​for the color layers of the display and pixel values ​​for the colors measured by the calibration camera. In the following examples, there are three color layers, assumed to be R, G, and B. However, this is for illustration purposes and is not limiting. In other cases, any number and hues of color layers can be used with embodiments of the color balancing technique. Furthermore, appropriate scaling between the pixel size of the display and the camera can be taken into account prior to application of the model.

number

[0146] In equation (1), [Rd,Gd,Bd] represent intensity values ​​for an RGB image sent to a display. In many cases (e.g., standard RGB or sRGB), the intensity values ​​are between 0 and 255. Gamma1{} represents a first non-linear gamma function (with exponent γ1) that maps the display color levels to an intermediate color representation [R1 G1 B1]. Coupling() represents a function that maps color values ​​[R1 G1 B1] to a second intermediate color representation [R2 G2 B2]. The coupling() function can be a linear function, e.g., a 3×3 matrix (for three color layers). In other implementations, the coupling() function can be non-linear. Gamma2{} represents the second intermediate color representation [R2 G2 B2] to pixel intensities [Rc Gc Bc] aligned by the calibration camera.

[0147] In some implementations, the first and second gamma functions are global functions across the FOV of the display (e.g., the exponents γ1 and γ2 are constant across the FOV). Coupling() can be a local (pixel-dependent) function that varies from pixel to pixel across the FOV. The per-pixel color mapping provided by the Coupling() function allows for per-pixel color balancing.

[0148] To determine the functions Gamma1{}, Gamma2{}, and Coupling(), a series of one or more images of the display may be captured by a camera and analyzed by an analysis system programmed to perform an iterative optimization algorithm (e.g., hill climbing, local search, simplex method, genetic algorithm, etc.) to find a suitable fit for the gamma and coupling functions that provides a reasonable color balance for the display. The analysis system may use feedback during the iterative process by capturing additional images of the display as the analysis system searches for a suitable fit for the gamma and coupling functions. For example, the functions Gamma1{}, Gamma2{}, and Coupling() may be determined by iteratively adjusting these functions to improve or optimize the color balance of the camera images across the FOV of the display. The functions may be iteratively adjusted until the white point of the camera images acquired during the iterative process is substantially uniform across the FOV of the display. In various implementations, a substantially uniform white point distribution is associated with a variation in the white point across the FOV of less than 10%, less than 5%, or less than 1% of the white point value in the measured color system. For example, a color space provided by the International Commission on Illumination (CIE) may be used. In some implementations, a substantially uniform white point distribution may be associated with a variation in the white point less than a threshold amount based on the just noticeable difference (JND) of the color space. In some implementations, gamma transfer functions Gamma1{} and Gamma2{} are first calculated iteratively, and then a Coupling() function is calculated once the gamma functions (e.g., exponents γ1 and γ2) are calculated.

[0149] A production process for calibrating displays in a manufacturing environment can automatically characterize the displays as they are transported along a production line. For example, at a suitable point in the production process, a calibration camera and analysis system as described herein can perform an iterative analysis to determine the gamma transfer function and coupling function for a particular display, and store the resulting gamma and coupling function in a memory associated with the display. The display then has the ability to automatically perform color balancing.

[0150] During use of a particular display, once the gamma transfer functions, Gamma1{} and Gamma2{}, and the Coupling() function are known for a particular display, the appropriate display pixel values ​​[Rd Gd Bd] can be input into Equation (1) to achieve a chromatically balanced output. For example, the gamma exponents and Coupling() functions determined for a particular display can be stored in a memory accessible to the display and accessed to convert the input image pixel color values ​​and provide a chromatically balanced output from the display. In some implementations, the local processing and data module 224 of the wearable display system 200 can store the gamma transformation and coupling functions, and the processing module can utilize Equation (1) to output a real-time chromatically balanced image (FIG. 2). In other implementations, the controller 450 of the display system 400 can perform color balancing based on Equation (1) and the stored gamma and coupling functions (FIG. 4). In yet another implementation, the dynamic calibration processor 2610 of the dynamic calibration system 2600 can perform color balancing for the display 2500 using equation (1) and stored gamma and combination functions, as will be described below (FIG. 26).

[0151] An embodiment of the method 2700 or process flow 2805 for dynamically calibrating a display based on eye tracking, described with reference to Figures 27, 28, respectively, described in more detail below, can perform color balancing and other error correction / calibration functions. For example, the calibration accessed in block 2720 of the method 2700 can include gamma and combination functions, and in block 2730, chromatic imperfections of the display can be corrected by use of Equation (1) and the accessed gamma and combination functions. As another example, block 2880 of the process flow 2805 can access the gamma and combination functions and apply them during the calibration.

[0152] FIG. 21 is a flow chart illustrating an example of a method 2150 for calibrating a display. The display can be a light field display. The display can be a display 208 (FIG. 2), a display system 400 (FIGS. 4-6), and a display 2500 (FIGS. 25A, 25B, 26). The method 2150 can be performed by an analysis system (including a camera and an analysis program executed by computer hardware such as the metrology system 1800 shown in FIG. 18) as part of a production line of a manufacturing process for a display (e.g., as part of the process 2805 described with reference to FIG. 28). The method 2150 can be performed as part of a camera calibration, described with reference to block 1602 of the process 1600 described with reference to FIG. 16. In some implementations, the method 2700 applies equation (1) to determine an appropriate transformation between the display and the camera (assumed to represent the visual perception of a human viewer of the display). In block 2160, an image of the display is acquired by the camera. At block 2170, global transformation parameters for the transformation between the display and the camera are determined. The global transformation parameters may include parameters that do not vary across the FOV of the display (e.g., parameters that are not pixel-dependent). For example, the global transformation parameters may include Gamma1{} and Gamma2{} functions. In some cases, the method 2150 may return to block 2160 to obtain one or more additional images as part of an iterative feedback process for determining the global transformation parameters. After a suitable fit to the global transformation parameters is obtained, the method 2150 moves to block 2180 where local (e.g., pixel-dependent) transformation parameters are fitted to the camera image. For example, the local transformation parameters may include a Coupling() function (e.g., values ​​of this function at pixel locations across the FOV of the display). In some cases, the method 2150 may return to block 2160 to obtain one or more additional images as part of an iterative feedback process for determining the local transformation parameters.In some implementations, after acquiring the additional image at block 2160, the method 2150 may return to block 2180 to continue adapting the local transformation parameters rather than proceeding to block 2170 since the global transformation parameters were previously determined. After a suitable fit to the local transformation parameters has been adapted to the camera images, the method 2150 proceeds to block 2190 where the local and global transformation parameters are stored in a memory associated with the display (e.g., local data module 71). As mentioned above, at block 2720 of the method 2700 for dynamically calibrating a display, the local and global transformation parameters may be accessed as part of the calibration for the display, and at block 2730 the local and global transformation parameters and equation (1) may be applied to generate a chromatically balanced image from the display.

[0153] Although described with respect to the case of color balancing for a display, the present systems and methods are not so limited and can be applied to correct other chromatic (or spatial) imperfections of a display (e.g., any of the chromatic or spatial imperfections described above). For example, as described above, a display may exhibit luminance flatness variations, and embodiments of the disclosed analysis techniques can determine a luminance flatness calibration that corrects for the luminance flatness imperfections. Additionally or alternatively, a display may exhibit spatial imperfections, including in-plane translation, rotation, scaling, or warping errors and out-of-plane (e.g., depth of focus) errors. Embodiments of the disclosed analysis techniques can determine a calibration for some or all of such spatial errors.

[0154] (Example of display calibration using a calibration pattern) Imperfections in a display may cause virtual objects projected by the display to appear spatially or chromatically distorted. To correct for these distortions, the display may be calibrated by first measuring the distortions and then making any necessary error corrections (e.g., using a metrology system 1800 illustrated in FIG. 18). Display calibration may involve using the display to project a calibration pattern, e.g., a checkerboard pattern (e.g., as illustrated in FIG. 7), and capturing the resulting images with a camera. The captured images may then be processed to determine distortions at the feature points of the calibration pattern by quantifying the error between the expected position of the pattern feature points versus their measured positions. For displays with separate color layers (e.g., red (R), green (G), and blue (B) layers), this calibration may also correct for color registration and image quality.

[0155] FIG. 22 illustrates an example calibration system 2200 using a calibration pattern. A display 2202 can be configured to project a calibration pattern 2204 as a generated light field 2206 that can be captured using an imaging device such as a camera 2208. In some embodiments, the display 2202 comprises a stacked waveguide assembly (e.g., as illustrated in FIG. 4 or 6) or other type of light field display. In some embodiments, the camera 2208 (or the display 2202) is configured to be movable, which would allow the system 2200 to capture images of the light field 706 from different lateral positions, depths, or angles. In some embodiments, the calibration system 2200 may be similar to the metrology system 1800 of FIG. 18. For example, the display 2202, light field 2206, and camera 2208 may correspond to the display 1802, light field 1804, and camera 1806 of the metrology system 1800.

[0156] In this example, the calibration pattern 2204 comprises a checkerboard pattern, with different regions having different (e.g., alternating) optical properties, such as, for example, brightness (e.g., light or dark), color, hue, saturation, color, etc. The checkerboard pattern can be a regular pattern (e.g., as illustrated in FIG. 22) or an irregular pattern. The calibration pattern 2204 contains a number of feature points that can be used to measure the amount of distortion in an image captured by the camera 2208. For example, feature points of a checkerboard pattern include points on the boundaries and points at the corners between the checkerboards or at the centers of the checkerboards. The calibration pattern 2204 can be the same size as the display 2202 or smaller. The smaller calibration pattern can be shifted across the display 2202, and the camera 2208 may take multiple images of the calibration pattern 2204 as it shifts across the display as the system 2200 measures the distortion of the display 2202. In some implementations, the calibration pattern 2204 may be stochastically sampled according to a mathematically optimized sequence.

[0157] Due to errors in the display 2202 (e.g., imperfections in one or more waveguides or lenses), the light field 2206 may contain imperfections that cause virtual objects or patterns in the light field to appear distorted. This may result in deviations between the expected focus position (lateral or depth) of a feature point on the calibration pattern 2204 and its actual measured position in the image captured by the camera 2208. By comparing the actual measured positions of the feature points of the calibration pattern 2204 with the expected positions of these feature points, deviations caused by distortion can be identified and measured. In some implementations, the calibration pattern includes color information so that color errors of the display 2202 can be quantified by the system 2200. In some embodiments, a distortion map may be generated to be used for error correction of spatial or color errors of the display 2202 (e.g., as illustrated in FIG. 8).

[0158] In some implementations, each grid 2304 in the calibration pattern 2204 corresponds to a single pixel of the display 2202, which may allow for direct measurement of display imperfections per pixel. In other implementations, each grid 2304 corresponds to multiple pixels (e.g., an N×M grid of pixels, where at least one of N or M is greater than 1). In some such implementations, the coarse quality of the calibration pattern means that distortion information can be obtained at sample points and interpolated to obtain per-pixel distortion information. For example, in the checkerboard pattern illustrated in FIG. 23A, distortion information may be measured for pattern locations corresponding to feature points 2302, such as points on the boundaries, corners, or center of the grid. Distortion information for other points within the grid area 2304 of the pattern can be inferred or interpolated with measured distortion values ​​associated with nearby feature points 2302.

[0159] The grating projection capture procedure identifies feature points (e.g., grating edges) and quantifies the error of expected versus measured positions for distortion calibration. The feature points may be low density compared to the number of pixels in the display. For example, a high-resolution display may comprise millions of pixels (e.g., 2.1 million pixels for a 1920x1080 pixel resolution), while the number of gratings 804 in the calibration pattern may be substantially less (e.g., for 50x50, 100x100, 500x500 patterns). Thus, an embodiment of the system 2200 using a single projection capture approach results in sampled measurements that can be interpolated to estimate per-pixel distortion.

[0160] To obtain accurate per-pixel distortion information for a display, an embodiment of the system 2200 can automate the task of obtaining distortion information by implementing different or shifted calibration patterns. Different calibration patterns can be projected, or the same pattern can be gradually shifted so that the entire pixel space of the display 2202 is measured. Automated image projection and capture or different shifted calibration patterns allow pixel-precise mapping of the distortion of the display 2202.

[0161] By automatically repeating the plaid projection capture (but with the calibration pattern shifted, e.g., by 1-pixel), the system 2200 can obtain improved distortion information per pixel. For example, the camera 2208 can obtain an image of the pattern each time the pattern is shifted. With each repeated image capture, the feature points of the projected calibration pattern correspond to a different set of pixels. This shifting of the calibration pattern can be repeated until a dense sampling of the distortion field of the display is obtained. For example, the plaid can be projected and shifted through several positions that correspond to pixels of the plaid grid, allowing distortion information to be measured for each pixel of the display. In other implementations, the shift can differ from one pixel, e.g., 2, 3, 4, 5, 8, 16, or more pixels. The shift can differ for different directions of the display, e.g., the x-shift need not be the same as the y-shift.

[0162] While this disclosure primarily refers to checkerboard patterns for purposes of example, it should be understood that other types of patterns may also be used. For example, other geometric patterns may be used, random stochastic patterns may be used, or any other type of calibration or test pattern may be used. In some embodiments, a calibration pattern is used in which only a single pixel in the display is turned on at a time. FIG. 23B illustrates an exemplary single-pixel calibration pattern in which only a single pixel 2306 is turned on. From the captured images of each resulting frame, the pixel-by-pixel transfer function from the display device to the viewer scene can be quantified. After each image capture, the location of the displayed pixel 2306 may be shifted a set distance (e.g., a single pixel) across the display (e.g., in the direction indicated by arrow 2308). By automatically sweeping through the pixels of each display, a complete quantification of the quality of the display device can be obtained. In other implementations, the shift of the illuminated pixels can be a different number of pixels, for example, 2, 3, 4, 5, 8, 16 or more pixels, the shift can be different for different lateral directions on the display, or multiple pixels (rather than a single pixel as shown in FIG. 23B) can be illuminated in each image capture.

[0163] 24 is a flow chart of an example process 2400 for performing automated display calibration. Process 2400 can be performed, for example, as part of processes 2700 and 2805 described with reference to FIGS. 27 and 28. At block 2402, a calibration pattern is projected by the display. The calibration pattern may comprise any pattern having one or more feature points that can be generated by the display. In some embodiments, the calibration pattern comprises a checkerboard pattern. In other embodiments, other types of calibration patterns, such as single pixel patterns, may be used.

[0164] In block 2404, an image of the calibration pattern to be displayed is captured using a camera or other type of image capture device. If errors or imperfections are present in the light field generated by the display, portions of the calibration pattern displayed may be distorted, one or more feature points in the calibration pattern may appear in a different location than expected, and the brightness or color of the image may differ from that expected from the calibration pattern.

[0165] At block 2406, a distortion corresponding to an error between an expected location of a feature point of the calibration pattern and the captured location of the feature point is determined. For example, for a single pixel calibration pattern, the distortion information can be calculated for a particular pixel location of the pattern. For a checkerboard pattern, the distortion information can be calculated for pixels corresponding to checkerboard feature points (e.g., edges, corners, or centers of the checkerboard). In some implementations, a brightness or color error between a brightness or color of the calibration pattern and a corresponding brightness or color of a captured image of the calibration pattern is determined.

[0166] At block 2408, a determination is made whether there are any additional positions across the display at which the calibration pattern should be projected. If it is determined that there are additional positions, at block 2410, the calibration pattern may be shifted and projected at the new position, and an image of the calibration pattern may be captured (block 2404) and used to calculate the amount of distortion (block 2406). In some embodiments, the number of different positions for displaying the calibration pattern is based on the calibration pattern used. For example, for a single pixel calibration pattern, the number of positions may correspond to the total number of pixels displayable by the display. For a checkerboard pattern, the number of positions may be based on the number of pixels in each checkerboard.

[0167] Once the calibration pattern is displayed at all desired positions, in block 912, the calculated distortions may be aggregated and used to generate a distortion map comprising distortion information for each pixel (or group of pixels) of the display. The distortion information may include spatial distortion due to focus errors (e.g., in-plane errors or out-of-plane errors) or color errors (e.g., luminance or chromatic errors). In block 2414, error correction may be performed on the display using the calculated distortion map. For example, the distortion information (e.g., the distortion map) may be stored by the data modules 224, 232 of the wearable display system 200 illustrated in FIG. 2. The processing modules 224, 228 of the wearable display system 200 may use the distortion information to correct spatial or chromatic errors in the display 208 such that an image perceived by a wearer 204 of the display system 80 is at least partially compensated.

[0168] In some embodiments, the process 2400 illustrated in FIG. 24 can be performed for a light field display. As an example, the process 2400 can be performed for each waveguide in the waveguide assembly 405 of the light field display to calibrate each of the waveguides. In some cases, there may be multiple waveguides corresponding to multiple depth planes and multiple waveguides corresponding to multiple colors (e.g., red (R), green (G), and blue (B)). For example, for some displays where there are three color planes per depth plane, therefore, a waveguide assembly with two depth planes may have 2×3=6 waveguides. In addition, in addition to pixel position, color registration and quality may also be calibrated to correct for chromatic (color) imperfections of the display. For example, the camera 2208 can be a combination of multiple color-sensitive cameras or cameras each sensitive to a subset of colors, used to capture images of the light field 2208, where deviations between captured and expected colors or luminance values ​​of the projected pattern 2204 can be identified.

[0169] Exemplary Waveguide Displays Figure 25A is a top view diagrammatically illustrating an example of a display 2500 including a waveguide 2505, an in-coupling optical element 2507, a light redistribution element 2511, and an out-coupling optical element 2509. Figure 25B diagrammatically illustrates a cross-sectional view of the display 2500 depicted in Figure 25A along axis A-A'.

[0170] 4. For example, the waveguide 2505 may correspond to one of the waveguides 420, 422, 424, 426, 428, and the outcoupling optical element 2509 may correspond to the light extraction optical elements 460, 462, 464, 466, 468 of the display system 400.

[0171] The display 2500 is configured such that incoming incident light of different wavelengths, represented by light rays 2503i1, 2503i2, and 2503i3 (solid, dashed, and dashed double-dotted lines, respectively), is coupled into the waveguide 2505 by an internal coupling optical element 2507. The incoming incident light into the waveguide 2505 can be projected from an image injection device (such as one of the image injection devices 440, 442, 444, 446, 448 illustrated in FIG. 4). The internal coupling optical element 2507 can be configured to couple the wavelengths of incident light into the waveguide 2505 at an appropriate angle to aid in propagation through the waveguide 2505 via total internal reflection (TIR).

[0172] The light re-dispersion element 2511 can be disposed in the optical path in which the different wavelengths of light 2503i1, 2503i2, and 2503i3 propagate through the waveguide 2505. The light dispersive element 2511 can be configured to redirect a portion of the light from the internal coupling optical element 2507 towards the external coupling optical element 2509, thereby expanding the beam size interacting with the light along the propagation direction. Thus, the light dispersive element 2511 can be advantageous in expanding the exit pupil of the display device 2500. In some embodiments, the light dispersive element 2511 can thus function as an orthogonal pupil expansion element (OPE).

[0173] The out-coupling optical element 2509 can be configured to redirect the in-coupled light incident on the element 2509 from the xy plane of the waveguide 2505 at an appropriate angle (e.g., in the z-direction) and efficiency to promote proper overlay of light at different wavelengths and different depth planes so that a viewer may perceive a color image of good visual quality. The out-coupling optical element 2509 can have an optical power that provides divergence to the light exiting through the waveguide 2505 so that an image formed by the light exiting through the waveguide 2505 appears (to a viewer) to originate from a certain depth. The out-coupling optical element 2509 can expand the exit pupil of the display 2500 and can be referred to as an exit pupil expansion element (EPE), which directs the light to the viewer's eye.

[0174] The internal coupling optical element 2507, the external coupling optical element 1009, and the light dispersive element 2511 can include multiple gratings, such as, for example, analog surface relief gratings (ASR), binary surface relief structures (BSR), volume holographic optical elements (VHOE), digital surface relief structures, and / or volume phase holographic materials (e.g., holograms recorded in volume phase holographic materials), or switchable diffractive optical elements (e.g., polymer dispersed liquid crystal (PDLC) gratings). In various embodiments, the internal coupling optical element 2507 can include optical components including one or more optical prisms, or one or more diffractive and / or refractive elements. Various sets of diffractive or grating structures can be disposed on the waveguide by using fabrication methods such as injection compression molding, UV replication, or nanoimprinting of diffractive structures.

[0175] The in-coupling optical element 2507, the out-coupling optical element 1009, or the light dispersive element 2511 need not be a single element (e.g., as depicted diagrammatically in FIGS. 25A and 25B), and each such element can include multiple such elements. These elements can be disposed on one (or both) of the major surfaces 2505a, 2505b of the waveguide 2505. In the embodiment shown in FIGS. 25A and 25B, the in-coupling optical element 2507, the out-coupling optical element 2509, and the light dispersive element 2511 are disposed on the major surface 2505a of the waveguide 2505.

[0176] In some embodiments, one or more wavelength selective filters may be integrated with or disposed adjacent to the in-coupling optical element 2507, the out-coupling optical element 2509, or the light dispersive element 2511. The display 2500 illustrated in FIG. 25A includes a wavelength selective filter 2513 integrated into or on a surface of the waveguide 2505. The wavelength selective filter can be configured to filter out portions of light at one or more wavelengths that may propagate along various directions within the waveguide 2505. The wavelength selective filter can be an absorptive filter, such as a color band absorber.

[0177] (Example of dynamic calibration of AR or VR displays based on eye tracking) The display system can be calibrated (spatially and / or chromatically) to produce improved quality images. For some eyepiece displays (e.g., the display 208 shown in FIG. 2 or the stacked waveguide assembly 405 shown in FIG. 4 as used in the display 2500 described with reference to FIGS. 25A and 25B), this calibration is reasonably accurate for a nominally fixed eye position (e.g., a wearer looking straight through the display 208), but may be less accurate for other eye posture directions or positions. Thus, the calibration for the display may depend on the eye position or eye direction. If a calibration for only a single (e.g., reference) position is used, there may be errors that are not corrected when the wearer looks towards a different position (e.g., away from the reference position).

[0178] This disclosure also describes an example of dynamic calibration for the wearable display system 400 using eye tracking, where spatial and / or color calibration can change in response to changes in eye position (or eye orientation, in some cases). One such calibration provides a feed-forward calibration system that can result in maintaining high quality images over a wide range of eye movements. In some implementations, the calibration is performed in real time via a hardware processor (e.g., processing modules 224, 228 of the wearable display system 200 or controller 450 of the display system 400) without the addition of specialized hardware.

[0179] Calibration can compensate (or correct) for spatial and / or chromatic (color) errors within the field of view of the display. For example, spatial errors can include in-plane translation, rotation, scaling, or warping errors and out-of-plane (e.g., focal depth) errors. Chromatic errors can include luminance flatness or color uniformity errors for each color (e.g., R, G, and B) that may be displayed.

[0180] FIG. 26 diagrammatically illustrates an example of a dynamic calibration system 2600 for a display 2500, where calibrations can be applied to correct spatial and / or chromatic errors in a grid of reference positions (shown by dots 2602). The dynamic calibration system 2600 can include the display 2500, an inwardly facing imaging system such as an eye tracking camera 500, and a dynamic calibration processor 2610 (which reads and applies the calibrations). FIG. 26 diagrammatically illustrates another example of a display 2500, including an embodiment of the optical elements described with reference to FIGS. 25A and 25B. An outcoupling optical element 2509 directs light to a viewer's eye. As the viewer's eye is positioned at different positions 2602 relative to the outcoupling optical element 2509, the optical calibration of the display 2500 for that particular eye position (shown diagrammatically as dots 2602 in FIG. 26) may be different. For example, a calibration when the eye is positioned over position 2602a near the center of the external coupling optical element 2509 will be different from a calibration when the eye is positioned over position 2602b toward the upper left corner of the external coupling optical element 2509, and may similarly be different from a calibration for any of the other example positions 2602 on the optical element 2509.

[0181] As the user's eyes move relative to the display, the field of view (FOV) of the display remains approximately the same, but spatial and / or chromatic distortions within the display may change as the eyes translate relative to the display. Because the FOV includes the range of angles at which an image is presented to the user, the calibration data (at a given position relative to the display) can account for substantially all orientations or viewing angles of the eye. For example, when a user orients his or her eyes to different angles (while maintaining the same position relative to the display), the user may simply view different portions of the image with the same overall distortion. Thus, at any given position, as the eye orientation changes (e.g., the eye gaze direction changes), the eye's field of view generally remains within the FOV of the display, and the same calibration (for that given eye position) can be used for substantially all eye orientations. Thus, certain embodiments of the calibration system utilize position-dependent calibration, which is additionally not orientation-dependent.

[0182] It should be noted that the dots 2602, 2602a, 2602b are for reference only and do not form part of the outcoupling optical element 2509 or the display 2500. Additionally, while nine locations 2602 in a 3×3 grid are diagrammatically illustrated in FIG. 26, it should be understood that this is for illustrative purposes and that the number (or arrangement) of locations for calibration of the display 2500 may differ from that shown in FIG. 26. For example, in various implementations, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 16, 20, 25, 100, 256, or more calibration locations are used. The calibration locations can be arranged in 2×2, 3×3, 4×4, 5×5, 6×6, 7×7, 9×9, or other patterns or arrangements of other dimensional grids or locations.

[0183] Calibration for one or more positions on the display 2500 can be determined using a light field metrology system that measures the error in a calibration pattern (e.g., a checkerboard) projected from the display. The calibration can depend on the positions across the display at which the display is viewed. For example, the metrology system can sweep an eye proxy camera across the display (e.g., by translating the camera and the display relatively) that simulates a range of positions relative to the user's eyes. As the camera is swept across the display at each sample point 2602, the metrology system can build calibration (correction) values, thereby resulting in a set of calibrated eye-to-eye proxy positions. The calibration for a particular display may be stored as a look-up table (LUT) (or other efficient data structure) by the data modules 224, 228 of the wearable display system 200. In other implementations, an analytical model can be fitted to calibration data obtained from the metrology system, and the fitted analytical model can be stored by the wearable display system 200. Other modeling or data referencing methods can also be used to store the calibration. As mentioned above, the calibration may include spatial and / or chromatic corrections generated for each calibration position of the display (e.g., a 3×3 grid of calibration positions for the example display 2500 shown in FIG. 26). Note that in various implementations, to capture the calibration, the display is swept (translated) relative to a fixed camera, the camera is swept (translated) relative to a fixed display, or both the camera and display are swept (translated) relative to each other.

[0184] In implementations where the field of view (FOV) of the eye proxy camera exceeds the FOV of the display, placing a calibration camera at several discrete positions relative to the display (e.g., across the positions indicated by dots 2602) and taking one or more calibration images provides sufficient information about the imperfections of the display to determine the calibration for each discrete position. In some such implementations, the camera can capture the entire FOV of the display, and there may be no need to vary the orientation (e.g., facing direction) of the camera at each of the positions 2602. In other implementations, the orientation of the calibration camera (at each position 2602) may be varied to acquire additional images and map the FOV of the display (e.g., when the FOV of the camera is less than the FOV of the display).

[0185] The calibration positions can represent eye positions relative to the display 2500. For example, a wearer of the display 2500 will typically position the display such that the wearer's eye (in the xy plane) is approximately near the center of the outcoupling optical element 2509, e.g., the wearer's eye is positioned over position 2602a. A calibration for position 2602a (near the center of the optical element 2509) thus corresponds to light propagating approximately perpendicular to the display 2500 (e.g., substantially along the z-direction) and can be applied by the dynamic calibration processor 2610. If the wearer's eye moves upward and left over position 2602b (near the top left corner of the optical element 2509), a calibration for position 2602b can be applied by the processor 2510. The eye tracking camera 500 can image the eye (e.g., in real time), and a dynamic calibration processor 2510 can use the eye tracking data to determine the eye position, select an appropriate calibration (based on the determined eye position), and apply the calibration to the display. In some implementations, the eye position is determined from the corneal position and gaze direction. Additionally, in other embodiments, the eye orientation (e.g., gaze direction) may be determined and a calibration that is orientation dependent may be used.

[0186] An embodiment of the wearable display system 200 may include an embodiment of a dynamic calibration system 2600, which is diagrammatically illustrated in FIG. 8. For example, an eye tracking camera 500 (described with reference to FIG. 4) may be affixed to the frame of the wearable display system 200 and may dynamically measure the eye posture (e.g., eye position or eye orientation) of the wearer. Images from the camera 500 may be used by the dynamic calibration processor 2610 to determine the eye posture of the wearer in real time or near real time. When a dynamically calibrated system is in operation, the eye tracking camera may inform the dynamic calibration processor 2610 of the current eye posture of the wearer in real time or near real time. The dynamic calibration processor 2610 may fetch and apply the appropriate calibration (e.g., the appropriate calibration LUT stored in the data module 224, 228) based on the measured eye posture (e.g., position or orientation). If the wearer is not looking directly at the stored calibration position or if the wearer's eyes are not directly above the calibration position, the dynamic calibration processor can interpolate (or extrapolate) between calibrations for nearby calibration positions (e.g., including at least the calibration positions closest to the wearer's eye position) to determine the appropriate calibration to apply for the wearer's current eye position. Thus, the display system 200 (with the dynamic calibration system 2600) can correct imperfections (spatial or chromatic) in the display, thereby providing the wearer with a good quality color image. As described herein, in some cases, the calibration depends on the eye position relative to the display, rather than the eye orientation (e.g., gaze direction), but this is not a limitation.

[0187] The dynamic calibration processor 2610 can be implemented as software stored in memory (e.g., data modules 224, 228), and the software instructions can be executed by one or both of the processing modules 224, 228, or by the controller 450. Thus, continuous adjustment of the calibration can result in high quality images over a wide range of input movements of the wearer's eyes.

[0188] In some implementations, the calibrations are stored in a reduced number of calibration locations (e.g., a 2×2 or 3×3 grid) to reduce the amount of data storage. As described above, the dynamic calibration processor can interpolate or extrapolate to determine calibrations for eye poses that are not directly in the stored calibration locations.

[0189] In some embodiments, the wearable display system 200 uses a single eye tracking camera to measure the pose of the wearer's single eye, and the dynamic calibration processor 2610 infers the pose of the wearer's other eye relative to the display system 200 (as both eyes typically point toward the same direction). In other embodiments, the wearable display system 200 uses two eye tracking cameras (one per eye) to measure the pose of each eye independently. In some embodiments, a separate calibration is stored in the wearable system for each display (often there are two displays, one in front of each of the wearer's eyes, and therefore two calibrations are stored). In other embodiments, a single calibration (e.g., an average calibration) is stored and used for all of the displays in the wearable system 200.

[0190] An eye-tracking camera (or other type of inward-facing imaging system) can image the periocular region of the user's face. The periocular region can include the eye and the region around the eye. For example, the periocular region can include the eye (such as the eye socket) and the region around the eye. The region around the eye may include, for example, the eyebrows, part of the nose, cheeks, and forehead. The periocular region may have various features, such as eyebrow shape, eye corners, eyelid characteristics, etc. In some implementations, one or more of these features may be represented by key points, point clouds, or other types of mathematical representations. The wearable device can identify these features in the image and use these features to determine the relative position between the wearable display system and the user's face. In an embodiment, the wearable display system 200 may calculate the relative position separately for each eye. For example, when a wearable device has one or two eye cameras, each configured to image one eye of a user, the wearable device may calculate one relative position between the left eye and the wearable display system and another relative position between the right eye and the wearable display system. The wearable device may also track the relative positions for each eye separately. Because the relative position between the left eye and the wearable display system may be different from the relative position between the right eye and the wearable display system (such as when the wearable system is tilted to one side), adjustments to the rendering location of virtual objects may be different for the left and right eye displays.

[0191] The wearable display system may use neural network or visual feature techniques such as Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Fast and Rotational BRIEF with Orientation (ORB), Binary Robust Invariant Scalable Feature Points (BRISK), Fast Retinal Feature Points (FREAK), etc. to compute and track periocular features. In some embodiments, a particular facial feature may be tracked using a detector specifically designed for that particular facial feature. For example, periocular features such as the corners of the eyes, nose features, corners of the mouth, etc. may be identified and tracked separately using various algorithms. Tracking one or more of these periocular features separately may be advantageous because they are subject to substantial movement while a user is expressing themselves or speaking. Detectors associated with these periocular features may take into account a range of mobility. As an example, some facial features may be more likely to move in certain directions and are stable in others (e.g., eyebrows tend to move up and down but not side to side). The wearable system may statistically analyze the movement of the facial features. These statistics may be used to determine the likelihood that the facial features will move in a certain direction. In some embodiments, one or more facial features may be removed or detracked. For example, the wearable display system may ignore eye movement when tracking the position of the periocular region.

[0192] The wearable display system may also use visual simultaneous location and mapping (vSLAM) techniques such as sequential Bayesian estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, etc. to identify and track facial features. In some embodiments, the wearable device may be configured to enable depth perception. For example, the wearable system may build a dense map that encodes at least a portion of the face from data acquired by one or more cameras. Rather than a keypoint map, the dense map may comprise patches or regions of the face whose 3D shape is measured. The patches or regions may be used to calculate the location of the HMD relative to the user's face using techniques such as an iterative nearest neighbor algorithm or similar algorithm.

[0193] In some implementations, the images captured by the eye camera may be low-resolution images since high quality images are not required for the wearable display system 200 to track periocular features. Additionally or alternatively, the resolution of the images obtained from the eye imager may be downsampled relative to its original resolution or to a resolution used for other applications (e.g., eye tracking).

[0194] The wearable display system 200 can use various techniques to analyze images obtained by one or both eye cameras to determine the relative position between the display of the display system and the user. The relative position between the display and the user's eyes may be the normal rest position of the display system 200 relative to the user's face. The normal rest position of the display system 200 may be determined during an initial phase of the wearable system. For example, when a user first uses the wearable system, the wearable system may build a face model (e.g., a map of the user's face) and determine the normal rest position of the display relative to the user's eyes based on the face model.

[0195] While the user is using the wearable system 200, the wearable system can track the relative position between the display and the user using various techniques. For example, the wearable device can identify and track visual key points associated with periocular features. The wearable system can also match facial regions identified in the captured images against a dense map of the user's face and calculate the location of the display relative to the face.

[0196] Thus, various eye tracking or facial imaging techniques can be used to determine (statically or dynamically) the relative position between the user's eyes and the display of the display system. The display system 200 can then select and apply an appropriate spatial and / or chromatic calibration to the display based, at least in part, on the determined relative eye position, as described further herein.

[0197] FIG. 27 is a flow chart illustrating an example method 2700 for dynamically calibrating a display based on eye tracking. The method 2700 can be performed by the dynamic calibration system 2600. In block 2710, the user's eyes are tracked to determine the user's eye position relative to the display. For example, the camera 500 of the display system 2600 can determine the user's eye position. One or both eyes can be tracked. In block 2720, a calibration based on the determined eye position is accessed. In block 2730, the calibration is applied to the display to correct for spatial and / or chromatic imperfections in the display. For example, the dynamic calibration processor 2610 can apply corrections and adjust the nature of the light injected into the display's waveguide so that a desired light beam is output by the display. In some cases, the light may be injected at a slightly different color or position or orientation to adjust for display imperfections. For example, one or more of the RGB color values ​​in an input image to be projected by the display can be modified via a corresponding RGB calibration (based on the user's eye position) and the modified RGB values ​​can be sent to the display for projection. The net effect of an imperfect display causing the modified RGB values ​​to be projected is to generate a projected image that at least partially corrects the imperfections (spatial and / or chromatic) of the display. In other cases, actively controlled diffractive optical elements in the waveguide assembly can be adjusted by a dynamic calibration processor such that a light beam is projected from the display that at least partially corrects the imperfections in the display. In some implementations, the method 2700 is performed in real-time as a feedback loop such that the eye tracking camera 500 monitors the user's eyes and if a change in eye position is detected, a new calibration (for the new eye position) is used to calibrate the display. In some cases, if the change in eye position exceeds a threshold (e.g., a percentage of the spacing between the grid of calibration positions), the new calibration is applied. Some such implementations can advantageously provide a calibrated display for the user to view persistently.In some implementations, method 2700 may be performed occasionally (e.g., when a user places the display over the user's eye) or periodically (e.g., to correct for accidental slippage between the display and the user's eye).

[0198] FIG. 28 is a process flow diagram 2805 that diagrammatically illustrates an example of the interaction of a factory calibration system and a dynamic calibration system associated with a particular display. In this example, an eye proxy camera calibration system 2810 is used in a factory (manufacturing) setting to determine a position-dependent calibration for the display being manufactured. In block 2820, the process analyzes one or more calibration images for each particular display being manufactured and generates a calibration for each eye proxy position. In block 2830, the calibration is stored in a memory associated with the particular display such that each display has access to a calibration that is customized for that particular display during the manufacturing process. For example, the calibration may be stored as a look-up table (LUT) in the data module 224 of the display 208 or in the remote data repository 232. This portion of the process flow 2805 may be performed once per display during manufacturing to provide a customized calibration for each display.

[0199] In this example, each display system (e.g., an embodiment of the wearable display system 200) can perform real-time calibration using the calibration stored in block 2830. For example, the display's eye tracking system 2840 (which may include the eye tracking camera 500) may determine the position of the eye's cornea and the eye's gaze direction to determine the eye position. In block 2850, the display system (e.g., via the dynamic calibration processor 2610) may fetch the appropriate calibration from memory based on the determined eye position. In block 2860, the calibration is applied to the display (e.g., via the dynamic calibration processor 2610) to correct for spatial and / or chromatic errors of the display. In block 2870, the wearer can view the image projected by the calibrated display. As the position of the wearer's eye relative to the display changes, a process flow within the display system may update the calibration, for example, in real time.

[0200] Although an embodiment of the dynamic calibration system 2600 has been described in the context of a display in a wearable display system, this is not a limitation and the dynamic calibration system (e.g., eye tracking camera and dynamic calibration processor) can be used for any display (wearable or non-wearable) whose calibration is only good by approximating the nominal viewing position (e.g., perpendicular to the center of the display). For example, the dynamic calibration system can be used for flat panel displays, liquid crystal displays, light emitting diode displays, microelectromechanical system (MEMS) displays, etc.

[0201] (Additional aspects for image correction) In a first aspect, a computer-implemented method for performing image correction on a display is disclosed. The method includes the steps of calibrating the camera under the control of a display calibration system comprising computer hardware and a camera, capturing an image of a light field projected by the display using the camera, the light field being associated with a display layer of the display, generating a vector field comprising vectors corresponding to deviations between projected positions and expected positions of points of the display layer based at least in part on the captured image, using the generated vector field to perform at least one of decentering correction, collective rotation correction, collective scaling correction, or spatial mapping for the display, determining a plurality of luminance values ​​corresponding to a plurality of points on the display layer based at least in part on the captured image, and using the determined plurality of luminance values ​​to perform at least one of luminance flattening or color balancing for the display.

[0202] In a second aspect, the computer-implemented method of aspect 1, wherein the step of performing decentering correction includes the steps of identifying a center point of the projected display layer and determining a translation vector, the translation vector corresponding to a translation error between the identified center point and an expected center point position.

[0203] In a third aspect, the computer-implemented method of aspect 1 or aspect 2, wherein the step of performing the collective rotation includes the steps of identifying a center point of the projected display layer and determining an amount of rotation, the amount of rotation corresponding to a rotation of the projected display layer about the center point such that an amount of pixel error between the projected position and the expected position is minimized.

[0204] In a fourth aspect, the step of performing collective scaling includes the steps of identifying a center point of the projected display layer and determining an amount of scaling, the amount of scaling corresponding to scaling the projected display layer about the center point such that an amount of pixel error between the projected position and the expected position is minimized.

[0205] In a fifth aspect, the computer-implemented method of any one of aspects 1-4, wherein the step of performing spatial mapping includes identifying a nonlinear transformation and aligning projected positions of the display layer with expected positions.

[0206] In a sixth aspect, the computer-implemented method of any one of aspects 1-5, wherein the step of performing luminance equalization includes a step of determining a minimum luminance value of the plurality of luminance values ​​and a step of reducing all luminance values ​​of the plurality of luminance values ​​to the minimum luminance value.

[0207] In a seventh aspect, the computer-implemented method of any one of aspects 1-5, wherein the step of performing luminance flattening includes a step of determining a threshold luminance value and a step of reducing all luminance values ​​of a plurality of luminance values ​​that exceed the threshold luminance value to the threshold luminance value.

[0208] In an eighth aspect, the step of performing color balancing includes a step of identifying a color cluster associated with the display layer, the color cluster including at least one additional display layer; for each of a plurality of points on the display layer, a step of comparing a luminance value corresponding to the point on the display layer with a luminance value corresponding to the point on the additional display layer; and a step of reducing each luminance value of the plurality of luminance values ​​to a minimum luminance value associated with its corresponding point.

[0209] In a ninth aspect, the computer-implemented method of any one of aspects 1-8, wherein performing the collective rotation correction includes calculating a curl of the vector field.

[0210] In a tenth aspect, the computer-implemented method of any one of aspects 1-9, wherein performing the ensemble scaling correction comprises calculating the divergence of the vector field.

[0211] In an eleventh aspect, the computer-implemented method of any one of aspects 1-10, wherein the display comprises a light field display.

[0212] In a twelfth aspect, the computer-implemented method of aspect 11, wherein the light field display comprises stacked waveguide assemblies.

[0213] In a thirteenth aspect, the computer-implemented method of aspect 12, wherein the stacked waveguide assemblies each include two or more waveguides corresponding to two or more depth planes.

[0214] In a fourteenth aspect, the computer-implemented method of aspect 13, wherein each depth plane is associated with a red display layer, a green display layer, and a blue display layer.

[0215] In a fifteenth aspect, a method of calibrating a display is disclosed, the method including the steps of accessing an image of a calibration pattern projected by the display under control of a display calibration system comprising computer hardware, determining spatial distortion between expected positions of calibration points in the projected light field and their actual displayed positions in the image, analyzing the spatial distortion and determining a spatial calibration for the display, and storing the spatial calibration in a non-transitory memory associated with the display.

[0216] In a sixteenth aspect, the method of aspect 15, wherein the spatial calibration corrects one or more of in-plane spatial error or out-of-plane spatial error.

[0217] In a seventeenth aspect, the method of aspect 15 or aspect 16, wherein the spatial calibration corrects for one or more of translation error, rotation error, scaling error, or pixel warping.

[0218] In an 18th aspect, the method of any one of aspects 15-17 further includes the steps of determining color distortion from the image, analyzing the color distortion and determining a color calibration for the display, and storing the color calibration in a non-transitory memory associated with the display.

[0219] In a nineteenth aspect, the method of aspect 18, wherein the color calibration corrects for luminance flatness or color uniformity of the display.

[0220] (Additional aspects of optical metrology systems) In a twentieth aspect, an optical metrology system for measuring imperfections in a light field generated by a display is disclosed, the optical metrology system comprising: a display configured to project a target light field comprising a virtual object having an intended focus position; a camera configured to obtain images of the target light field; and a processor programmed with executable instructions to access one or more images corresponding to a portion of the light field, analyze the one or more images, identify a measured focus position corresponding to a position at which the virtual object will be in focus, and determine imperfections in the light field based, at least in part, on a comparison of the measured focus position and the intended focus position.

[0221] In a twenty-first aspect, the optical measurement system of aspect 20 is configured such that the display comprises a light field display.

[0222] In a twenty-second aspect, the optical measurement system of aspect 20 or aspect 21, wherein the display comprises a stack of waveguides configured to output light and project a virtual object at a particular depth plane.

[0223] In a twenty-third aspect, the optical measurement system of any one of aspects 20-22 is provided, wherein the camera is a digital camera having a small focal depth.

[0224] In a 24th aspect, the optical measurement system of aspect 23, wherein the camera has a focus and the system is configured to sweep the focus of the camera over a range of focuses to obtain one or more images.

[0225] In a 25th aspect, the optical measurement system of any one of aspects 20-22, wherein the camera comprises a light field camera.

[0226] In a 26th aspect, the optical measurement system of any one of aspects 20-25, wherein the virtual object has a checkerboard pattern, a geometric pattern, or a stochastic pattern.

[0227] In a twenty-seventh aspect, an optical measurement system as described in any one of aspects 20-26, wherein the display comprises a plurality of pixels and the target light field corresponds to a subset of less than all of the plurality of pixels that are illuminated.

[0228] In a twenty-eighth aspect, the optical measurement system of any one of aspects 20-27, wherein the measured focal position includes a focal depth.

[0229] In a twenty-ninth aspect, the optical measurement system of aspect 28, wherein the measured focal positions further include lateral focal positions.

[0230] In a 30th aspect, the optical metrology system of aspect 29, wherein the determined imperfection is based, at least in part, on an error vector between the intended focus position and the measured focus position.

[0231] In a thirty-first aspect, the optical metrology system of any one of aspects 20-30, wherein the determined imperfection comprises spatial imperfection.

[0232] In a thirty-second aspect, the optical measurement system of any one of aspects 20-31, wherein the determined imperfection comprises a chromatic imperfection.

[0233] In a thirty-third aspect, the optical measurement system of any one of aspects 20-32, wherein the processor is further programmed to determine an error correction for the display based, at least in part, on the determined imperfection.

[0234] In a thirty-fourth aspect, a method for measuring imperfections in a light field is disclosed, the method including the steps of accessing one or more images corresponding to a portion of a light field projected by a display, the portion of the light field having an intended focus position; analyzing the one or more images to identify a measured focus position corresponding to a position at which the portion of the light field is in focus; and determining imperfections in the light field based, at least in part, on a comparison of the measured focus position to the intended focus position.

[0235] In a thirty-fifth aspect, the method of aspect 34 includes a step of sweeping the focus of the camera to obtain one or more images.

[0236] In a thirty-sixth aspect, the method of aspect 34 includes a step of obtaining one or more images using a light field camera.

[0237] In a thirty-seventh aspect, a method according to any one of aspects 34-36 further comprises projecting a light field image having a checkerboard pattern.

[0238] In a thirty-eighth aspect, the method of any one of aspects thirty-four to thirty-seven further comprises determining an error correction for the light field based, at least in part, on the determined imperfections.

[0239] (Additional aspects of display calibration) In a thirty-ninth aspect, a calibration system for a display is provided comprising a camera configured to acquire an image of the display, and a hardware processor in communication with the camera, the hardware processor programmed to receive the image of the display, determine a calibration for the display, and store the calibration in a memory associated with the display.

[0240] In a fortieth aspect, the calibration system of aspect 39, wherein the calibration comprises spatial calibration to correct for spatial imperfections in the display.

[0241] In a forty-first aspect, the calibration system of aspect 39, wherein the calibration comprises color calibration for correcting color imperfections in the display.

[0242] In a forty-second aspect, the display has a plurality of pixels in a field of view, and to determine the calibration, the hardware processor is programmed to determine global transformation parameters that are independent of the pixels in the field of view of the display and to determine local transformation parameters that are dependent on the pixels in the field of view of the display, a calibration system described in any one of aspects 39-41.

[0243] In a 43rd aspect, the calibration system of aspect 42, wherein the global transformation parameters include one or more nonlinear gamma corrections.

[0244] In a 44th aspect, the calibration system of aspect 42 or aspect 43, wherein the local transformation comprises a linear function.

[0245] In a forty-fifth aspect, a calibration system described in any one of aspects 39-44, wherein to determine the calibration, the hardware processor is programmed to iteratively determine the calibration using feedback from images acquired by the camera.

[0246] In a 46th aspect, the calibration comprises color calibration, the display comprises a plurality of color levels that can provide a white point, and to determine the calibration, the hardware processor is programmed to adjust the intensity of the color levels so that the white point is substantially uniform across the field of view of the display. A calibration system as described in any one of aspects 39-45.

[0247] In aspect 47, the calibration system of aspect 46 is programmed to determine the calibration, wherein the hardware processor is programmed to determine a first gamma correction that maps color levels sent to the display to a first intermediate color representation, determine a pixel-dependent combination function that maps the first intermediate color representation to a second intermediate color representation, and determine a second gamma correction that maps the second intermediate color representation to color levels aligned by the camera.

[0248] In a forty-eighth aspect, the calibration system of aspect 47, wherein the hardware processor is programmed to determine the first gamma correction and the second gamma correction prior to determining the pixel-dependent combination function.

[0249] In a forty-ninth aspect, the calibration system of any one of aspects 39-48, wherein the display comprises a light field display.

[0250] In a 50th aspect, the calibration system of any one of aspects 39-49, wherein the display comprises a stackable waveguide assembly comprising a plurality of waveguides.

[0251] In a 51st aspect, the calibration system of any one of aspects 39-50, wherein the display is configured for a wearable display system.

[0252] In a fifty-second aspect, a method for calibrating a display is provided, the method including the steps of accessing a calibration for the display under control of a dynamic calibration system performed by computer hardware, determining, at least in part, based on the accessed calibration, a correction to apply to the display to correct for imperfections in the display, and applying the correction to the display.

[0253] In a 53rd aspect, the method of aspect 52, wherein the step of accessing a calibration includes a color calibration.

[0254] In a 54th aspect, the method of aspect 53, wherein the display comprises a plurality of pixels within a field of view, and the color calibration comprises a plurality of pixel-independent nonlinear gamma corrections and a pixel-dependent combination function.

[0255] In a 55th aspect, the method of any one of aspects 52-54, wherein the display comprises a light field display.

[0256] In a 56th aspect, a head-mounted display is provided comprising a display, a memory configured to store a calibration, and a hardware processor in communication with the non-transitory memory and programmed to perform a method according to any one of aspects 14-17.

[0257] (Additional aspects of the calibration pattern) In a fifty-seventh aspect, an optical system for calibrating a light field generated by a display includes a display configured to project a target light field comprising a calibration pattern containing feature points; a camera configured to obtain an image of the target light field; and a processor programmed with executable instructions to: for each of a plurality of locations, cause the display to project the calibration pattern onto a location within the plurality of locations, cause the camera to obtain an image of the projected calibration pattern, calculate a distortion of the feature points, the distortion corresponding to an error between an expected location of the feature point and a measured location of the feature point or an error between an expected luminance or color of the calibration pattern and a measured luminance or color of the calibration pattern, and in response to determining a next location within the plurality of locations, shift the calibration pattern to be displayed to the next location.

[0258] In aspect 58, the optical system of aspect 57, wherein the calibration pattern comprises a grating pattern.

[0259] In a fifty-ninth aspect, the optical system of aspect 57, wherein the number of the multiple positions corresponds to the number of pixels in a grid of the checkerboard pattern.

[0260] In a sixtieth aspect, the optical system of aspect 57, wherein the calibration pattern comprises a single pixel pattern.

[0261] In aspect 61, the optical system of aspect 60, wherein the number of multiple positions corresponds to the number of pixels to be displayed.

[0262] In aspect 62, the optical system of any one of aspects 57-61 is further programmed to generate a distortion map based, at least in part, on the calculated distortion corresponding to the multiple positions.

[0263] In aspect 63, the optical system of any one of aspects 57-62, wherein the processor is further programmed to determine an error correction for the display based, at least in part, on the calculated distortion corresponding to the multiple positions.

[0264] In a 64th aspect, the optical system of any one of aspects 57-63, wherein the display comprises separate red, green, and blue layers.

[0265] In a 65th aspect, the optical system of any one of aspects 57-64, wherein the display comprises a light field display.

[0266] In a 66th aspect, the optical system of aspect 65, wherein the light field display comprises stacked waveguide assemblies.

[0267] In a 67th aspect, the optical system of aspect 66, wherein the stacked waveguide assemblies each include two or more waveguides corresponding to two or more depth planes.

[0268] In a 68th aspect, the optical system of any one of aspects 57-67, wherein the calculated distortion further comprises luminance distortion or chromatic distortion.

[0269] In a sixty-ninth aspect, a method is provided for calibrating a light field generated by a display, the method including, for each of a plurality of locations, causing the display to project a calibration pattern onto a location within the plurality of locations, causing a camera to obtain an image of the projected calibration pattern, calculating a distortion of a feature point, the distortion corresponding to an error between an expected location of the feature point and a measured location of the feature point or an error between an expected luminance or color of the feature point and a measured luminance or color of the feature point, and, in response to determining a next location within the plurality of locations, shifting the calibration pattern to be displayed to the next location.

[0270] In a seventieth aspect, the method of aspect 69 is characterized in that the calibration pattern is a checkerboard pattern.

[0271] In a seventy-first aspect, the method of aspect 70, wherein the number of the multiple positions corresponds to the number of pixels in a grid of the checkerboard pattern.

[0272] In a seventy-second aspect, the method of aspect 69, wherein the calibration pattern comprises a single pixel pattern, a stochastic pattern, or a geometric pattern.

[0273] In a 73rd aspect, the method of aspect 72, wherein the number of the multiple positions corresponds to the number of pixels to be displayed.

[0274] In a 74th aspect, the method of any one of aspects 69-73 further comprises generating a distortion map based, at least in part, on the calculated distortions corresponding to the multiple positions.

[0275] In a 75th aspect, the method of any one of aspects 69-74, further comprising determining an error correction for the display based, at least in part, on the calculated distortion corresponding to the multiple positions.

[0276] In a 76th aspect, the optical system of any one of aspects 69-75, wherein the display comprises separate red, green, and blue layers.

[0277] In a 77th aspect, the optical system of any one of aspects 69-76, wherein the display comprises a light field display.

[0278] In a 78th aspect, the optical system of aspect 77, wherein the light field display comprises stacked waveguide assemblies.

[0279] In a 79th aspect, the optical system of aspect 78, wherein the stacked waveguide assemblies each include two or more waveguides corresponding to two or more depth planes.

[0280] In an 80th aspect, the optical system of any one of aspects 69-79, wherein the calculated distortion further comprises luminance distortion or chromatic distortion.

[0281] (Additional aspects of performing dynamic calibration) In an eighty-first aspect, a display system is provided comprising: an eye tracking camera, a display, a non-transitory data store configured to store a plurality of calibrations for the display, each calibration in the plurality of calibrations being associated with a calibration position relative to the display, and a hardware processor in communication with the eye tracking camera, the display, and the non-transitory data store, the hardware processor being programmed to: determine an eye position relative to the display for a user of the display, access one or more of the plurality of calibrations based at least in part on the determined eye position, determine a correction to apply to the display to correct for imperfections in the display based at least in part on the one or more of the plurality of calibrations, and apply the correction to the display.

[0282] In an 82nd aspect, the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9 or more.

[0283] In aspect 83, the display system of aspect 81 or aspect 82, wherein the calibration positions are distributed across the display in a grid.

[0284] In an 84th aspect, the display system of aspect 83, wherein the grid comprises a 2x2, 3x3, 5x5, or 9x9 grid.

[0285] In aspect 85, a display system described in any one of aspects 81-84, wherein one or more of the plurality of calibrations comprises a calibration associated with a calibration position closest to the eye position.

[0286] In an 86th aspect, a display system described in any one of aspects 81-85, wherein to determine the correction, the hardware processor is programmed to interpolate or extrapolate between one or more of the multiple calibrations.

[0287] In an 87th aspect, a display system described in any one of aspects 81-86, wherein each calibration in the plurality of calibrations corrects spatial imperfections of the display, chromatic imperfections of the display, or both spatial imperfections and chromatic imperfections.

[0288] In an 88th aspect, the display system of any one of aspects 81-87 comprises a light field display.

[0289] In an 89th aspect, the display system of any one of aspects 81-88 comprises a stackable waveguide assembly comprising a plurality of waveguides.

[0290] In a 90th aspect, the display system of any one of aspects 81-89 is configured as a wearable display system.

[0291] In a 91st aspect, a head-mounted display is provided comprising a display system described in any one of aspects 81-90.

[0292] In a ninety-second aspect, there is provided a method for calibrating a display, the method including the steps of determining, under control of a dynamic calibration system performed by computer hardware, an eye position relative to a user of the display, accessing a calibration for the display based at least in part on the determined eye position, the calibration being associated with a calibration position proximate the determined eye position, determining a correction to apply to the display to correct for imperfections in the display, based at least in part on the accessed calibration, and applying the correction to the display.

[0293] In a 93rd aspect, the method of aspect 92, wherein the step of accessing a calibration includes a step of selecting one or more calibrations from a plurality of calibrations, each calibration being associated with a different calibration position relative to the display.

[0294] In a 94th aspect, the method of aspect 93, wherein the calibration positions are arranged in a grid across the display.

[0295] In a 95th aspect, the calibration comprises any of aspects 92-94, correcting for spatial imperfections of the display, chromatic imperfections of the display, or both spatial imperfections and chromatic imperfections. 2. The method according to claim 1.

[0296] In aspect 96, a method according to any one of aspects 92-95, wherein the step of determining the correction includes a step of interpolating or extrapolating between one or more calibrations associated with calibration positions in the vicinity of the eye posture.

[0297] In a 97th aspect, the method of any one of aspects 92-96, wherein the display comprises a light field display.

[0298] In a 98th aspect, a head-mounted display is provided comprising an eye tracking system and a hardware processor programmed to perform a method according to any one of aspects 92-97.

[0299] (Additional aspects of optical metrology systems) In a ninety-ninth aspect, an optical metrology system for measuring imperfections in a light field generated by a display is provided, the optical metrology system comprising: a display configured to project a target light field comprising a virtual object having an intended focus position; a camera configured to obtain images of the target light field; and a hardware processor programmed with executable instructions to access one or more images corresponding to a portion of the light field, analyze the one or more images, identify a measured focus position corresponding to a position at which the virtual object will be in focus, and determine imperfections in the light field based, at least in part, on a comparison of the measured focus position to the intended focus position.

[0300] In aspect 100, the display comprises a stack of waveguides configured to output light and project virtual objects into at least one depth plane, the optical measurement system of aspect 99.

[0301] In aspect 101, the optical measurement system of any of aspects 99-100 includes a digital camera having a small focal depth.

[0302] In aspect 102, the optical measurement system of aspect 101, wherein the camera has a focus and the system is configured to sweep the focus of the camera over a range of focuses to obtain one or more images.

[0303] In aspect 103, the optical measurement system of any of aspects 99-102, wherein the camera comprises a light field camera.

[0304] In aspect 104, an optical measurement system described in any of aspects 99-103, wherein the virtual object has a checkerboard pattern, a geometric pattern, or a stochastic pattern.

[0305] In aspect 105, an optical measurement system as described in any of aspects 99-104, wherein the display comprises a plurality of pixels and the target light field corresponds to a subset of less than all of the plurality of pixels that are illuminated.

[0306] In aspect 106, the optical measurement system of any of aspects 99-105, wherein the measured focal position includes a focal depth.

[0307] In aspect 107, the optical measurement system of aspect 106, wherein the measured focal position further includes a lateral focal position.

[0308] In aspect 108, an optical metrology system as described in any of aspects 99-107, wherein the determined imperfection is based, at least in part, on an error vector between the intended focus position and the measured focus position.

[0309] In aspect 109, the optical metrology system of any of aspects 99-108, wherein the hardware processor is further programmed to determine an error correction for the display based, at least in part, on the determined imperfections.

[0310] In aspect 110, the optical measurement system of any of aspects 99-109, wherein the hardware processor is further programmed to apply display / camera pixel mapping and convert display pixel values ​​to camera pixel values.

[0311] In aspect 111, the optical measurement system of aspect 110, wherein the display / camera pixel mapping comprises a first gamma correction that maps color levels of the display to a first intermediate color representation, a pixel-dependent combination function that maps the first intermediate color representation to a second intermediate color representation, and a second gamma correction that maps the second intermediate color representation to color levels aligned by the camera.

[0312] In a 112th aspect, the optical metrology system of any of aspects 99-111, wherein the determined imperfection comprises spatial imperfection.

[0313] In aspect 113, the optical metrology system of aspect 112, wherein the spatial imperfections comprise one or more of in-plane translation, rotation, scaling, or warping errors, or out-of-plane or depth of focus errors.

[0314] In aspect 114, the optical measurement system of any of aspects 99-113, wherein the determined imperfection comprises a chromatic imperfection.

[0315] In aspect 115, the optical measurement system of aspect 114, wherein the chromatic imperfections comprise one or more of luminance flatness or chromatic uniformity errors associated with the colors displayable by the display.

[0316] In a 116th aspect, an optical metrology system for performing image correction on a display is provided, the system comprising: a camera configured to capture an image of a light field projected by the display, the light field being associated with a display layer of the display; and a hardware processor programmed with executable instructions to: generate a vector field based at least in part on the image captured by the camera, the vector field comprising vectors corresponding to deviations between projected positions and expected positions of points of the display layer; calculate at least one of a decentering correction, a collective rotation correction, a collective scaling correction, or a spatial mapping for the display based at least in part on the vector field; calculate luminance values ​​corresponding to a plurality of points on the display layer based at least in part on the image captured by the camera; and calculate a luminance flattening correction or a color balance correction for the display based at least in part on the determined luminance values.

[0317] In aspect 117, the optical measurement system of aspect 116, wherein the display layer of the display comprises a color layer or a depth layer.

[0318] In aspect 118, the optical measurement system of any of aspects 116-117, wherein the camera comprises a light field camera or a digital camera having a small focal depth.

[0319] In aspect 119, an optical measurement system as described in any of aspects 116-118, wherein to calculate the decentering correction, the hardware processor is programmed to determine a translation vector corresponding to a translation error between an identified center point of the projected display layer and an expected center point position.

[0320] In aspect 120, an optical measurement system as described in any of aspects 116-119, wherein to calculate the collective rotation correction, the hardware processor is programmed to determine an amount of rotation corresponding to a rotation of the projected display layer about a center point such that the amount of pixel error between the projected position and the expected position is reduced or minimized.

[0321] In aspect 121, an optical measurement system described in any of aspects 116-120, wherein in order to calculate the collective rotation correction, the hardware processor is programmed to calculate the curl of the vector field.

[0322] In aspect 122, an optical measurement system as described in any of aspects 116-121, wherein to calculate the collective scaling correction, the hardware processor is programmed to determine a scaling amount corresponding to scaling of the projected display layer around a center point such that the amount of pixel error between the projected position and the expected position is reduced or minimized.

[0323] In aspect 123, an optical measurement system described in any of aspects 116-122, wherein in order to calculate the ensemble scaling correction, the hardware processor is programmed to calculate the divergence of the vector field.

[0324] In aspect 124, an optical measurement system described in any of aspects 116-123, wherein to calculate the spatial mapping, the hardware processor is programmed to determine a nonlinear transformation to align the projected position of the display layer with the expected position.

[0325] In aspect 125, an optical measurement system as described in any of aspects 116-124, wherein to calculate the brightness flattening correction, the hardware processor is programmed to determine a threshold brightness value and calculate an amount by which each brightness value above the threshold brightness value is reduced to the threshold brightness value.

[0326] In aspect 126, an optical measurement system as described in any of aspects 116-125, wherein to calculate the color balance correction, the hardware processor identifies a color cluster associated with the display layer, the color cluster constituting at least one additional display layer, and is programmed to, for each point of the display layer, compare a luminance value corresponding to the point on the display layer with a luminance value corresponding to a point on the additional display layer, and calculate an amount by which each luminance value is reduced to a minimum luminance value associated with its corresponding point.

[0327] (Additional aspects of dynamic display calibration) In a 127th aspect, a display system is provided that includes an eye tracking camera, a display, a non-transitory data store configured to store a plurality of calibrations for the display, each calibration in the plurality of calibrations being associated with a calibration position relative to the display, and a hardware processor in communication with the eye tracking camera, the display, and the non-transitory data store, the hardware processor being programmed to: determine an eye position of a user of the display relative to the display based on information from the eye tracking camera, access one or more of the plurality of calibrations based at least in part on the determined eye position, calculate a correction to apply to the display to correct for imperfections in the display based at least in part on the one or more of the plurality of calibrations, and apply the correction to the display.

[0328] In aspect 128, the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more.

[0329] In a 129th aspect, a display system described in any of aspects 127-128, wherein the calibration positions are distributed across the display in a grid.

[0330] In aspect 130, the display system of aspect 129, wherein the grid comprises a 2x2, 3x3, 5x5, or 9x9 grid.

[0331] In aspect 131, a display system described in any of aspects 127-130, wherein one or more of the plurality of calibrations comprises a calibration associated with a calibration position closest to the eye position.

[0332] In aspect 132, a display system as described in any of aspects 127-131, wherein to calculate the correction, the hardware processor is programmed to interpolate or extrapolate between one or more of the multiple calibrations based, at least in part, on the calibration position of the one or more of the multiple calibrations and the determined eye position.

[0333] In aspect 133, a display system described in any of aspects 127-132, wherein the display includes a first display associated with a first eye of a user and a second display associated with a second eye of the user, and the hardware processor is programmed to determine an eye position of the user relative to the first display and apply the determined eye position to calculate a correction for the second display.

[0334] In aspect 134, the display system of any of aspects 127-133, wherein the display includes a first display associated with a first eye of a user and a second display associated with a second eye of the user, and at least some of the multiple calibrations represent average calibrations for the first display and the second display.

[0335] In aspect 135, the display system of any of aspects 127-134 comprises a light field display.

[0336] In aspect 136, the display system of any of aspects 127-135 comprises a stackable waveguide assembly comprising a plurality of waveguides.

[0337] In aspect 137, the display system of any of aspects 127-136 is configured as a head-mounted wearable display system.

[0338] In aspect 138, a display system described in any one of aspects 127-137, wherein each calibration in the plurality of calibrations corrects spatial imperfections of the display, chromatic imperfections of the display, or both spatial imperfections and chromatic imperfections.

[0339] In aspect 139, the display system of aspect 138, wherein the spatial imperfections comprise one or more of in-plane translation, rotation, scaling, or warping errors, or out-of-plane or depth of focus errors.

[0340] In aspect 140, the display system of aspect 138, wherein the chromatic imperfections comprise one or more of luminance flatness or chromatic uniformity errors associated with colors displayable by the display.

[0341] In a one hundred forty-first aspect, there is provided a method for calibrating a display, the method including the steps of determining, under control of a dynamic calibration system performed by computer hardware, an eye position relative to a user of the display, accessing a calibration for the display based at least in part on the determined eye position, the calibration being selected based on an associated calibration position and the determined eye position, calculating a correction to apply to the display to correct for imperfections in the display, based at least in part on the accessed calibration, and applying the correction to the display.

[0342] In aspect 142, the method of aspect 141, wherein the step of accessing a calibration includes selecting one or more calibrations from a plurality of calibrations, each calibration being associated with a different calibration position relative to the display.

[0343] In aspect 143, the method of aspect 142, wherein the calibration positions are arranged in a grid across the display.

[0344] In aspect 144, the method of any of aspects 142-143, wherein the step of calculating the correction includes a step of interpolating or extrapolating between one or more of the multiple calibrations based on an associated calibration position of the one or more of the multiple calibrations and the determined eye position.

[0345] In aspect 145, a method described in any of aspects 141-144, further comprising accessing an image of the user's eye on a display and determining eye position based, at least in part, on the image of the eye.

[0346] In aspect 146, the method of any one of aspects 141-145, wherein the step of calculating the calibration includes a step of calibrating spatial imperfections of the display, chromatic imperfections of the display, or both spatial imperfections and chromatic imperfections.

[0347] In a one hundred forty-seventh aspect, a wearable display system is provided, comprising: an inwardly-facing imaging system; a display; a non-transitory data storage device configured to store a plurality of calibrations for the display, each calibration in the plurality of calibrations being associated with a calibration position relative to the display; and a hardware processor in communication with the inwardly-facing imaging system, the display, and the non-transitory data storage device, the hardware processor being programmed to: determine an eye position of a user of the display relative to the display using the inwardly-facing imaging system; and based at least in part on the determined eye position and one or more of the plurality of calibrations, calculate a correction to apply to the display to correct for one or more of spatial imperfections in the display or chromatic imperfections in the display, and apply the correction to the display.

[0348] In aspect 148, the hardware processor is programmed to monitor changes in eye position and apply corrections via a feedback loop, a wearable display system as described in aspect 147.

[0349] In aspect 149, a wearable display system as described in any of aspects 147-148, wherein the hardware processor is programmed to determine a change in eye position relative to a previous eye position and calculate a correction if the change exceeds a threshold.

[0350] In aspect 150, the wearable display system of any one of aspects 147-149, wherein the spatial imperfections comprise one or more of in-plane translation, rotation, scaling, or warping errors, or out-of-plane or depth of focus errors.

[0351] In aspect 151, a wearable display system as described in any one of aspects 147-150, wherein the chromatic imperfections comprise one or more of luminance flatness or chromatic uniformity errors associated with colors displayable by the display.

[0352] (Conclusion) Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and may be fully or partially automated thereby. For example, the computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer programmed with specific computer instructions, special-purpose circuits, etc. The code modules may be written in a programming language that may be compiled and linked into an executable program, installed in a dynamic link library, or interpreted. In some implementations, certain operations and methods may be performed by circuitry specific to a given function.

[0353] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real-time. For example, a video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0354] The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage devices, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, and / or the like. The methods and modules (or data) may also be transmitted as data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) generated over a variety of computer readable transmission media, including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or communicated via a computer readable transmission medium.

[0355] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. The various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other sequences as appropriate, e.g., serially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product, or packaged into multiple computer products. Many implementation variations are possible.

[0356] The process, method, and system may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0357] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the present disclosure, the principles, and novel features disclosed herein.

[0358] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination and may be primarily claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be subject to subcombinations or variations of the subcombination. No single feature or group of features is necessary or essential to every embodiment.

[0359] In particular, conditional statements used herein, such as "can," "could," "might," "may," "eg," and the like, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to suggest that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps should be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprise," "include," "have," and the like, are synonymous and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the appended claims, unless otherwise specified, should be interpreted to mean "one or more" or "at least one."

[0360] As used herein, a phrase referring to a list of items "at least one of" refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitive phrases such as "at least one of X, Y, and Z" are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such transitive phrases are generally not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0361] Similarly, although operations may be depicted in the figures in a particular order, it should be appreciated that such operations need not be performed in the particular order depicted, or in sequential order, or that all of the depicted operations need not be performed to achieve desirable results. Additionally, the figures may diagrammatically depict one or more exemplary processes in the form of a flow chart. However, other operations not depicted may also be incorporated within the diagrammatically depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. In some circumstances, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Claims

1. 1. A display system, comprising: An eye-tracking camera; A display and a non-transitory data storage device configured to store a plurality of calibration parameters for the display, each calibration parameter in the plurality of calibration parameters being associated with a calibration position relative to the display, each calibration parameter in the plurality of calibration parameters being further associated with a vector field including vectors corresponding to deviations between expected and actual positions of points of a displayed image, or a plurality of non-linear gamma transfer functions and combination functions, or both, the combination functions mapping between intermediate color representations of the display; a hardware processor in communication with the eye tracking camera, the display, and the non-transitory data storage device; The hardware processor includes: determining an eye position of a user of the display relative to the display based on information from the eye tracking camera; accessing one or more of the plurality of calibration parameters based at least in part on the determined eye position; calculating corrections to apply to the display to correct spatial imperfections and chromatic imperfections of the display based at least in part on the one or more of the plurality of calibration parameters, the spatial imperfections of the display comprising one or more of an in-plane translation, rotation, scaling, or warping error between an expected position of an image point compared to a displayed position of the image point in the display, and the chromatic imperfections of the display comprising one or more of a luminance flatness or chromatic uniformity error associated with colors displayable by the display, the corrections for the spatial imperfections calculated based on the plurality of calibration parameters associated with the vector field, and the corrections for the chromatic imperfections calculated based on the plurality of calibration parameters associated with the plurality of nonlinear gamma transfer functions and the combination function; applying said correction to said display; a display system programmed to:

2. 10. The display system of claim 1, wherein the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more.

3. The display system of claim 1 , wherein the calibration positions are distributed across the display in a grid.

4. The display system of claim 3 , wherein the grid comprises a 2×2, 3×3, 5×5, or 9×9 grid.

5. The display system of claim 1 , wherein the one or more of the plurality of calibration parameters comprises a calibration parameter associated with a calibration position closest to the eye position.

6. 2. The display system of claim 1, wherein to calculate the correction, the hardware processor is programmed to interpolate or extrapolate between the one or more of the plurality of calibration parameters based, at least in part, on the calibration position of the one or more of the plurality of calibration parameters and the determined eye position.

7. 2. The display system of claim 1, wherein the display comprises a first display associated with a first eye of the user and a second display associated with a second eye of the user, and the hardware processor is programmed to determine an eye position of the user relative to the first display and apply the determined eye position to calculate a correction for the second display.

8. 2. The display system of claim 1, wherein the display comprises a first display associated with a first eye of the user and a second display associated with a second eye of the user, and at least some of the plurality of calibration parameters represent average calibration parameters for the first display and the second display.

9. The display system of claim 1 , wherein the display comprises a light field display.

10. The display system of claim 1 , wherein the display comprises a stackable waveguide assembly comprising a plurality of waveguides.

11. The display system of claim 1 , wherein the display is configured as a head-mounted wearable display system.

12. 1. A method for calibrating a display, the method comprising: Under the control of a dynamic calibration system implemented by computer hardware, determining an eye position of a user of the display; accessing calibration parameters for the display based at least in part on the determined eye position, the calibration parameters being selected based on the determined eye position and a calibration position associated with a selected calibration parameter, the calibration parameters being associated with a vector field including vectors corresponding to deviations between expected and actual positions of points of a displayed image, or a plurality of nonlinear gamma transfer functions and a combination function, or both, the combination function mapping between intermediate color representations of the display; calculating, at least in part based on the accessed calibration parameters, corrections to apply to the display to correct spatial imperfections and chromatic imperfections of the display, the spatial imperfections of the display comprising one or more of an in-plane translation, rotation, scaling, or warping error between an expected position of an image point compared to a displayed position of the image point in the display, and the chromatic imperfections of the display comprising one or more of a luminance flatness or chromatic uniformity error associated with colors displayable by the display, the corrections for the spatial imperfections calculated based on a plurality of calibration parameters associated with the vector field, and the corrections for the chromatic imperfections calculated based on a plurality of calibration parameters associated with the plurality of nonlinear gamma transfer functions and the combination function; applying said correction to said display; The method of claim 1,

13. The method of claim 12 , wherein accessing the calibration parameters includes selecting one or more calibration parameters from a plurality of calibration parameters, each calibration parameter being associated with a different calibration position relative to the display.

14. The method of claim 13 , wherein the calibration positions are arranged in a grid across the display.

15. 14. The method of claim 13, wherein calculating the correction comprises interpolating or extrapolating between the one or more of the plurality of calibration parameters based on the associated calibration position of the one or more of the plurality of calibration parameters and the determined eye position.

16. The method of claim 12 , further comprising: accessing an image of the user's eye on the display; and determining the eye position based, at least in part, on the image of the eye.

Images