Waveguide pose estimation via retroreflection

Abstract

Techniques are disclosed for estimating a pose of a waveguide in a WHUD device based on interactions of display light with a retroreflective optical element and one or more photoreceptive sensors. Display light representing the virtual image is directed into a waveguide via an incoupler that is optically coupled to the waveguide, while a portion of the display light is reflected towards at least one photoreceptive sensor via one or more retroreflective optical elements, such as a cross-dichroic prism. A pose of the waveguide is estimated based on interactions of the reflected portion of display light with the at least one photoreceptive sensor.

Description

WAVEGUIDE POSE ESTIMATION VIA RETROREFLECTIONBACKGROUND

[0001] The present disclosure relates generally to augmented reality (AR) eyewear, which fuses a view of the real world with a heads up display overlay. Wearable heads-up displays (WHUDs) are wearable electronic devices that use optical combiners to combine real world and virtual images. The optical combiner may be integrated with one or more lenses to provide a combiner lens that may be fitted into a support frame of a WHUD. In operation, the combiner lens provides a virtual display that is viewable by a user when the WHUD is worn on the head of the user. One class of optical combiner uses a waveguide (also termed a lightguide) to transfer light. In general, light from a projector of the WHUD enters the waveguide of the combiner through an incoupler, propagates along the waveguide via total internal reflection (TIR), and exits the waveguide through an outcoupler. If the pupil of the eye is aligned with one or more exit pupils provided by the outcoupler, at least a portion of the light exiting through the outcoupler will enter the pupil of the eye, thereby enabling the user to see a virtual image. Since the optical combiner lens is transparent, the user will also be able to see the real world.BRIEF SUMMARY OF EMBODIMENTS

[0002] Embodiments are described herein in which a virtual image is displayed to a user via a light engine to generate a display light representing the virtual image, a diffractive waveguide, and an incoupler and outcoupler that are each optically coupled to the diffractive waveguide. In operation, a pose of the waveguide is estimated based on interactions of the display light with one or more retroreflective optical elements and at least one photoreceptive sensor.

[0003] In an embodiment, a system to display a virtual image to a user comprises a waveguide coupled to an incoupler, the incoupler to receive and direct display light representing the virtual image into the waveguide; at least one photoreceptive sensor; one or more retroreflective optical elements to reflect a portion of the display light towards the at least one photoreceptive sensor; and a controller to estimate apose of the waveguide based on interactions of the reflected portion of display light with the at least one photoreceptive sensor.

[0004] The system may further comprise a light engine to generate the display light and to direct the display light towards the incoupler via at least one of the one or more retroreflective optical elements. The controller may be configured to modify a position to which the light engine directs the display light based on the estimated pose of the waveguide. The light engine may include the at least one photoreceptive sensor.

[0005] The one or more retroreflective optical elements may include a cross-dichroic prism.

[0006] The one or more retroreflective optical elements may include the incoupler.

[0007] The system may further comprise a non-visible light source to generate a non- visible portion of the display light, and to direct the non-visible portion towards the at least one photoreceptive sensor via at least one of the one or more retroreflective optical elements. The non-visible portion of the display light may be a near-infrared (NIR) portion, such that the at least one photoreceptive sensor includes an NIR sensor. The one or more retroreflective optical elements may include a dichroic reflector to selectively reflect the NIR portion of the display light while allowing visible components of the display light to substantially pass through the dichroic reflector.

[0008] In an embodiment, a method for displaying a virtual image comprises directing display light representing the virtual image into a waveguide via an incoupler that is optically coupled to the waveguide; reflecting a portion of the display light towards at least one photoreceptive sensor via one or more retroreflective optical elements; and estimating a pose of the waveguide based on interactions of the reflected portion of display light with the at least one photoreceptive sensor.

[0009] Directing the display light into the waveguide may comprise generating the display light with a light engine, such that the method further comprises modifying a position to which the light engine directs the display light based on the estimated pose of the waveguide. The light engine may include at least one photoreceptive sensor, such that reflecting a portion of the display light towards the at least onephotoreceptive sensor includes reflecting the portion of the display light towards the light engine.

[0010] Reflecting the portion of the display light via one or more retroreflective optical elements may include reflecting the portion of the display light via a cross-dichroic prism.

[0011] Reflecting the portion of the display light via one or more retroreflective optical elements may include reflecting the portion of the display light via the incoupler.

[0012] The method may further comprise generating a non-visible portion of the display light, and directing the non-visible portion towards the at least one photoreceptive sensor via at least one of the one or more retroreflective optical elements. Directing the non-visible portion of the display light towards the at least one photoreceptive sensor may include directing a near-infrared (NIR) portion of the display light towards an NIR sensor. Directing the non-visible portion of the display light towards the at least one photoreceptive sensor may include selectively reflecting the non-visible portion of the display light via a dichroic reflector.

[0013] The reflected portion of display light may form a reflected image at the at least one photoreceptive sensor, such that estimating the pose of the waveguide comprises comparing a position of the reflected image with a nominal position for the reflected image.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items.

[0015] FIG. 1 illustrates a portion of an augmented reality (AR) display system, in accordance with some embodiments.

[0016] FIG. 2 illustrates a portion of an augmented reality (AR) display system, in accordance with some embodiments.

[0017] FIG. 3 illustrates a partial component view of a WHUD device that utilizes interactions of display light with a retroreflective optical element and photoreceptive sensor to estimate a pose of a waveguide of the WHUD device, in accordance with some embodiments.

[0018] FIG. 4 illustrates a partial component view of a WHUD device that utilizes interactions of display light with a retroreflective optical element and photoreceptive sensor to estimate a pose of a waveguide of the WHUD device, in accordance with some embodiments.

[0019] FIG. 5 illustrates a partial component view of a WHUD device that utilizes interactions of a display light with a retroreflective optical element to estimate a pose of a waveguide of the WHUD device, in accordance with some embodiments.

[0020] FIG. 6 illustrates a flow diagram of an operational routine performed in accordance with some embodiments.

[0021] FIG. 7 is a component-level block diagram illustrating an example of a WHUD computing system suitable for implementing one or more embodiments.DETAILED DESCRIPTION

[0022] World-locked content, in which virtual content to be displayed to a user of a WHUD device is positioned in place with respect to real world objects or points that are viewed by that user, is an important feature of AR displays. To “lock” virtual content shown on the WHUD device to the world, the pose (or orientation) of the waveguide must be known with respect to the user’s eye. However, due to thermal and mechanical stresses on the WHUD device during use, portions of the WHUD device can warp elastically and plastically over time, resulting in an offset between the designed orientation and current conditions. In certain scenarios, such an offset may also be caused by manufacturing tolerance factors. Aside from incorrectly positioned world-locked content, misalignment between the waveguide and the light engine can have negative impacts on display efficiency and / or uniformity, and in extreme cases can cause clipping of the field of view of the AR display.

[0023] To ensure good user AR experiences throughout the lifetime of the WHUD device, it is therefore important to measure, monitor, and recalibrate the device to accommodate changes in the pose of the waveguide. Previous approaches for pose estimation include using strain gauges in the frame, or optical fiber shape sensors. Moreover, passive mechanical designs that include stiffer, larger frames and increased weight have also been utilized to avoid unintended changes in waveguide orientation. However, such approaches are associated with a deteriorated user experience.

[0024] Embodiments of techniques discussed herein generally exemplify estimating a pose of a waveguide in a WHUD device based on interactions of display light with a retroreflective optical element and one or more photoreceptive sensors. Display light representing the virtual image is directed into a waveguide via an incoupler that is optically coupled to the waveguide, while a portion of the display light is reflected towards at least one photoreceptive sensor via one or more retroreflective optical elements, such as a cross-dichroic prism. A pose of the waveguide is estimated based on interactions of the reflected portion of display light with the at least one photoreceptive sensor.

[0025] As used herein, retroreflective is used to refer to a reflective interface that redirects some or all light rays encountering that reflective interface generally backwards — that is, with an angle of reflection between 90° and 180° in at least one dimension of travel with respect to the light rays’ most recent point of origination or reflection, such as display light that is reflected from a waveguide back towards a light engine generating that display light. A retroreflective optical element may, in certain embodiments, operate as a reflective optical element for a portion of light rays encountering its relevant reflective interface while operating as a retroreflective optical element for some other portion of light rays encountering that reflective interface. Thus, as used herein, any reflective optical element may be considered a retroreflective optical element based on the directionality provided by its reflective interface or interfaces.

[0026] FIG. 1 illustrates an example display system 100 employing an AR optical system in accordance with some embodiments. The display system 100 has a support structure 102 that includes an arm 104, which houses a projector (e.g., alaser projector, a micro-LED projector, a Liquid Crystal on Silicon (LCOS) projector, or the like). The projector is configured to project images toward the eye of a user via a waveguide (not shown here), such that the user perceives the projected images as being displayed in a field of view (FOV) area 106 of a display at one or both of lens elements 108, 110. In the depicted embodiment, the display system 100 is a neareye display system in the form of a WHLID in which the support structure 102 is configured to be worn on the head of a user and has a general shape and appearance (that is, form factor) of an eyeglasses (e.g., sunglasses) frame.

[0027] The support structure 102 contains or otherwise includes various components to facilitate the projection of such images toward the eye of the user, such as a projector and a waveguide. In some embodiments, the support structure 102 further includes various sensors, such as one or more front-facing cameras, rear-facing cameras, other light sensors, motion sensors, accelerometers, and the like. In some embodiments, the support structure 102 includes one or more radio frequency (RF) interfaces or other wireless interfaces, such as a Bluetooth(TM) interface, a WiFi interface, and the like. Further, in some embodiments, the support structure 102 further includes one or more batteries or other portable power sources for supplying power to the electrical components of the display system 100. In some embodiments, some or all of these components of the display system 100 are fully or partially contained within an inner volume of support structure 102, such as within the arm 104 in region 112 of the support structure 102. It should be noted that while an example form factor is depicted, it will be appreciated that in other embodiments the display system 100 may have a different shape and appearance from the eyeglasses frame depicted in FIG. 1. It should be understood that instances of the term “or” herein refer to the non-exclusive definition of “or”, unless noted otherwise. For example, herein the phrase “X orY” means “either X, orY, or both”.

[0028] One or both of the lens elements 108, 110 are used by the display system 100 to provide an augmented reality (AR) display in which rendered graphical content can be superimposed over or otherwise provided in conjunction with a real-world view as perceived by the user through the lens elements 108, 110. For example, a projection system of the display system 100 uses light to form a perceptible image or series of images by projecting the display light onto the eye of the user via a projector of theprojection system, a waveguide formed at least partially in the corresponding lens element 108 or 110, and one or more optical elements (e.g., one or more retroreflective optical elements, scan mirrors, optical relays, or collimation lenses that are disposed between the projector and the waveguide or integrated with the waveguide), according to various embodiments.

[0029] One or both of the lens elements 108, 110 comprises a lens stack having multiple layers, at least one of which layers includes at least a portion of a waveguide that routes display light received by an incoupler of the waveguide to an outcoupler of the waveguide. The waveguide outputs the display light toward an eye of a user of the display system 100. The display light is modulated and projected onto the eye of the user such that the user perceives the display light as an image. In addition, each of the lens elements 108, 110 is sufficiently transparent to allow a user to see through the lens elements to provide a field of view of the user’s real-world environment such that the image appears superimposed over at least a portion of the real-world environment.

[0030] In some embodiments, the projector of the projection system of the display 100 is a digital light processing-based projector, a scanning laser projector, or any combination of a modulative light source, such as a laser or one or more lightemitting diodes (LEDs), and a dynamic reflector mechanism such as one or more dynamic scanners, reflective panels, or digital light processors (DLPs). In some embodiments, a display panel of the projector is configured to output light (representing an image or portion of an image for display) into the waveguide of the projector. The waveguide expands the display light and outputs the display light toward the eye of the user via an outcoupler.

[0031] The projector is communicatively coupled to the controller and a non-transitory processor-readable storage medium or memory storing processor-executable instructions and other data that, when executed by the controller, cause the controller to control the operation of the projector. In some embodiments, the controller controls the projector to selectively set the location and size of the FOV area 106. In some embodiments, the controller is communicatively coupled to one or more processors (not shown) that generate content to be displayed at the display system 100. The projector outputs display light toward the FOV area 106 of the display system 100 viathe waveguide. In some embodiments, at least a portion of an outcoupler of the waveguide overlaps the FOV area 106. Herein, the range of different user eye positions that will be able to see the display is referred to as the eyebox of the display.

[0032] FIG. 2 illustrates a portion of a display system 200 that includes a projection system having a projector 206 and a waveguide 212 with multiple optical paths between an incoupler 214 and an outcoupler 216 of the waveguide 212. In some embodiments, the display system 200 represents the display system 100 of FIG. 1. In the present example, the arm 204 of the display system 200 houses the projector 206, which includes an optical engine 208 (e.g., a display panel), a retroreflective optical element 210, the incoupler 214, and a portion of the waveguide 212.

[0033] In certain embodiments, retroreflective optical element 210 is a cross-dichroic prism, also known as an X-cube. In embodiments, the retroreflective optical element 210 partially reflects at least a portion of the display light passing to and / or through the retroreflective optical element 210 towards one or more photoreceptive sensors (not shown) placed in proximity to the retroreflective optical element 210, such that a controller of the WHLID device estimates a pose of the waveguide 212 based on interaction of the reflected portion of display light with the one or more photoreceptive sensors.

[0034] The display system 200 includes an optical combiner lens 218, which in turn includes a first lens 220, a second lens 222, and the waveguide 212, with the waveguide 212 embedded or otherwise disposed between the first lens 220 and the second lens 222.

[0035] Light exiting through the outcoupler 216 travels through the first lens 220 (which corresponds to, for example, an embodiment of the lens element 110 of the display system 100 or portion thereof). In use, the display light exiting the first lens 220 enters the pupil of an eye 224 of a user wearing the display system 200, causing the user to perceive a displayed image carried by the display light output by the optical engine 208. The optical combiner lens 218 is substantially transparent, such that at least some light from real-world scenes corresponding to the environment around the display system 200 passes through the second lens 222, the waveguide212, and the first lens 220 to the eye 224 of the user. In this way, images or other graphical content output by the projector 206 are combined (e.g., overlayed) with real-world images of the user’s environment when projected onto the eye 224 of the user to provide an AR experience to the user.

[0036] The waveguide 212 of the display system 200 includes two diffraction structures: the incoupler 214 and the outcoupler 216. In some embodiments, one or more exit pupil expanders, such as a diffraction grating, is arranged in an intermediate stage between incoupler 214 and outcoupler 216 to receive light that is coupled into the waveguide 212 by the incoupler 214, expand the display light received at each exit pupil expander, and redirect that light towards the outcoupler 216, where the outcoupler 216 then couples the display light out of the waveguide 212 (e.g., toward the eye 224 of the user).

[0037] The term “waveguide,” as used herein, will be understood to mean a combiner using one or more of total internal reflection (TIR), specialized filters, or reflective surfaces, to transfer light from an incoupler (such as the incoupler 214) to an outcoupler (such as the outcoupler 216). In some display applications, the display light is a collimated image, and the waveguide transfers and replicates the collimated image to the eye. In general, the terms “incoupler” and “outcoupler” will be understood to refer to any type of optical grating structure, including, but not limited to, diffraction gratings, holograms, holographic optical elements (e.g., optical elements using one or more holograms), volume diffraction gratings, volume holograms, surface relief diffraction gratings, or surface relief holograms. In some embodiments, a given incoupler or outcoupler is configured as a transmissive grating (e.g., a transmissive diffraction grating or a transmissive holographic grating) that causes the incoupler or outcoupler to transmit display light. In some embodiments, a given incoupler or outcoupler is a reflective grating (e.g., a reflective diffraction grating or a reflective holographic grating) that causes the incoupler or outcoupler to reflect light. In the present example, the incoupler 214 relays received display light to the outcoupler 216 via multiple optical paths through the waveguide. In some embodiments, the incoupler 214 redirects a first portion of display light to the outcoupler 216 via a first optical path along which a first exit pupil expander (not shown; implemented as a fold grating in some embodiments) is disposed andredirects a second portion of display light toward the outcoupler 216 via a second optical path along which a second exit pupil expander (not shown; implemented as a fold grating in some embodiments) is disposed. The display light propagates through the waveguide 212 via TIR. The outcoupler 216 then outputs the display light to the eye 224 of the user.

[0038] In some embodiments, the projector 206 is coupled to a driver or other controller (not shown), which controls the timing of emission of display light from light sources (e.g., LEDs) of the optical engine 208 in accordance with instructions received by the controller or driver from a computer processor (not shown) coupled thereto to modulate the output light to be perceived as images when output to the retina of the eye 224 of the user. For example, during operation of the display system 200, the light sources of the optical engine 208 output light of selected wavelengths, and the output light is directed to the eye 224 of the user via the retroreflective optical element 210 and the waveguide 212. The optical engine 208 modulates the respective intensities of each light source of the optical engine 208, such that the output light represents pixels of an image. For example, the intensity of a given light source or group of light sources of the optical engine 208 corresponds to the brightness of a corresponding pixel of the image to be projected by the projector 206 of the display system 200.

[0039] FIGs. 3-6 provide partial component views of various WHUD device embodiments that utilize interactions of a display light passing at least partially through a retroreflective optical element towards one or more photoreceptive sensors to estimate a pose of a waveguide of the WHUD device. It will be appreciated that while exemplary light beams are depicted in order to generally illustrate central paths of display light, such depicted light beams are merely exemplary and are not intended to accurately depict those or other paths of the display light.

[0040] FIG. 3 illustrates a partial component view of a WHUD device that utilizes interactions of display light with a retroreflective optical element and photoreceptive sensor to estimate a pose of a waveguide of the WHUD device, in accordance with some embodiments. In the depicted embodiment, a cross-sectional side view 300 presents a light engine 308 (e.g., a display panel) that generates and directs adisplay light 302 through an optical element 310 towards an incoupler 314, which is optically coupled to a waveguide 312.

[0041] The incoupler 314 directs a majority (not shown) of the display light 302 into the waveguide 312, such that the majority of the display light 302 propagates through the waveguide 312 towards an outcoupler (not shown) and then to an eye of a user, which perceives the majority of the display light as a virtual image (superimposed, for example, over a view of the real world external to the WHLID device as worn by the user). However, the incoupler 314 is configured to reflect a portion of the display light 302 back towards the optical element 310 (as reflected display light 304) towards photoreceptive sensor 320 via a reflective interface 306 of the optical element 310. In this manner, in the depicted embodiment the incoupler 314 itself operates as an additional retroreflective optical element.

[0042] Interactions of the reflected display light 304 with the photoreceptive sensor 320 provides information indicative of a sensor image 350, such as a back-reflected image 390 received at the photoreceptive sensor 320. In certain embodiments, a controller (e g., processor 802 of FIG. 8 below) estimates a pose of the waveguide 312 based at least in part on a comparison of a position of the back-reflected image 390 with a nominal image position 380. Based on the estimated pose of the waveguide 312, the controller may compensate for the positional offset between the back-reflected image 390 and the nominal image position 380, such as by altering a position (e.g., via pixel shifting or other technique) and / or manner in which the light engine 308 directs the display light 302.

[0043] A dichroic prism is a type of prism that separates light into different wavelengths based on their respective polarizations. In some embodiments, the optical element 310 comprises two substantially perpendicular dichroic prisms that are coupled (e.g., via one or more optical adhesives) along their respective hypotenuse faces in an X-cube configuration. By selecting the refractive index for the optical interface formed between the two dichroic prisms, the reflective interface 306 is formed.

[0044] FIG. 4 illustrates a partial component view of a WHUD device that utilizes interactions of a display light 402 with an optical element 410 to estimate a pose of awaveguide 412 of the WHUD device, in accordance with some embodiments. In the depicted embodiment, a cross-sectional side view 400 presents a light engine 408 that, in a manner similar to that described above with respect to light engine 308 of FIG. 3, generates and directs display light 402 through optical element 410 towards an incoupler 414, which is optically coupled to the waveguide 412.

[0045] In contrast to the embodiment of FIG. 3, the photoreceptive sensor 320 is omitted in favor of integrating one or more photoreceptive sensors 420 into a display panel of the light engine 408. For example, as shown in the lower front view of light engine 408, each of multiple photoreceptive sensors 420 are interspersed within a grid of display image pixels 409. The incoupler 414 serves as a retroreflective optical element, reflecting a portion of the display light 402 (as reflected display light 404) back through the optical element 410 and its reflective interface 406 towards photoreceptive sensors 420, such that interactions of the reflected display light 404 with the photoreceptive sensors 420 indicates a sensor image 450 that includes back-reflected image 490 received at the photoreceptive sensors 420. A controller (e.g., processor 802 of FIG. 8 below) of the WHUD device estimates a pose of the waveguide 412 based at least in part on a comparison of a position of the back- reflected image 490 with a nominal image position 480. Based on the estimated pose of the waveguide 412, the controller may compensate for the positional offset between the back-reflected image 490 and the nominal image position 480, such as by altering a position to which the light engine 408 directs the display light 402.

[0046] FIG. 5 illustrates a partial component view of a WHUD device that utilizes interactions of a display light with a retroreflective optical element to estimate a pose of a waveguide of the WHUD device, in accordance with some embodiments. In the depicted embodiment, a cross-sectional side view 500 presents a light engine 508 that, in a manner similar to that described above with respect to light engine 308 of FIG. 3, generates and directs display light (not separately depicted here for clarity) through a retroreflective optical element 510 towards incoupler 514, which is optically coupled to waveguide 512.

[0047] In the depicted embodiment, a reflective interface 508 of the retroreflective optical element 510 comprises a narrowband dichroic reflective interface, which operates as a reflector to selectively reflect one or more near-infrared (NIR)wavelengths interfacing with the dichroic reflective interface 508 while allowing visible components of the display light to pass through the retroreflective optical element 510 to the incoupler 514 and waveguide 512. A NIR light source 525 adds NIR light 504 to the display light directed towards the incoupler 514. In particular, the NIR light source 525 directs the NIR light 504 to reflect from the dichroic reflective interface 508 towards the incoupler 514.

[0048] In embodiments including that of FIG. 5, the incoupler 514 is configured to admit into the waveguide 512 visible light components of display light interfacing with the incoupler, while reflecting NIR light 504 back towards the dichroic reflective interface 508, which in turn reflects the NIR light 504 towards NIR sensor 520. Interactions of the reflected NIR light 504 with the NIR sensor 520 indicates a sensor image 550 that includes back-reflected image 590 received at the photoreceptive sensors 520. A controller (e.g., processor 802 of FIG. 8 below) of the WHUD device estimates a pose of the waveguide 512 based at least in part on a comparison of a position of the back-reflected image 590 with a nominal image position 580. Based on the estimated pose of the waveguide 512, the controller may compensate for the positional offset between the back-reflected image 590 and the nominal image position 580, such as by altering a position to which the light engine 508 directs the display light 502.

[0049] In certain embodiments, additional techniques may be utilized to increase the optical efficiency of one or more elements. For example, in embodiments one or more optical coatings could be disposed on at least a portion of the side of waveguide 512 opposite the incoupler 514 (the world side of the waveguide 512, in the depicted embodiment), with those optical coatings being configured to increase reflection of NIR wavelengths. As another example, in various embodiments one or both of the NIR light source 525 and / or NIR sensor 520 are integrated into a display panel of the light engine 508 in a manner similar to that described above with respect to the photoreceptive sensors 420 of FIG. 4. In such embodiments, the NIR reflectivity of the dichroic reflective interface 508 is modified to allow passage of at least a portion of NIR wavelengths through the retroreflective optical element 510.

[0050] It will be appreciated that the techniques described for waveguide pose estimation with respect to the embodiments of FIGs. 3-5 may, in various embodiments, be utilized in lieu or in conjunction with one another.

[0051] FIG. 6 illustrates a flow diagram of an operational routine in accordance with some embodiments, such as may be collectively performed by one or more components of a WHLID device (e.g., display system 100 of FIG. 1 , display system 200 of FIG. 2, and / or WHLID computing system 700 of FIG. 7).

[0052] The routine begins at block 605, in which the WHLID device directs display light representing the virtual image into a waveguide (e.g., one of waveguides 212, 312, 412, 512 of respective FIGs. 2-5) via an incoupler that is optically coupled to the waveguide (e.g., one of incouplers 214, 314, 414, 514, 614 of respective FIGs. 2-5). The routine proceeds to block 610.

[0053] At block 610, the WHLID device reflects a portion of the display light towards at least one photoreceptive sensor (e.g., one or more of photoreceptive sensors 320, 420, 520 of respective FIGS. 3-5) via one or more retro reflective optical elements (e.g., one of retroreflective optical elements 310, 410, 510, and / or one of incouplers 314, 414, 514 of FIGs. 3-5, respectively). The routine proceeds to block 615.

[0054] At block 615, the WHUD device estimates a pose of the waveguide based on interactions of the reflected portion of display light with the at least one photoreceptive sensor. For example, and as discussed elsewhere herein, in certain embodiments estimating the pose of the waveguide includes comparing a position of a reflected image formed by the reflected portion of display light at a photoreceptive sensor with a nominal position for the reflected image.

[0055] FIG. 7 is a component-level block diagram illustrating an example of a WHUD computing system 700 suitable for implementing one or more embodiments. In alternative embodiments, the WHUD computing system 700 may operate as a standalone device or may be connected (e.g., networked) to other systems. In various embodiments, one or more components of the WHUD computing system 700 may be incorporated as or within one or more server computing systems to provide, as one non-limiting example, graphics rendering for display to a user via anincorporating WHUD device, such as display system 100 of FIG. 1 and / or display system 200 of FIG. 2. It will be appreciated that an associated server computing device may include some components of WHUD computing system 700, but not necessarily all of them. In a networked deployment, the WHUD computing system 700 may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In at least one example, the WHUD computing system 700 may act as a peer system in peer-to-peer (P2P) (or other distributed) network environment. The WHUD computing system 700 may therefore operate in certain embodiments as a server computer, a personal computer (PC), , a mobile computing device, a web appliance, a network router, switch or bridge, or any system capable of executing instructions (sequential or otherwise) that specify actions to be taken by that system. Further, while only a single system is illustrated, the term "system" shall also be taken to include any collection of systems that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0056] Examples, as described herein, may include, or may operate by, logic or a number of components, or mechanisms. Circuitry is a collection of circuits implemented in tangible entities that include hardware (e.g., simple circuits, gates, logic, etc.). Circuitry membership may be flexible over time and underlying hardware variability. Circuitries include members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuitry may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuitry may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuitry in hardware via the variable connections to carry out portions of the specific operation when in operation.Accordingly, the computer readable medium is communicatively coupled to the othercomponents of the circuitry when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuitry. For example, under operation, execution units may be used in a first circuit of a first circuitry at one point in time and reused by a second circuit in the first circuitry, or by a third circuit in a second circuitry at a different time.

[0057] The WHLID computing system 700 may include one or more hardware processors 702 (e.g., a central processing unit (CPU), a hardware processor core, or any combination thereof), a main memory 704, and a graphics processing unit (GPU) 706, some or all of which may communicate with each other via an interlink (e.g., bus) 708. The WHUD computing system 700 may further include a display unit 710 (such as a display monitor or other display device), an alphanumeric input device 712 (e.g., a keyboard or other physical or touch-based actuators), and a user interface (Ul) navigation device 714 (e.g., a pointing device, such as a touch-based, eyetracking based, or other pointing / selecting interface). In one example, the display unit 710, input device 712, and Ul navigation device 714 may include a touch screen display. The WHUD computing system 700 may additionally include a storage device (e.g., drive unit) 716, a signal generation device 718 (e.g., a speaker), one or more sensors 720, and a network interface device 721 .The sensors 720 may include, as non-limiting examples, one or more photoreceptive sensors (e.g., photoreceptive sensors 320, 420, 520, 620 of respective FIGS. 3-6), a global positioning system (GPS) sensor, compass, accelerometer, or other sensor. The WHUD computing system 700 may include an output controller 728, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0058] The storage device 716 may include a computer readable medium 722 on which is stored one or more sets of data structures or instructions 724 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 724 may also reside, completely or at least partially, within the main memory 704, within GPU 706, or within the hardware processor 702 during execution thereof by the WHUD computing system 700. In an example, one or anycombination of the hardware processor 702, the main memory 704, the GPU 706, or the storage device 716 may constitute computer readable media.

[0059] While the computer readable medium 722 is illustrated as a single medium, the term "computer readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 724.

[0060] The term "computer readable medium" may include any medium that is capable of storing, encoding, or carrying instructions for execution by the WHUD computing system 700 and that cause the WHUD computing system 700 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting computer readable medium examples may include solid- state memories, and optical and magnetic media. In an example, a massed computer readable medium includes a computer readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, massed computer readable media are not transitory propagating signals. Specific examples of massed computer readable media may include non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0061] The instructions 724 may further be transmitted or received over a communications network 726 using a transmission medium via the network interface device 720 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 702.11 family of standards known as WiFi®, IEEE 702.16 family of standards known as WiMax®), IEEE 702.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the networkinterface device 720 may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network 726. In an example, the network interface device 720 may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the WHLID computing system 700, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

[0062] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software comprises one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer readable storage medium. The software can include the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer readable storage medium can include, for example, a magnetic or optical disk storage device, solid state storage devices such as Flash memory, a cache, random access memory (RAM) or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.

[0063] A computer readable storage medium may include any storage medium, or combination of storage media, accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disk, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer readable storage medium may be embedded in the computing system (e.g., systemRAM or ROM), fixedly attached to the computing system (e g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disc or Universal Serial Bus (USB)-based Flash memory), or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).

[0064] Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device may not be required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0065] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.

Claims

WHAT IS CLAIMED IS:1 . A system to display a virtual image to a user, the system comprising: a waveguide coupled to an incoupler, the incoupler to receive and direct display light representing the virtual image into the waveguide; at least one photoreceptive sensor; one or more retroreflective optical elements to reflect a portion of the display light towards the at least one photoreceptive sensor; and a controller to estimate a pose of the waveguide based on interactions of the reflected portion of display light with the at least one photoreceptive sensor.

2. The system of claim 1 , further comprising a light engine to generate the display light and to direct the display light towards the incoupler via at least one of the one or more retroreflective optical elements.

3. The system of claim 2, wherein the controller is configured to modify a position to which the light engine directs the display light based on the estimated pose of the waveguide.

4. The system of any one of claims 2 to 3, wherein the light engine includes the at least one photoreceptive sensor.

5. The system of any one of claims 1 to 4, wherein the one or more retroreflective optical elements comprise a cross-dichroic prism.

6. The system of any one of claims 1 to 5, wherein the one or more retroreflective optical elements include the incoupler.

7. The system of any one of claims 1 to 6, further comprising a non-visible light source to generate a non-visible portion of the display light and to direct the non-visible portion towards the at least one photoreceptive sensor via at least one of the one or more retroreflective optical elements.

8. The system of claim 7, wherein the non-visible light source is a near-infrared (NIR) light source, and wherein the at least one photoreceptive sensor includes an NI sensor.

9. The system of any one of claims 7 or 8, wherein the one or more retroreflective optical elements include a dichroic reflector to selectively reflect the non-visible portion of the display light while allowing visible components of the display light to substantially pass through the dichroic reflector.

10. A method for displaying a virtual image to a user, the method comprising: directing display light representing the virtual image into a waveguide via an incoupler that is optically coupled to the waveguide; reflecting a portion of the display light towards at least one photoreceptive sensor via one or more retroreflective optical elements; and estimating a pose of the waveguide based on interactions of the reflected portion of display light with the at least one photoreceptive sensor.11 . The method of claim 10, wherein directing the display light into the waveguide comprises generating the display light with a light engine, and wherein the method further comprises modifying a position to which the light engine directs the display light based on the estimated pose of the waveguide.

12. The method of claim 11 , wherein the light engine includes the at least one photoreceptive sensor, and wherein reflecting a portion of the display light towards the at least one photoreceptive sensor includes reflecting the portion of the display light towards the light engine.

13. The method of any one of claims 10 to 12, wherein reflecting the portion of the display light via one or more retroreflective optical elements comprises reflecting the portion of the display light via a cross-dichroic prism.

14. The method of any one of claims 10 to 13, wherein reflecting the portion of the display light via one or more retroreflective optical elements comprises reflecting the portion of the display light via the incoupler.

15. The method of any one of claims 10 to 14, further comprising generating a non- visible portion of the display light and directing the non-visible portion towards the at least one photoreceptive sensor via at least one of the one or more retroreflective optical elements.

16. The method of claim 15, wherein the non-visible light source is a near-infrared(NIR) light source, and wherein directing the non-visible portion of the display light towards the at least one photoreceptive sensor includes directing a NIR portion of the display light towards an NIR sensor.

17. The method of any one of claims 15 or 16, wherein directing the non-visible portion of the display light towards the at least one photoreceptive sensor includes selectively reflecting the non-visible portion of the display light via a dichroic reflector.

18. The method of any one of claims 10 to 17, wherein the reflected portion of display light forms a reflected image at the at least one photoreceptive sensor, and wherein estimating the pose of the waveguide comprises comparing a position of the reflected image with a nominal position for the reflected image.

19. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, causes the one or more processors to display a virtual image to a user, the method comprising: directing display light representing the virtual image into a waveguide via an incoupler that is optically coupled to the waveguide; reflecting a portion of the display light towards at least one photoreceptive sensor via one or more retroreflective optical elements; and estimating a pose of the waveguide based on interactions of the reflected portion of display light with the at least one photoreceptive sensor.

20. The non-transitory computer-readable medium of claim 19, wherein the method performed by the one or more processors further comprises modifying a position at which the display light is directed based on the estimated pose of the waveguide.

Images