Optical waveguide combiner systems and methods

Abstract

An optical display system has an optical waveguide combiner and one or more cameras, cameras is optically coupled to the optical waveguide combiner and have a field of view of at least one real object and at least one virtual object displayable by the optical display system. The cameras, which may be for example wafer level waveguide cameras, may be disposed outside the usable field of view of an output coupler. The cameras may be self-calibrated electronically using images captured by the cameras of one or more virtual objects displayable by the optical display system. AR / VR / MR registration of devices and / or displayed virtual objects with real objects may be implemented using the images captured by the cameras of the displayed virtual objects and real world objects. Real object distance and / or spatial location relative to the optical waveguide combiners may be determined or estimated from the captured images.

Description

(19) *EP004657140A2* (11) EP 4 657 140 A2 (12) EUROPEAN PATENT APPLICATION (43) Date of publication: 03.12.2025 Bulletin 2025 / 49 (21) Application number: 25208906.5 (22) Date of filing: 30.11.2022 (51) International Patent Classification (IPC): G02B 27 / 01 (2006.01) (52) Cooperative Patent Classification (CPC): G02B 27 / 0081; G02B 27 / 0172; G02B 2027 / 0138; G02B 2027 / 0178 (84) Designated Contracting States: AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR (30) Priority: 07.12.2021 US 202117544870 (62) Document number(s) of the earlier application(s) in accordance with Art. 76 EPC: 22840533.8 / 4 445 207 (71) Applicant: Snap Inc. Santa Monica, CA 90405 (US) (72) Inventor: LEIGHTON, James Raymond Santa Monica, 90405 (US) (74) Representative: Mathys & Squire The Shard 32 London Bridge Street London SE1 9SG (GB) Remarks: This application was filed on 15‑10‑2025 as a divisional application to the application mentioned under INID code 62. (54) OPTICAL WAVEGUIDE COMBINER SYSTEMS AND METHODS (57) An optical display system has an optical wave- guide combiner and one or more cameras, cameras is optically coupled to the optical waveguide combiner and have a field of view of at least one real object and at least one virtual object displayable by the optical display sys- tem. The cameras, whichmay be for example wafer level waveguide cameras, may be disposed outside the usable field of view of an output coupler. The cameras may be self-calibrated electronically using images cap- tured by the cameras of one or more virtual objects displayable by the optical display system. AR / VR / MR registration of devices and / or displayed virtual objects with real objects may be implemented using the images captured by the cameras of the displayed virtual objects and real world objects. Real object distance and / or spa- tial location relative to the optical waveguide combiners may be determined or estimated from the captured images. EP 4 65 7 14 0 A 2 Processed by Luminess, 75001 PARIS (FR) 2 1 EP 4 657 140 A2 2 Description CLAIM OF PRIORITY

[0001] This application claims priority to U.S. Non- Provisional Patent Application No. 17 / 544,870, filed De- cember 7, 2021, which is hereby incorporated by refer- ence herein in its entirety. TECHNICAL FIELD

[0002] Thepresent disclosure relates to optical display systems and, in particular but not exclusively, to optical waveguide combiner systems. Additionally or alterna- tively, embodiments relate to methods for camera cali- bration in optical display systems. Additionally or alter- natively, embodiments relate to apparatus and / or meth- ods for registration of one ormore virtual objects in virtual reality and / or augmented reality display systems. BACKGROUND

[0003] The utilisation of virtual reality and augmented reality display systems has increased in recent years, with such devices being used for both personal and commercial / industrial purposes. Devices may be used in monocular or binocular formats and may be used for purposes ranging from monochrome display of simple iconography, through presentation of text based instruc- tions to full video display.When such devices are used to supplement a real-world scene with computer generated images, the display system needs to accurately display virtual or augmented reality content in context. SUMMARY

[0004] According to some aspects of the present tech- nology, an optical display system is provided. The optical display systemmay comprise an optical waveguide com- biner and a plurality of cameras. The optical waveguide combiner may comprise a waveguide optical substrate, an optical input coupler on, in or adjacent the waveguide optical substrate; and an optical output coupler on or in the waveguide optical substrate. The optical input cou- pler may be coupled via the waveguide optical substrate to theoptical output coupler. Theplurality of camerasmay comprise a first camera and a second camera. The first camera may be optically coupled to the output coupler and has a first field of view of one or more virtual object displayable at an image plane by the optical waveguide combiner. The displayable virtual object(s) may be in the field of view of an eyebox of the optical waveguide combiner. The second camera may be optically coupled to the output coupler and has a second field of view of the virtual object(s) displayable by the optical waveguide combiner. At the image plane, a combined field of view of the first camera and the second cameramay comprise the first field of view of the first camera and the second field of view of the second camera and may include an overlapping region of the first field of viewand the second field of view. At the image plane, the displayable one or more virtual object(s) may bewithin the combined field of view of the of the first camera and the second camera.

[0005] In some embodiments, the optical waveguide combiner is see-through. The one or more real object may be viewable within the combined field of view of the first camera and the second camera.

[0006] In some embodiments, at the image plane, the field of view of the eyebox and / or field of view of the projector is within the combined field of view of the first and second cameras.

[0007] In some embodiments, the plurality of cameras are located in a region outside the field of view of an eyebox of the optical waveguide combiner system.

[0008] In some embodiments, the output coupler com- prises a pre-exit pupil expansion region and an exit pupil expansion region. The first camera and the second cam- era may be optically coupled to the pre-exit pupil expan- sion region.

[0009] In some embodiments, the system further com- prises a projector configured to project a display image of the virtual object(s) into the input coupler.

[0010] In some embodiments, the system further com- prises one or more processor(s) operably coupled to the projector and the first camera and the second camera. The one or more processor(s) may be configured to cause the projector to project a display image of the virtual object(s) which includes fiducials, wherein the fiducials of the displayable virtual object(s) are within the overlapping region of the first field of view of the first camera and the second field of view of the second camera. The one or more processor(s) is configured to electronically stitch together first and second images captured from thefirst andsecondcameras, respectively, by adjusting the first and second images to align the fiducials in the first image with the fiducials in the second image.

[0011] In some embodiments, output coupler com- prises any one or combination of prism arrays, cascaded mirrors, diffractive gratings and holograms. The output coupler may comprise optical diffractive elements.

[0012] In some embodiments, the optical diffractive elements may comprise two diffractive optical elements at least partially overlaid on one another in or on the waveguide substrate.

[0013] In some embodiments, the output coupler com- prises a combined 2D expansion output diffractive grat- ing.

[0014] In some embodiments, each of the plurality of cameras may comprise a wafer level camera. The wafer level camera is incorporated in or on the optical wave- guide combiner.

[0015] In some embodiments, the optical input coupler comprises an optical input diffractive grating, and where- in the optical input diffractive grating is in optical com- munication with the optical output grating without any 5 10 15 20 25 30 35 40 45 50 55 3 3 EP 4 657 140 A2 4 intermediate optical diffractive grating therebetween.

[0016] In some embodiments, the optical waveguide combiner is incorporated in an eyepiece of a pair of eyeglasses.

[0017] According to some aspects of the present tech- nology, there is provided a method of camera calibration for an AR / MR / VR display. The method may comprise projecting into an optical waveguide combiner a display image including fiducials, optically coupling first and second cameras to an output of the optical waveguide combiner, the first and second cameras having a com- binedfieldof viewcomprisingafirst fieldof viewof thefirst camera and a second field of view of the second camera and including an overlapping region of the first field of view and the second field of view; displaying, by the optical waveguide combiner, at an image plane, the dis- play image as a virtual image with the fiducials, wherein the displayed virtual image is within the combined field of view of the first camera and the second camera, wherein the fiducials of the displayed virtual image at the image plane are within the overlapping region of the first field of view and the second field of view; capturing by the first camera a first image of the virtual image; capturing by the second camera a second image of the virtual image; and electronically stitching the captured first image and sec- ond image together; wherein electronically stitching the captured first image and second image together com- prises electronically adjusting the captured first image and captured second image by moving the captured first image and the captured second image to align the fidu- cials of the first image with the fiducials of the second image.

[0018] In some embodiments, electronically stitching the captured first image and second image together further comprises determining the overlapping region of the first field of view and the second field of view by comparing the adjusted first and second images with the display image projected into the combiner; and electro- nically correcting the adjusted first and second images according to the determined overlapping region.

[0019] In some embodiments, optically coupling the first camera and the second camera to the output of the optical waveguide combiner comprises optically cou- pling the first camera and the second camera to a pre-exit pupil expansion output of an output coupler of the optical waveguide combiner.

[0020] According to aspects of the present technology, there is provided a method of registering a VR / AR / MR display. The method may comprise: providing one or more virtual object(s) for display; projecting into an op- tical waveguide combiner a display image of the virtual object(s); displaying, by the optical waveguide combiner, the display image as the virtual object(s); capturing an image of the displayed virtual object(s) and real world scene using one or more optical camera(s), the cam- era(s) being optically coupled to an output of the optical waveguide combiner and having a field of view of the real world scene and of the displayed virtual object(s); com- paring, using the captured image, the virtual object(s) to one or more real object(s) of the real world scene; de- termininga registrationerror between thevirtual object(s) and the real object(s) basedon thecomparison; adjusting the virtual object(s) being displayed to substantially cor- rect the registration error.

[0021] In some embodiments, comparing, using the captured image, the virtual object(s) with the real ob- ject(s) comprises identifying a common feature or fiducial in the virtual object(s) and the real object(s); and compar- ing the identified common feature or fiducial in the virtual object(s) with the identified common feature or fiducial in the real object(s); wherein determining a registration error between the virtual object(s) and the real object(s) based on the comparison comprises determining a dif- ference in the position of the common feature or fiducial identified in the virtual object(s) and the position of the common feature or fiducial identified in the real object(s); and wherein adjusting the position of the virtual object(s) being displayed to substantially correct the registration error comprises adjusting anyoneor combination of size, shape and position of the virtual object(s) whereby the common feature or fiducial identified in the virtual ob- ject(s) and the common feature or fiducial identified in the real object(s) substantially coincide.

[0022] In some embodiments, providing the virtual ob- ject(s) comprise generating a virtual object from a real object using external sensors.

[0023] According to some aspects of the present tech- nology, there is provided a method of registering a VR / AR / MR display. The method may comprise generat- ing or providing one or more virtual object(s) of known size; based on an estimated distance of one or more real object(s) from an optical waveguide combiner, scaling the virtual object(s) for display by the optical waveguide combiner; projecting into theopticalwaveguide combiner a display image of the scaled virtual object(s); displaying, by the optical waveguide combiner, the display image as the virtual object(s); capturing the displayed virtual ob- ject(s) and real object(s) using one or more optical cam- era(s), the or each camera having a field of view of the real object(s) and of the displayed virtual object(s); com- paring, using the captured image, the real object(s) to the virtual object(s); determining a registration error between the virtual object(s) and the real object(s) based on the comparison; and adjusting the virtual object(s) being displayed to substantially correct the registration error.

[0024] In some embodiments, comparing the captured virtual object(s) to the real object(s) comprises determin- ing from the captured image an angular size and / or angle of the one or more real object(s) from the point of view of the camera(s); determining from the captured image an angular size and / or angle of the one or more virtual object(s) from the point of view of the camera(s); and wherein determining a registration error between the virtual object(s) and the real object(s) based on the comparison comprises determining the difference in an- gular sizeand / or angle of the oneormore virtual object(s) 5 10 15 20 25 30 35 40 45 50 55 4 5 EP 4 657 140 A2 6 relative to the one or more real object(s); and wherein adjusting the virtual object(s) being displayed to substan- tially correct the registration error comprises adjusting any one or combination of the size, shape and spatial location of the one or more virtual object(s) whereby the oneormorevirtual object(s) appear(s) substantially in the samespatial locationas the real objectwhenviewed from the eyebox.

[0025] According to the present technology, there is provided amethod of registration of a VR / MR / AR display system. The method may comprise: capturing stereo images using first and second optical camera(s) of a binocular system, the first and second cameras being optically coupled to respective outputs of first and second opticalwaveguidecombinersof thebinocular systemand having a field of view of one or more displayable virtual object(s) and of one or more real object(s); determining an estimated distance of the real object(s) from the optical combiners using the optical cameras and / or es- timated spatial location of the real object(s) using the optical cameras; generating the virtual object(s) for dis- play by the optical waveguide combiners based on the estimated distance and / or estimated spatial location; projecting into theopticalwaveguide combiners a display image of the virtual object(s); displaying, by the optical waveguide combiners, the display image as the virtual object(s); capturing an image of the displayed virtual object(s) and real object(s) using the optical cameras; comparing the displayed virtual object(s) to the real ob- ject(s) using the captured image; determining a registra- tion error between the real object(s) and the virtual ob- ject(s) based on the comparison; adjusting the virtual object(s) being displayed to substantially correct the registration error.

[0026] In some embodiments, determining an esti- mated distance of the real object(s) from the optical combiner(s) using the optical cameras comprises: ex- tracting from the captured images of the real object(s) a virtual image representing the real object(s); recognizing the extracted virtual object(s); determining depth percep- tion cues from the images captured by the optical cam- eras; estimating thedistanceof the real object(s) from the optical waveguide combiners based on the determined depth perception cues and object recognition.

[0027] In some embodiments, determining a registra- tion error between the real object(s) and the virtual ob- ject(s) based on the comparison comprises: determining the difference in spatial location between the displayed virtual object(s) and the real object(s) from the binocular disparity in the stereo images.

[0028] According to some aspects of the present tech- nology, there is provided a method of determining dis- tance in an AR / MR / VR display. The method may com- prise: capturing stereo images using optical camera(s) for a binocular optical waveguide combiner system, the optical cameras having a field of view of one or more displayable virtual object(s) and of one or more real object(s); extracting from the captured images of the real object(s) one or more virtual object(s) representing the real object(s); recognizing the extracted virtual object(s); determining depth perception cues from the images captured by the optical cameras; determining the dis- tance of the real object(s) from the optical combiners based on the determined depth perception cues and object recognition.

[0029] According to some aspects of the present tech- nology, there is provided a method of determining dis- tance in an AR / MR / VR display. The method may com- prise generatingor providingoneormore virtual object(s) for display; projecting into the optical waveguide combi- ners of a binocular optical combiner system a display image of the virtual object(s); displaying in stereo, by the optical wave combiners, the display image as a virtual object(s); capturing stereo imagesof thedisplayedvirtual object(s) and one or more real object(s) using one or more optical camera(s) incorporated into each optical waveguide combiner of the binocular system, the one or more camera(s) having a field of view of the real object(s) and of displayed virtual object(s); comparing the dis- played virtual object(s) to the real object(s) using the captured stereo images; determining the binocular dis- parity in the stereo images; and determining the distance of the real object(s) from virtual object(s) according to the binocular disparity.

[0030] In some aspects of the present technology, there is provided a non-transitory, computer-readable medium comprising program instructions, the program instructions, when executed by one or more processors of a computer system, cause the one ormore processors and / or the optical display system to perform any one or any combination of actions and / or method steps de- scribed herein.

[0031] In some embodiments, the present technology is directed to a system of one or more computers which can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of these installed on the system that in operation causes or cause the system to perform the actions and / or method steps described herein. An ex- ample system can comprise one or more processor; and amemory, the processor executing instructions stored in the memory to cause the one or more processors and / or the optical display system to perform actions and / or method steps described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG. 1 illustrates a perspective view of an optical waveguide combiner system according to some as- pects of the present technology; FIG. 2 illustrates a perspective view of the optical waveguide combiner system of FIG. 1 illustrating overlapping fields of view of different optical ele- ments of the system according to some embodi- 5 10 15 20 25 30 35 40 45 50 55 5 7 EP 4 657 140 A2 8 ments; FIG. 3 illustrates a perspective view of an optical waveguide combiner system according to some em- bodiments; FIG. 4 illustrates a plan view of a rear side of the optical waveguide combiner system formed to fit an eyepiece according to some embodiments; FIG. 5 illustrates a side viewof the optical waveguide combiner system of FIG. 4 according to some em- bodiments; FIG. 6 illustrates a perspective view of a pair of eyeglasses including eyepieces according to some embodiments; FIG. 7A and 7B are schematics illustrating a theo- dolite calibration technique for the purpose of ex- plaining camera self-calibration methods and sys- tems of the embodiments; FIG. 8 is a schematic illustrating inherent references of a waveguide for the purpose of explaining meth- ods and systems of camera self-calibration for ARNR displays according to some embodiments; FIG. 9 is a flow chart illustrating a method according to some aspects of the present technology; Fig. 10 is a schematic diagram illustrating fields of viewof optical elements of the optical display system according to some embodiments; Fig. 11 is a plan view of the field of view of the projector including alignment fiducials according to some embodiments; FIG. 12 is a plan view of virtual images captured by waveguide cameras with misaligned fields of views at the image plane of the optical display system; FIG. 13 is a plan view of the images of FIG. 12 electronically adjusted to align the alignment fidu- cials; FIG. 14 is a plan view of the images of FIG. 13 electronically corrected to represent undistorted the virtual image displayed by the optical waveguide combiner system; FIG. 15 illustrates a block diagram of an optical dis- play system according to some embodiments; FIG. 16 illustrates a flow chart of amethod according to some aspects of the present technology; FIG. 17 illustrates a schematic of an optical display system according to some embodiments showing a displayed virtual object and real object misaligned; FIG. 18 illustrates a block diagram of an optical dis- play system according to some embodiments; FIG. 19 illustrates a block diagram of an optical dis- play system according to some embodiments; FIG. 20 illustrates a flow chart of amethod according to some aspects of the present technology; FIG. 21 illustrates a schematic of an optical display system according to some embodiments showing a displayed virtual object and real object misaligned; FIG. 22 illustrates a flow chart of amethod according to some aspects of the present technology; FIG. 23 illustrates a binocular optical display system according to some embodiments; FIG. 24 illustrates a flow chart of amethod according to some aspects of the present technology; FIG. 25 illustrates schematic of a binocular optical display system according to some embodiments viewing a real object; FIG. 26 illustrates a flow chart amethod according to some aspects of the present technology; FIG. 27 illustratesHyperstereopsis in the binocularly optical waveguide combiner system according to some embodiments as a result of the displayed virtual object and real object misalignment; FIG. 28 illustrates Hypostereopsis in the binocularly optical waveguide combiner system according to some embodiments as a result of the displayed virtual object and real object misalignment; and FIG. 29 illustrates an exemplary computing device that may be used to implement some embodiments of the present technology. DETAILED DESCRIPTION OF PREFERRED EMBO- DIMENTS

[0033] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular embodiments, procedures, tech- niques, etc. in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced in other embodiments that depart from these specific details.

[0034] Applicant has identified herein one or more important technical problems with optical systems that are used in virtual reality (VR), mixed reality (MR) and / or augmented reality (AR) display, such as but not limited to headmounteddisplays, including near to eye displays. In AR / MR / VR display systems, cameras and sensors are configured tobuildupa3-Dmapof the realworld,which is then used to calculate imagery to describe the outside world in angle space. However, uncertainty in display angle and position leads to errors in projected boresight and parallax errors. For these reasons, AR and VR dis- plays require accurate registration of sensors and / or cameras to the projector / display system. In AR displays, virtual objects need to be in registration with real objects in the real world. For these reasons, accurate registration techniques need to be adopted in AR and VR displays.

[0035] Accurate mechanical registration may not be desirable or possible in some applications. There is a need to relax the physical registration requirements in applications where the cameras / sensors may move in- dependently from the display. One such example is a bomb disposal suit in which the display is attached to an innerhelmet of thesuit and thesensorsareattached toan outer visor which moves independently from the inner helmet. Furthermore, in some applications, mechanical tolerancing and build complexity issues further compli- cate physical registration. For example, stowable dis- 5 10 15 20 25 30 35 40 45 50 55 6 9 EP 4 657 140 A2 10 plays, or sensors associated with a head mounted dis- play that are designed to be moveable to reduce head borne weight, are inherently difficult to register.

[0036] Lack of accurate registration results in poor or unworkable devices. By way of non-limiting example, a processor connected to the AR display may be config- ured to cause the AR display to display virtual naviga- tional icons that would appear superimposed over the real world view observed through the waveguide, thus providing awearer of the ARdisplaywith visual guidance on how to reach a desired destination. The presentation of such navigational icons may accommodate a degree of error with respect to scale or sizewithin theARdisplay, since they do not relate to a real world object, but rather indicate turn left, right or go straight onwhenapproaching a junction. However, the GPS may suffer with general positional accuracy depending on the surroundings in which the wearer is seeking to navigate, in which case there may be some discrepancy in what the wearer sees through the display and the icons superimposed via the AR display. In other circumstances the AR system may be intended to superimpose virtual images, representing one of more aspects of real world objects, over the corresponding real world objects viewed through the device. For example, image data from a night vision sensor of poorly illuminated or dark terrain seen through an AR display can be superimposed on the AR display together terrain information from a depth imaging sensor to allow the wearer to navigate that terrain. Unlike the previous GPS icon driven navigation application exam- ple; using data from night vision or depth sensors re- quires more accurate calibration and registration of the AR display, the absence of which may result in a wearer colliding with an object in the real world being wrongly representedwithin the AR image. It has been recognized that accurate calibration and registration of the AR dis- play is needed.

[0037] Theaforementioned problems can bemitigated by aspects of the present technology which allow for optical systems and methods that provide improved or alternative approaches for device calibration and virtual model registration in virtual reality, mixed reality and / or augmented reality display systems. Optical systems and methods according to aspects of the present technology will now be described in more detail for use in some exemplary augmented reality and virtual reality display systems. However, it will be appreciated that in some other aspects and embodiments, the present technology can be configured for augmented reality, mixed reality and / or virtual reality systems other than those shown in the examples and even for other types of optical delivery systems.

[0038] Technical features described in this application can be used to construct various aspects of optical wa- veguide combiner systems and methods. In some ap- proaches, an optical display system has an optical wa- veguide combiner and one or more cameras. The one or more cameras may be optically coupled to an output of the optical waveguide combiner and have a field of view of one or more real objects and / or one or more virtual objects displayable by the optical display system. A plurality of wafer level waveguide cameras may be dis- posed outside the usable field of view of an output cou- pler. The plurality of waveguide cameras may be self- calibrated electronically using images captured by the waveguidecamerasof thevirtual object(s) displayableby the optical display system.

[0039] Furthermore, technical features described in this application can be used to construct various aspects of AR, MR and / or VR display registration methods and systems. In someapproaches, AR / VR / MR registration of devices and / or displayed virtual objects with real objects may be implemented using the images of the displayed virtual objects and real world objects captured by one or more cameras optically coupled to the waveguide com- biner(s) of the display device. Real object distance from the optical waveguide combiners and / or spatial location may be determined or estimated from the captured images.

[0040] Reference will now be made to the drawings in which the various elements of embodiments will be given numerical designations and in which embodiments will bediscussedsoas toenableoneskilled in theart tomake and use the present technology.

[0041] Specific reference to components, process steps, and other elements are not intended to be limiting. Further, it is understood that like parts bear the same reference numerals, when referring to alternate figures.

[0042] It will be further noted that the figures are sche- matic and provided for guidance to the skilled reader and are not necessarily drawn to scale. Rather, the various drawing scales, aspect ratios, and numbers of compo- nents shown in the figures may be purposely distorted to make certain features or relationships easier to under- stand.

[0043] Referring to FIG. 1 of the accompanying draw- ings, an optical display system 1 for a near-to-eye aug- mented reality display system according to one aspect of the present technology comprises an optical waveguide combiner 2 and a plurality of cameras 6,7. In the embodi- ment of FIG. 1, optical waveguide combiner 2 is a see- through combiner for augmented or mixed reality appli- cations. However, in some other embodiments, the op- tical waveguide combiner may be non-see-through such as for VR applications.

[0044] Optical waveguide combiner 2 comprises a wa- veguide optical substrate 10, an optical input coupler 3 on, in or adjacent thewaveguide optical substrate 10 and an optical output coupler 8 on or in the waveguide optical substrate 10. The optical input coupler 3 is optically coupled via a portion of waveguide optical substrate 10 to optical output coupler 8. An optical input, which com- prises collimated display image rays 5, is receivable by optical input coupler 3. The receivable display image rays 5 are input coupled by optical input coupler 3 into the waveguide optical substrate 10. Images rays 5 are 5 10 15 20 25 30 35 40 45 50 55 7 11 EP 4 657 140 A2 12 guidedalong thewaveguide tooutput coupler8bymeans of total internal reflection in the waveguide.

[0045] Waveguide optical substrate 10 is made from transparent waveguide materials such as but not limited to glass, plastic or other suitable materials. Waveguide optical substrate 10 is a slab or planar type waveguide substrate. In some other embodiments, waveguide op- tical substrate 10 may be curved or may have other shapesandconfigurationsandhavebent or curvedaxes.

[0046] Progressive extraction of display image rays 5 asoutput image rays9alongwaveguideoptical substrate 10 in the exit pupil expanding direction 4 is achieved by output coupler 8.Output coupler 8 output couples display image rays 5 to an eye 15 of an observer.

[0047] Various optical input couplers and coupling methods are possible. In the embodiment shown in FIG. 1, optical input coupler 3 is a diffractive input coupler integrated in the waveguide. In some other embodi- ments, optical input coupler 3 comprises or includes one or more diffraction optical elements separate from the waveguide substrate. In some other embodiments, optical input coupler 3 includes a reflective mirror incor- porated in the waveguide to reflectively couple optical input rays injected into the side of substrate into the waveguide. In yet someother embodiments, optical input coupler 3 includes one or more other optical coupling components integrated with or separate fromwaveguide optical substrate 10 for coupling optical input 5 into the waveguide. By way of example, such optical coupling components may be a prism or lens system.

[0048] Output coupler 8 is a diffractive output coupler. Diffractive output coupler comprisesdiffractive elements. The diffractive elements comprise gratings. In some other embodiments, the diffractive elements comprise photonic crystals. In some other embodiments, output coupler 8may be amirror or a series of mirrors / reflectors or cascaded mirrors / reflectors, prism arrays, holograms or a combination thereof.

[0049] The display virtual image formed by output rays 9 is a virtual object 13 displayable at image plane 12 at a spatial position relative to a real object 14 such that virtual object 13 and real object 14 are viewable through optical waveguide combiner 2 by eye 15 and appear super- imposed. In waveguide based displays, the selected initial focal plane is at infinity so asnot to forfeit resolution. This is because a waveguidemaintains angular informa- tion and not positional information. Typically, the dis- played image exists as a virtual image at image or focal plane 12 about threemetres in front of optical waveguide combiner 2. This is notionally equivalent to existing at an infinite distance due to the angular position the eyes would adopt being essentially the same when observing an object about 3 metres away versus for example one that is 100 metres or more away. In FIG. 1, virtual object 13 being displayed is a navigation icon but it will be understood that this is merely one example of a virtual object. In some embodiments, virtual object 13 repre- sents real object 14 that the virtual object is intended to overlay or superimpose.

[0050] A plurality of cameras 6, 7 are optically coupled to an output of the optical coupler. In the embodiment of FIG. 1, theplurality of cameras is incorporated in or on the optical waveguide substrate within a region of the output coupler 8. The plurality of cameras comprises a first camera 6 and a second camera 7 spaced apart from first camera 6. The cameras 6,7 may be spatially separated on or in proximity to thewaveguide surface such that they arewithin a portion of the field of view (FOV) of the output coupler 8. In someother embodiments, the camerasmay be located in other positions and / or there may be one camera or more than two cameras. Each camera is an ultra small camera having dimensions in the mm or sub mm region such as a wafer level camera module. One non-limiting example of such as camera is OVM6948 CameraCubeChip (dimensions being 0.65mm x 0.65mm x1.16mm).

[0051] FIG. 2 is a perspective view of optical display system1 illustrating overlapping fields of view of different elements of the system. Although not drawn to scale, FIG. 2 illustrates the relationship between the overlap- ping regions of the field of views of the first camera 6, second camera 7 and eyebox 16. Output coupler 8 has a surface area usable by eyebox 16 that is less than the entire surface area of output coupler 8. Eyebox 15 has a field of view 17 at the output coupler 8 (usable surface area of the output coupler 8) and a field of view 21 at image plane 12 of virtual object 13. Each camera 6,7 is positioned in a region of output coupler 8 beyond the eyeboxuseableareaof theoutput coupler, that is, outside the eyebox field of view 17 of output coupler 8 such that the cameras are not viewable from eyebox 16. First camera 6 has a first field of view 18 at image plane12. Second camera 7 has a second field of view 19 at image plane 12. Each camera 6,7 is arranged spaced apart with a point of view such that, at image plane 12, first field of view 18 is adjacent to and overlaps second field of view 19. First and second cameras 6,7 therefore have a com- bined field of view at the image plane which comprises first field of view 18 and second field of view 19 andwhich includes an overlapping region 20 of first field of view 18 and second field of view19. Field of view21of the eyebox at image plane 12 is substantially within the combined field of viewof first and second cameras 6,7. Splitting and arranging camera fields of view in this manner enables ultra small cameras, such as wafer level cameras, to be adopted for camera imaging in optical waveguide com- biner 2 and allows electronic self calibration of the cam- eras. Furthermore, since waveguide cameras 6,7 have a field of view of both the virtual object 13 and real world object 14, registration of virtual object 13 with real object 14 and / or registration of the system with external sen- sors / cameras is possible using waveguide cameras 6,7.

[0052] Each camera 6,7 is positioned such that it has a field of view equivalent to more than 50% of the field of view of a projector which projects the display image rays 5 into combiner 2. Whilst cameras 6,7 are shown in the 5 10 15 20 25 30 35 40 45 50 55 8 13 EP 4 657 140 A2 14 accompanying figures as being arranged symmetrically with similar FOVs, in other embodiments, the cameras maybearrangedasymmetrically and / ormayhave field of viewsdifferent fromeachother and / or theremaybemore than two cameras (provided the images captured by the cameras can be stitched to form the larger virtual image). In some embodiments, one of the cameras has a field of view that is greater than 50% of the field of view of the projector and the field of view of the other cameramay be less than 50%of the projector field of view provided there is an overlapping region 20 sufficient for image stitching. By way of non-limiting example, for the case of two cameras, one camera may have a field of view of 31% of the projector field of view and another 71% giving 100% and some overlap to complete image stitching.

[0053] In someotherembodiments, onlyonecamera is optically coupled to output coupler 8 of optical combiner 2 and field of view 21 of eyebox 16 at image plane 12 is substantially within the field of view of the camera. In yet some other embodiments, only part of field of view 21 of eyebox 16 in which virtual object(s) 13 is displayable at image plane 12 is within the field of view of the camera or within the combined field of view of the cameras. In yet some other embodiments, the field of view of the camera or the combined field of view of the cameras at the image plane substantially matches the field of view of the eye- box.

[0054] In some alternative embodiments, the one or more camera(s) is optically coupled to the exit-pupil expansion output of the output coupler. In some other embodiments, the one or more camera(s), optically coupled to the output of the output coupler, is located adjacent or remote from the waveguide substrate rather than being incorporated directly in or on the waveguide substrate. For example, in someembodiments theoneor more camera(s) is located in the eyebox or even behind the eyebox.

[0055] As best shown in FIG. 3, which illustrates a perspective view of optical waveguide combiner 2 ac- cording to someembodiments, output coupler 8 canhave a pre exit pupil expansion region 22 and an exit pupil expansion region 23. In the embodiment shown in FIG.3, output coupler 8 is a combined 2D expansion and output grating on a major surface or within the waveguide con- figured to receive image rays5 from input coupler 3 (input grating) and display the image. In FIG. 3, eyebox 16 is projected to the output grating for a given field of view. Theoutput grating is a crosseddiffractive grating inwhich twodiffractivegratingsoverlayeachother. Twodiffractive gratings are entirely overlaid on one another in or on the waveguide, wherein each of the two diffractive optical elements is configured to receive light from the input diffractive grating and couple it towards the other diffrac- tive grating which can then act as an output diffractive grating providing outcoupled orders towards a viewer. One example of an optical waveguide combiner using such an output grating is recited in US Patent 10114220B2, filed July 28 2015 and issued October 10 2018 in the name of Wave Optics Ltd and entitled "Exit pupil expanding diffractive optical waveguiding device", the entire contents of which is incorporated herein by reference. In some embodiments, the diffractive optical elements of the diffractive output coupler can be a photo- nic crystal rather than diffraction gratings.

[0056] Cameras 6,7 are located outside a regular usable area of output coupler 8, where the complete field of view is not available. Extracted image rays from pre- expansion area are not used for viewing the virtual object image from the eyebox. The output grating causes re- plication of the entrance pupil at the input of the wave- guide combiner into a plurality of exit pupils. The pre- expansion region causes initial spreading of the pupil of image rays across the surface of the output grating to replicate the entrance pupil and the expansion region expands the exit pupil so that the output grating directs image rays towards the eyebox for receiving by an eye of the observer. The pre-expansion region is defined as where pupil replication has not yet fully filled the eye box, but some incomplete image information is still pro- jected ’forward’. As the eye of a person looking through the waveguide moves through a range of angles in the eyebox, for example looking up, down, left, right, be- cause eyebox 16 carries multiple exit pupils of image rays, the person perceives the image across the entire area of the eyebox as their eye scans through a range of viewing angles.

[0057] In some embodiments, an optical waveguide combiner 2 and camera(s) 6,7 of any embodiments dis- closed herein forms an eyepiece for a pair of eyeglasses, for example, as shown in FIGS. 4 and 5. A projector 25 is arranged to inject image rays into input coupler 3. One non-limiting example of such a projector is shown in FIG. 6. In some embodiments, each camera 6,7 is a wafer level camera. By way of example, first camera 6 and second camera 7 are about 2.5mm x 4mm in volume (assuming use of Omnivision Ov01A10 laptop webcam module for the cameras and sold under the registered trade mark OMNIVISION™ of OmniVision Technologies, Inc. These are just merely some examples of the ultra small camera sizes. Other sizes and types of known and / or future developed ultra small cameras are envi- saged.

[0058] In FIG. 4, optical wave combiner 2 is orientated such that input coupler 3 is disposedwithinanendportion 42 of eyepiece 44. The area designed to be outside the field of view of eyebox 16 (in this case the pre-expansion region 22) is proximal to end portion 42 of the eyepiece. The output coupler area that is designed to be inside the usable field of view of eyebox 16 is spaced from end portion 42. The projector may either be disposed on the opposite side of the eyepiece facing away from the eye or it may be disposed on the same side of the eyepiece according to the mode of image introduction to input coupler 3. Typically, an image pupil is introduced to input coupler 3 in a direction orthogonal to the surface of waveguide combiner 2. The barrel of a projector may 5 10 15 20 25 30 35 40 45 50 55 9 15 EP 4 657 140 A2 16 be oriented essentially orthogonal to input coupler 3, or essentially parallel to input coupler 3, inwhich case a turn prism is used to direct light from the projector into input coupler 3. Another optical waveguide combiner 2, ar- ranged as shown in FIG. 4, can be provided as an eye- pieceof thepair of eyeglasses toprovidebinocular vision.

[0059] In some aspects of the present technology, any one of the optical display systems comprising one or more optical waveguide combiners 2 and optionally one or more cameras 6,7 of the embodiments described herein may be implemented in a near-eye optical display system having an eyeglass form factor or head set form factor. In some embodiments, the near-eye optical dis- play systemadditionally includes a light engine (projector or other light engine) and a battery. The near-eye optical display system may be an AR or MR optical display system.

[0060] By way of example, reference is made to FIGS. 4 and 5 taken in conjunction with FIG. 6, which is a perspective view of a pair of glasses 40 according to some embodiments. Glasses 40 include eyepieces 44 which include optical waveguide combiners 2 and cam- eras 6,7. Projectors 25 are arranged within the temples / arms of glasses 40, and input couplers 3 are oriented towards the outermost edges of the eyepieces 44 in end portions 42 to be orthogonal to respective projectors 25 located in the arms of glasses 40. Since the input cou- plers 3 are provided in end portions 42 of the eyeglasses, light can be projected into respective input couplers 3 from projectors 25 and can be directed towards output couplers 8 of eye pieces 44. Output couplers 8 of eye- pieces 44 expand the light in two dimensions and can couple it out of thewaveguide towards the user’s eyes. In some other embodiments, projectors may be aligned along the top of the frame which supports eyepieces 44; in which case a turn prism is used to couple image light into input couplers 3 whichmay be oriented beneath the top section of the frame supporting eye pieces 44.

[0061] Projector 25 shown in any of the embodiments disclosed herein may include any type of suitable near eye image projector whichmay include optics configured to generate a display image and output the display image collimatedat infinity intoopticalwaveguide combiner2. In some embodiments, projector 25 is a wafer level projec- tor.Bywayofnon-limitingexamples, theprojectormaybe a LED, LCD display, or laser based device or other light source device for generating thedisplay image.Projector 25may include a LCoS,DMD,OLEDor any other display panel that uses an emissive, transmissive or reflective display technology. The projector includes any optics required to collimate the display image to infinity such as a collimating lens. In some embodiments, the projec- tor does not include any optics (lenses, mirrors) such as for example in the case of phase only (holographic) dis- plays. In some embodiments the projector includes a self emitting micro LED display.

[0062] Reference will now be made to methods of camera self calibration in VR / AR / MR displays according to some aspects of the present technology. For the purpose of explaining such methods, reference is first made to FIGS. 7A, 7B, which are schematics illustrating existing theodolite calibration techniques.When calibrat- ing physically dislocated systems for boresight, it is ne- cessary to find a common reference. Often a pair of theodolites is used, first referencing to each other, then synchronously providinga reference to twoother objects. By way of non-limiting example, Theodolite 71 picks up angular reference 0° and Theodolite 72 picks up angular reference 180° (see FIG. 7A). Theodolite 71 turns to angular reference 280° and Theodolite 72 turns to angu- lar reference 280° (FIG. 7B).

[0063] As illustrated in FIG. 8, optical waveguide sub- strate 10 of optical waveguide combiner 2, inherently creates multiple references so that disposing the cam- eras on the waveguide substrate aligns the cameras without having to use calibration techniques, such as for example the aforementioned theodolite calibration techniques, to physically reference each other for bore- sight. Any misalignment of the fields of view 18,19 of first camera 6 and second camera 7 with the field of view of eyebox 16 at image plane 12 can be corrected electro- nically.

[0064] A method 90 of camera self-calibration of an VR / AR / MR display according to one aspect of the pre- sent technology will now be described with reference to figure 9. A display image including fiducials is projected into an optical waveguide combiner (step 91). First and second cameras are optically coupled to an output of the optical waveguide combiner, the first and second cam- eras having a combined field of view comprising a first field of view of the first camera and a second field of view of the second camera and including an overlapping re- gion of the first field of view and the second field of view (step 92). The display image is displayed by the optical waveguide combiner at the image plane as a virtual image with the fiducials being in the overlapping region of the field of views of the first and second cameras (step 93). The first waveguide camera captures a first image of the virtual image (step 94). The second waveguide cam- era captures a second image of the virtual image (step 95). The captured first image and second image are electronically stitched together. In order to combine the first and second images to correspond to the displayed virtual image, the captured first image and captured second image are electronically adjusted by moving the captured first image and the captured second image to align the common fiducials (step 95). In some embodi- ments, further correction of the combined image is achieved by determining the overlapping region of the field of views by comparing the adjusted first and second images with the display image projected into the combi- ner (step 97). The adjusted first and second images are electronically corrected according to the determined overlapping region of the field of views (step 98). The corrected stitched first and second images substantially match the display image being projected into the optical 5 10 15 20 25 30 35 40 45 50 55 10 17 EP 4 657 140 A2 18 combiner.

[0065] By method 90, a substantially full undistorted image of the virtual image at the image plane is derived from the first image captured by the first camera and second image captured by the second camera even when the first field of view of the first camera and the second field of viewof the second camera aremisaligned at the image plane. Consequently, calibration of the first and second cameras to allow for camera misalignment tolerances is achievable electronically without mechan- ical registration of the cameras or the use of additional sensors. Thecalibrated camera systemcan thenbeused for determining and correcting errors in registration of a displayed virtual object and a real object (so to correct virtual models of the real world). In some embodiments, method 90 is performed in real time during use of the optical display systembya user to account for any further misalignments caused during use of the display system.

[0066] Method 90 may be implemented in any one of the embodiments of the optical display systems dis- closed herein or any other optical display system using an optical waveguide combiner in which a plurality of cameras are optically coupled to the output coupler of the optical waveguide combiner. In some embodiments of method 90, a plurality of cameras are optically coupled to a pre-exit pupil expansion output of the output coupler.

[0067] By way of example, some implementations of method 90 in optical display system 1 according to some embodiments will now be described with reference to FIGS. 10 to 15.

[0068] FIG. 10 is a schematic of the optical display system showing FOVs 18,19 of first and second wave- guide cameras 6,7 of optical waveguide combiner 2 and fieldof view25ofprojector25.Anon-limitingexampleofa projector display image with fiducials or common fea- tures 27 is shown in FIG.11. The alignment fiducials 27 are provided on a central vertical line through the image at upper and lower positions. FIG. 11 is just one non- limiting example of the size and position of alignment fiducials or common features 27. The position of the alignment fiducials or common features 27 are arranged for display by combiner 2within theoverlapping region20 of first and second field of views of cameras 6,7 (see for example FIG. 1 or 2). Consequently, if the overlapping region 20 is in a different position rather than centrally located (due to the cameras having different sized FOV etc. or more than two cameras being used), the position of alignment fiducials or common features 27 will also be in a corresponding different location rather than centrally located.

[0069] The minimum size of alignment fiducials 27 is determined by the correction limit of cameras 6,7. In some embodiments, method 90 is performed using sin- glepixel registration inwhicheachfiducial 27 is thesizeof a single pixel. By way of example, the Camera FOVmay be 25x30, 960x1080 and projector FOV may be 50x30 1920x1080). In someembodiments,wafer level cameras are used for first and second cameras 6,7 with correction limits of less than 2 pixels (see for example omnivision camera sensor OV01A10, 1 / 11" 1.16um 1280x800, 2.5x2mm package size and sold under the registered trade mark OMNIVISION™ of OmniVision Technologies, Inc.) or less than 3 pixels (see for example omnivision camera sensor 0H0TA / 31" 1.008um 400x400 550x550umpackage and sold under the registered trade mark OMNIVISION™ of OmniVision Technologies, Inc.).

[0070] First image 13’ captured by camera 6 with FOV 18andsecond image13" capturedby camera7withFOV 19 are stitched together and include double imaged fiducials or common features as shown in FIG. 12. First and second images 13’, 13" in the image of FIG. 12 are electronically moved to align the misaligned fiducials 27’ and 27" and stitch the images together as shown in FIG. 13. In some embodiments, movement of first and second images13’,13" is achievedby rotational alignment,which manipulates each image to ensure the fiducials become superimposed. The projector field of view can be com- pared to the adjusted stitched image for determining the common area of first and second images 13’, 13" which represents the overlapping region 20 of first field of view 18 and second field of view 19. FIG. 14 illustrates first image 13’ and second image 13" of the stitched image in FIG. 13 corrected electronically according to the deter- mined overlapping region 20. Once this correction of the camera views has been performed, a unified "virtual camera" image canbe produced. The resulting corrected image corresponds substantially to an undistorted image that is viewable fromeyebox 16. Themisaligned first field of view 18 of first camera 6 and second field of view 19 of second camera 7 is effectively corrected without the cameras having to be physically moved. This corrected image can then be used as the computational feedback for determining and correcting errors in registration of a displayed virtual object and a real object (so to correct virtual models of the real world).

[0071] In some embodiments, first field of view 18 of first camera 6 and second field of view 19 of second camera 7 are sized such that the corrected stitched image is within a perimeter margin of the projector field of view to enable alignment correction of the first image and the second image within the alignment tolerances. By way of example, the area of the corrected image in FIG. 14 when compared to the projector field of view in FIG.11hasanarea that iswithinaperimetermarginof the projector field of view.

[0072] It will be understood that the FOVs 18 and 19 of respective cameras 6,7 are not limited to those shown in the figures. As previously mentioned, the FOVof camera 6 may be different from the field of view of camera 7. Furthermore, as previously indicated, the combined FOVs of camera 6,7 may be larger than the projector field of view. Yet furthermore, the projector field of view may be substantially within the combined field of view of first and second cameras 6,7. In some embodiments, only a portion of the projector field of view which includes a real object and a displayable virtual object for registra- 5 10 15 20 25 30 35 40 45 50 55 11 19 EP 4 657 140 A2 20 tion with the real object may be within the combined field of view of the cameras.

[0073] FIG. 15 is a block diagram of the optical display system including theopticalwaveguide combiner system 1 and projector 25. Processor(s) 302 are operably con- nected to cameras 6,7 and projector 25. It will be appre- ciated that the processor(s) described herein and illu- strated in the accompanying drawings may be located locally with the cameras / projector and / or located there- from and operably connected to the cameras / projector via a suitable network which may be wireless and / or wired.

[0074] Processor(s) 302, when executed by computer program instructions, causes the optical display system to perform method 90 or other camera calibration meth- ods disclosed herein to account for any misalignment of first and second cameras 6,7.

[0075] Referencewill nowbemade to variousmethods and systems for VR / AR / MR display registration accord- ing to some aspects of the present technology.

[0076] A method 160 of registering an AR / VR / MR dis- play according to one aspect is illustrated in FIG. 16. One or more virtual object(s) is provided or generated from one or more real object(s) using external sensors (step 161). A display image of the virtual object(s) is projected into an optical waveguide combiner (step 162). The optical waveguide combiner displays the display image as the virtual object(s) (step 163). An image of the dis- played virtual object(s) and real object(s) is captured using one or more optical camera(s) optically coupled to an output coupler of the optical waveguide combiner. The camera(s) has a field of view of the real object(s) and of the displayed virtual object(s) at the image plane of the combiner (step164). Thecaptured real object(s) image is compared with the captured virtual object(s) image. In some embodiments, a displayed virtual object in the captured image is individually compared with one or more real object(s) in the captured image. In some em- bodiments, at least some or all of the virtual objects in the captured image are grouped together and compared to a real object or a corresponding group of real objects. A registration error between the real object(s) and the dis- played virtual object(s) is determined based on the com- parison (step 165). The virtual image being displayed is adjusted (step 167) to substantially correct the registra- tion error (step 166).

[0077] Method 160 may be implemented in any one of the embodiments of the optical display systems dis- closed herein or any other optical display system using an optical waveguide combiner in which one or more camera(s) is optically coupled to the output of the optical waveguide combiner. Furthermore, each cameramaybe arranged in theopticalwavecombineraccording toanyof the embodiments herein. In some embodiments ofmeth- od 160, one or more of the camera(s) is optically coupled to a pre-exit pupil expansion output of the output coupler.

[0078] By way of example, FIGS. 17 and 18 illustrate an optical display system according to some embodi- ments inwhichmethod160maybe implemented.FIG.17 is a schematic of optical waveguide combiner 2 display- ing a virtual object at the image plane that is misaligned with the real object. FIG. 18 is a block diagram of the optical display system including the optical waveguide combiner 2, projector 25 and external sensor(s) 31. Processor(s) 302, when executed by computer program instructions, causes theoptical display system toperform the method 160 to account for any misalignment or mis- registration of displayed virtual object 13 and real object 14

[0079] Determination of the registration error from the comparison of virtual object 13 and real object 14 can be performed in various ways. In some embodiments, initi- ally, a common fiducial in the real world that external sensor(s) 31 and the waveguide camera 6 can identify is determined by the optical display system or a user of the system. By way of non-limiting example, a common feature identified to serve as the common fiducial can be a detected edge, projected point, perhaps a reference LED on glove or boot. Processor(s) 302, when executed by computer program instructions, uses the external sensor(s) sensing and / or imaging of its reference loca- tion, that is, the common fiducial, to generate the virtual object for display at an image plane of the optical combi- ner 2. The virtual object exists in a virtualmodel produced by processor 302. Waveguide camera system identifies the real world feature from the image(s) being captured by thewaveguide camera(s) and compares the identified real world feature to the common fiducial of the virtual object being displayed to determine the error in registra- tion between the real world object 13 and displayed virtual object 14. Processor(s) 302, when executed by computer program instructions, can then adjust any one or combination of the position, size, and shape of the virtual image being displayed to substantially correct the registration error. This can be achieved by processor(s) 302, when executed by computer program instructions, adjusting virtual object 13 for display by optical wave- guide combiner 2 according to the determined registra- tion error to display virtual object 13 with a position, size, and / or shape in which the common feature or fiducial identified in the real object and the common feature or fiducial identified in the virtual object substantially coin- cide as viewed from the eyebox / cameras.

[0080] In some embodiments of any one of the meth- ods and / or systems for AR / VR registration disclosed herein, includingmethod 160, generation of virtual object 13 for display can be performed by the optical display system without the use of external sensors / cameras of the AR display. In some embodiments, a predetermined virtualmodel of the realworld is provided to the system. In someotherembodiments,waveguidecameras6,7 them- selves are used to generate virtual object 13 for display by the system extracting virtual object 13 from the image of the real world being captured by the waveguide cam- eras. FIG. 19 illustrates a block diagram of an optical display systemaccording to someembodiments inwhich 5 10 15 20 25 30 35 40 45 50 55 12 21 EP 4 657 140 A2 22 method 160 is implemented with step 161 modified to provide the virtual object with known size using a pre- determined virtual model 32 to generate virtual object 13 rather than generating virtual object 13 using external sensors of the AR display. Processor(s) 302, when exe- cuted by computer program instructions, causes the optical display system to perform method 160.

[0081] A method 200 of registering an AR / VR / MR dis- play according to another aspect is illustrated in FIG. 20. One or more virtual object(s) of known size for display is provided generated (step 201). The estimated distance and / or estimated spatial location of real object(s) from an optical waveguide combiner is determined (step 202). Based on the estimated distance and / or estimated spa- tial location, the virtual object(s) is scaled for display by the optical waveguide combiner (step 203). A display image of the scaled virtual object(s) is projected into the optical waveguide combiner (step 204). The display image is displayed by the optical waveguide combiner as the scaled virtual object(s) (step 205). The image of the displayed virtual object(s) and real object(s) is captured using one or more optical camera(s) optically coupled to an output coupler of the optical combiner. The camera(s) have a field of view of the real object(s) and of the displayed virtual object(s) (step 206). The captured real object(s) is compared to the captured virtual object(s) (step 207). A registration error between the real object(s) and the displayed virtual object(s) is determined based on the comparison (step 208). The virtual object(s) being displayed is adjusted to substantially correct the regis- tration error (step 209).

[0082] Method200maybe implementedusinganyone of the embodiments of the optical display systems dis- closedhereinoranyotheroptical displaysystemusingan optical waveguide combiner in which one or more cam- era(s) is optically coupled to an output coupler of the optical waveguide combiner. Furthermore, each camera may be arranged in the optical wave combiner according to any of the embodiments herein. In someembodiments of method 200, a plurality of cameras is optically coupled to a pre-exit pupil expansion output of the output coupler.

[0083] For example, method 200may be implemented by the optical display systemof FIG. 18 or FIG.19accord- ing to some embodiments in which processor(s) 302, when executed by computer program instructions, causes the optical display system to perform method 200.

[0084] By way of example, FIG 21 illustrates an exam- ple implementation of method 200 using the optical dis- play system FIG. 19 according to some embodiments. FIG. 21 is a schematic of opticalwaveguide combiner 2of the optical display system displaying a virtual object 13 that is misaligned with real object 14. As illustrated, the system initially displays virtual object 13 according to the estimated distance and / or spatial location of real object 14 from optical waveguide combiner 2. This estimated distance and / or estimated spatial location of real object 14 from optical waveguide combiner 2 may be deter- mined using a depth sensor. In some embodiments, the estimateddistanceand / or estimated spatial locationmay be determined using waveguide cameras themselves without anyexternal sensors, aswill beexplained inmore detail below with reference to some embodiments. Wa- veguide cameras 6,7 are used to compare the angular size, of real object 14 and displayed virtual object 13. The difference in angular size of displayed real object 14 and virtual object 13 reflects the registration error between virtual object 13 and real object 14. In this case, the estimate of distance to real world object is wrong. The correction in distance to real world object is calculated from the differential. Since the size of virtual object 13 is predetermined, virtual object 13 is scaled for the cor- rected distance of real object 14 and displayed by the optical combiner system based on the corrected dis- tance. Processor(s) 302 determines and applies a cali- bration factor such that images projected into the wave- guide are accurately scaled to match the real world. Following such calibration, virtual object 13 and real world object 14 are superimposed as viewed from eye- box / camera. The optical display system and / or method effectively achieves fitting of the real world to the virtual model 32 that has known dimensions.

[0085] In some aspects, methods and systems for AR / VR display registration according to any embodi- ments described herein with reference to a single optical combiner 2 can be implemented in binocular configura- tions comprising right and left optical combiners 2. The waveguide cameras capture a displayed stereo virtual image and a stereo real world image and compare the displayed stereo virtual image and a stereo real world image to determine the registration error. Such a regis- tration error may for example be the result of misalign- ment of the optical wave combiners. The stereo virtual image being displayed can be corrected by adjusting the virtual imagesdisplayedby theoptical combinersaccord- ing to the registrationerror so that the stereovirtual image is displayed correctly.

[0086] Determining the actual location in space of the real object is necessary to perform accurate VR / AR / MR display. A computer can hold a 3-Dimensional model of theworld, where all dimensions of theworld are known to a good accuracy. This world can, by making a number of assumptions about a virtual position in space, be ren- dered in 2 dimensions and provided to the user bymeans of a display such as a computer monitor, virtual reality or augmented reality display. For the case of augmented reality, there often arises a problem in that the displayed 2D dimensional image of the world may not match the real world. This is because the assumptions made about the virtual position in space, do not match the actual location in space. Much effort is made with a number of sensors (318 in FIG. 29) including GPS, LIDAR, RA- DAR, Time of flight (ToF; 290 in FIG. 29) and many other sensors to determine this position with accuracy.

[0087] A method 220 of registering a VR / MR / AR dis- play according to one aspect of the present technology is 5 10 15 20 25 30 35 40 45 50 55 13 23 EP 4 657 140 A2 24 illustrated in FIG. 22. Images are captured using first camera(s) and second camera(s) optically coupled to respective outputs of first and second optical waveguide combiners of a binocular display system. Each camera has a field of viewof one ormore real object(s) and one or more displayable virtual object(s) (step 221). An esti- mated distance and / or estimated spatial location of the real object(s) from the optical waveguide combiners is determined using the optical cameras (step 222). The virtual object(s) for display by the optical waveguide combiner is generated based on the estimated distance and / or estimated spatial location (step 223). A display image of the virtual object(s) is projected into the optical waveguide combiners (step 224). The display image is displayed by the optical waveguide combiners as the virtual object(s) (step 225). An image of the displayed virtual object(s) and real object(s) is captured using the optical cameras (step 226). The displayed virtual ob- ject(s) is compared with the real object(s) using the captured image (step 227). A registration error between the real object(s) and the displayed virtual object(s) is determined based on the comparison (step 228). The virtual object(s) being displayed is adjusted to substan- tially correct the registration error (step 229).

[0088] Method 220 may be implemented in any one of the embodiments of the optical display systems dis- closed herein arranged, in a binocular configuration or in any other optical display system using optical wave- guide combiners in a binocular system. The first optical waveguide combiner and / or the second optical wave- guide combinermay be any one of the optical waveguide combiners of the embodiments disclosed herein. Furthermore, each first cameramay be optically coupled to the first optical waveguide combiner according to any of the embodiments herein. Yet furthermore, each sec- ond camera may be optically coupled to the second optical waveguide combiner according to any of the embodiments herein. In some embodiments of method 220, for eachopticalwaveguide combiner, one ormore of camera(s) is optically coupled to a pre-exit pupil expan- sion output of an output coupler of the optical waveguide combiner.

[0089] Byway of example, an implementation of meth- od 220 according to some embodiments will now be described with reference to FIGS. 23 which illustrates ablockdiagramofabinocularoptical displaysystem.The binocular system comprises optical waveguide combi- ners 2L,2R, projectors 25 and processors 302. Proces- sor(s) 302, when executed by computer program instruc- tions, causes the binocular optical display system to perform method 220.

[0090] The process of determining the estimated dis- tance and / or estimated spatial location of the real object (s) from the optical combiners using the optical cameras (step 222) can be performed in different ways according to different embodiments.

[0091] In some embodiments of the present technol- ogy, the estimated distance of the real object(s) from the optical waveguide combiners using the optical wave- guidecameras (step222) is determinedbasedonadepth perception approach. In other embodiments, the esti- mated distance of the real object(s) from the optical waveguide combiners using the optical cameras (step 222) is determined based on stereopsis.

[0092] Furthermore, when the optical waveguide com- biners are used in a binocular configuration (one for the right eye and one for the left eye), implementingmethods and / or systems of registration and / calibration of any of the embodiments disclosed herein for each optical wa- veguide combiner enables electronic correction of the images displayed by each waveguide to ensure correct alignment of the combined binocular image. As a con- sequence, a person looking through the pair of wave- guides will perceive the augmented image correctly; if there were no self correction system present the person would not perceive an accurate binocular image andmay in fact experience headaches due to themisalignment of the images projected by each waveguide.

[0093] Method 240 of determining distance in an AR / MR / VR display according to one aspect of the pre- sent technology is illustrated in FIG. 24. Images are captured in stereo using first camera(s) and second camera(s) optically coupled to respective outputs of first and second optical waveguide combiners of a binocular display system. The cameras having a field of view of one or more real object(s) (step 241). One or more virtual object(s) is extracted from the captured images of the real object(s). The extracted virtual object(s) is recognized electroni- cally (step 242). Depth perception cues are determined from the images captured by the optical cameras (step 243). The distance of the real object(s) from the optical combiners is determined (step 245) based on the deter- mined depth perception cues and object recognition (step 244).

[0094] Method 240 may be implemented in any one of the embodiments of the optical display systems dis- closedherein (or inanyotheroptical displaysystemusing optical waveguide combiners in a binocular system in which one or more camera(s) is optically coupled to respective outputs of the optical waveguide combiners. The first optical waveguide combiner and / or the second optical waveguide combiner may be any one of the optical waveguide combiners of the embodiments dis- closed herein. Furthermore, each camera may be ar- ranged in the optical waveguide combiner according to any of the embodiments herein. Yet furthermore, in some embodiments, for each optical waveguide combiner, the one or more camera(s) is optically coupled to a pre-exit pupil expansion output of the output coupler of the optical waveguide combiner.

[0095] Byway of example, according to some embodi- ments,method 240may be implemented in the binocular optical display system of FIG. 23. Processor(s) 302, when executed by computer program instructions, causes the binocular optical display system to perform 5 10 15 20 25 30 35 40 45 50 55 14 25 EP 4 657 140 A2 26 method 240.

[0096] In step 244, depth perception cues can be de- termined from projection angles of the waveguide cam- eras. By way of example, FIG. 25 illustrates exemplary differences in projection angles of eyes / cameras incor- porated in right and left optical waveguide substrates 2L and 2R of the optical display system of FIG. 23, viewing real object 14, for the purpose of object recognition and determining the depth perception cues according to some embodiments. Depth perception cues determined from the images captured by the optical waveguide cam- eras can comprise or include the following: 1st Cue (Strongest) (Close to long distance), object size Vs ex- pected object size (for example Big Cow = close, Small Cow = far); 2nd Cue (close to moderate distance) object vergence, discrepancy in angle between the eyes / ca- meras. Always inward - convergence; 3rd (weak) cue (close only) object accommodation the eye / camera has to focus.

[0097] In some aspects of the present technology, method 240 is employed in any one of the methods of registering ARNR / MR displays of the embodiments dis- closed herein to estimate or determine the distance or spatial position of the real object from the optical combi- ners.Bywayof example, processstep222maycomprise method 240.

[0098] Furthermore, object recognition techniques employed in method 240 may be adopted in any of the registration / calibration methods disclosed herein for generating a predetermined virtual object for display.

[0099] Method 260 for AR / MRNR display based on stereopsis according to another aspect of the present technology is illustrated in FIG. 26. One or more virtual object(s) for display is generated or provided (step 261). A display image of the virtual object(s) is projected into first and second optical waveguide combiners of a bino- cular optical combiner system (step 262). The display image is displayed in stereo, by the optical waveguide combiners, as a virtual object(s) (step 263). Stereo images are captured using first camera(s) and second camera(s) optically coupled to respective outputs of the first and second optical waveguide combiners. The cam- eras have a field of view of the displayable virtual ob- ject(s) and real object(s) (step 264). The displayed virtual object(s) is compared to the real object(s) using the captured stereo images (step 265). The distance and / or spatial location of the real object(s) from the optical waveguide combiners is determined (step 267) based on the comparison of the displayed virtual object(s) compared to the real object(s) using the captured stereo images (step 266).

[0100] In some embodiments of step 265, a displayed virtual object in the captured image is individually com- pared with a real object or a group of real objects in the captured image. In some embodiments, at least some or all of thevirtual objects in thecaptured imagearegrouped together and compared to a real object or a correspond- ing group of real objects.

[0101] In some embodiments, step 266 is performed by making a primary estimate of the distance (if object of known size) by determining from the stereo images the difference in angular extent from the cameras of the displayed virtual object relative to the real object. In some embodiments, a secondary estimate of distance can be calculated from the stereo images for close objects based on stereoscopic discrepancy (binocular disparity).

[0102] Using the determined distance and / or spatial location, the virtual object being displayed is adjusted by scaling to substantially correct the difference in spatial position between the displayed virtual object and the real object.

[0103] Method 260 may be implemented in any one of the embodiments of the optical display systems dis- closed herein or in any other optical display system using optical waveguide combiners in a binocular system in which one or more camera(s) is optically coupled to an output coupler of the respective optical waveguide com- biner. The first optical waveguide combiner and / or the second optical waveguide combiner may be any one of the optical waveguide combiners of the embodiments disclosed herein. Furthermore, each camera may be optically coupled to the optical waveguide combiner ac- cording to any of the embodiments herein. In some embodiments, for each optical waveguide combiner, one or more camera(s) is optically coupled to a pre-exit pupil expansion output of an output coupler of the optical waveguide combiner.

[0104] By way of example, method 260 according to some embodiments may be implemented by the optical display system of FIG. 23. Processor(s), when executed by computer program instructions, causes the binocular optical display system to perform method 260.

[0105] In step 266, the binocular disparity in the stereo images is determined according to whether there is a Hyperstereopsis or Hypostereopsis viewing condition. By way of example, FIG. 27 illustrates example differ- ences of projection lines of cameras incorporated in right and left opticalwaveguide combiners2L, 2Rof theoptical display system of FIG. 23 stereo viewing real object 14 and virtual object 13 according to some embodiments. Cameras 6,7 see a larger angle discrepancy than ex- pected when displayed virtual object 13 is too close to optical waveguide combiners 2L, 2R compared to real object 14 as shown in FIG. 27 (Hyperstereopsis). Cam- eras 6,7 see a smaller angle discrepancy than expected when displayed virtual object 13 is too far from optical waveguide combiners 2L, 2R compared to real object a14s shown in FIG. 28 (Hypostereopsis). The distance of real world object 14 from combiner 2 can be estimated from the virtual stereo error.

[0106] In some aspects of the present technology, method 260 is employed in any one of the methods of registering AR / VR / MR displays of the embodiments dis- closed herein to estimate or determine the distance or spatial position of the real object from the optical combi- ners.Bywayof example, processstep222maycomprise 5 10 15 20 25 30 35 40 45 50 55 15 27 EP 4 657 140 A2 28 method 260.

[0107] An exemplary use of the methods and optical display systems of some embodiments disclosed herein is a firefighter navigating through a poorly lit and smoke filled room. The firefighter may be equipped with a pro- tective helmet that comprises an array of external sen- sors, including imagedepth sensor, thermal sensor, night vision camera, camera along with augmented reality waveguides comprising optical waveguide cameras on the inside of the helmet. Image data from the external sensors may be combined and presented on the wave- guide display, enabling the firefighter to safely navigate through theenvironment, even invery low light,where the eyes may be unable to distinguish objects, in particular those that are low to theground thatmight represent a trip hazard. Because the system constantly updates scaling parameters, the risk of an object appearing to be at a particular location based on the projected image from the waveguide, when in reality it is at a different location, or of an object appearing to have a certain size that is different to reality, is greatly reduced. In this instance the firefighter is less likely to have a collision with an object than might otherwise be the case. The systems and methods of the embodiments will ensure projected images are accurate both spatially and in size in conditions in which the cameras are more accurate than the human eye. Con- sequently, the present technology and the optical and computational feedback is able to function, and produce a corrected image of the world, even if the user (but not the cameras) is effectively blind to all but the virtual display.

[0108] When an object is close (arms length) the an- gleswill be different towhen theobject is at a far distance; thus it is important to correct for scale and angular posi- tion. A further benefit is that the optical display systems andmethods of some embodiments allow for compensa- tion for differences in focal depth, permitting a user to view an object that may be at arm’s length in addition to objects at a further distance.

[0109] FIG. 29 is a diagrammatic representation of an example machine in the form of a computer system 300 which is an example of one or more of the computers or processors referred to herein and, within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In various example embodiments, the ma- chine operates as a standalone device or may be con- nected (e.g., networked) to other machines. In a net- worked deployment, the machine may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to- peer (or distributed) network environment. The machine may be a module onboard an AR / VR / MR display or a remote computing device that interfaces with module interface electronics onboard the ARNR / MR display. The remote computing devices may be a personal com- puter (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a portable music player (e.g., a portable hard drive audio device such as a Moving Picture Experts Group Audio Layer 3 (MP3) player), a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that indivi- dually or jointly execute a set (or multiple sets) of instruc- tions to perform any one or more of the methodologies discussed herein.

[0110] The example computer system 300 includes a processor or multiple processors 302 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), and a main memory 306 and optional static memory 308, which communicate with each other via a bus 328. The computer system 300 may further include a video display 314 (e.g., a liquid crystal display (LCD), touch sensitive display). The computer system 300 may also include an alpha-numeric input device(s) 296 (e.g., a keyboard, keypad, touchpad, touch display, buttons; see also 316 in FIG. 29), a cursor control device (e.g., a mouse), a voice recognition or biometric verifica- tion unit (not shown), a drive unit 320 (also referred to as disk drive unit), a signal generation device 324 (e.g., a speaker), motor / servo device(s) 326, and a network in- terface device 312. The computer system 300 may further include a data encryption module (not shown) to encrypt data. The drive unit 320 includes a computer ormachine-readablemedium322 onwhich is stored one or more sets of instructions and data structures (e.g., instructions 304) embodying or utilizing any one or more of the methodologies or functions described herein. The instructions 304 may also reside, completely or at least partially, within the main memory 306 and / or within the processors302duringexecution thereof by thecomputer system 300. The main memory 306 and the processors 302 may also constitute machine-readable media.

[0111] The instructions 304 may further be transmitted or received over a network via the network interface device 312 utilizing any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP)). While the machine-readable medium 322 is shown inanexample embodiment to beasinglemedium, the term"computer-readablemedium" shouldbe taken to include a single medium or multiple media (e.g., a cen- tralized or distributed database and / or associated ca- ches and servers) that store the one or more sets of instructions. The term "computer-readable medium" shall also be taken to include anymedium that is capable of storing, encoding, or carrying a set of instructions for executionby themachineand that causes themachine to perform any one or more of the methodologies of the present application, or that is capable of storing, encod- ing, or carrying data structures utilized by or associated with such a set of instructions. The term "computer-read- able medium" shall accordingly be taken to include, but not be limited to, solid-state memories, optical and mag- 5 10 15 20 25 30 35 40 45 50 55 16 29 EP 4 657 140 A2 30 netic media, and carrier wave signals. Such media may also include, without limitation, hard disks, floppy disks, flash memory cards, digital video disks, random access memory (RAM), read only memory (ROM), and the like. The example embodiments described herein may be implemented in an operating environment comprising software installed on a computer, in hardware, or in a combination of software and hardware.

[0112] For purposes of explanation, numerous specific details are set forth in order to provide a thorough under- standing of the disclosure. It will be apparent, however, to one skilled in the art, that the disclosuremay be practiced without these specific details. In other instances, struc- tures and devices are shown at block diagram form only in order to avoid obscuring the disclosure.

[0113] Reference throughout this specification to "one embodiment" or "anembodiment"means thataparticular feature, structure, or characteristic described in connec- tion with the embodiment is included in at least one embodiment of the present technology. Thus, the ap- pearances of the phrases "in one embodiment" or "in an embodiment" or "according to one embodiment" (or other phrases having similar import) at various places throughout this specification are not necessarily all re- ferring to the same embodiment. Furthermore, the parti- cular features, structures, or characteristicsmay be com- bined in any suitable manner in one or more embodi- ments. Furthermore, depending on the context of discus- sion herein, a singular term may include its plural forms and a plural termmay include its singular form. Similarly, a hyphenated term (e.g., "on-demand") may be occa- sionally interchangeably used with its non-hyphenated version (e.g., "on demand"), a capitalized entry (e.g., "Software") may be interchangeably used with its non- capitalized version (e.g., "software"), a plural term may be indicated with or without an apostrophe (e.g., PE’s or PEs), and an italicized term (e.g., "N+1") may be inter- changeably used with its non-italicized version (e.g., "N+1"). Such occasional interchangeable uses shall not be considered inconsistent with each other.

[0114] Also, some embodiments may be described in terms of "means for" performing a task or set of tasks. It will be understood that a "means for" may be expressed herein in terms of a structure, such as a processor, a memory, an I / O device such as a camera, or combina- tions thereof. Alternatively, the "means for" may include an algorithm that is descriptive of a function or method step, while in yet other embodiments the "means for" is expressed in terms of a mathematical formula, prose, or as a flow chart or signal diagram.

[0115] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present technology. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addi- tion of one ormore other features, integers, steps, opera- tions, elements, components, and / or groups thereof.

[0116] It is noted at the outset that the terms "coupled," "connected", "connecting," "electrically connected," etc., are used interchangeably herein to generally refer to the condition of being electrically / electronically connected. Similarly, a first entity is considered to be in "communica- tion" with a second entity (or entities) when the first entity electrically sends and / or receives (whether throughwire- line or wireless means) information signals (whether containing data information or non-data / control informa- tion) to the secondentity regardless of the type (analogue or digital) of those signals. It is further noted that various figures (including component diagrams) shown and dis- cussed herein are for illustrative purpose only, and are not drawn to scale.

[0117] One skilled in the art will recognize that any one or more processors, cameras or other electronic devices disclosed herein may be configured to provide Internet access to one or more computing devices that are coupled to the Internet service or other network, and that the computing devices may include one or more proces- sors, buses,memory devices, display devices, input / out- put devices, and the like. Furthermore, thoseskilled in the art may appreciate that the Internet service or other net- work may be coupled to one or more databases, repo- sitories, servers, and the like, which may be utilized in order to implement any of the embodiments of the dis- closure as described herein.

[0118] The corresponding structures, materials, acts, and equivalents of all means or step plus function ele- ments in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present technology has been presented for purposes of illustration and descrip- tion, but is not intended to be exhaustive or limited to the present technology in the form disclosed. Many modifi- cations and variations will be apparent to those of ordin- ary skill in the art without departing from the scope and spirit of the present technology. Exemplary embodiments were chosen and described in order to best explain the principles of the present technology and its practical application, and to enable others of ordinary skill in the art to understand the present technology for various embodiments with various modifications as are suited to the particular use contemplated. Aspects of the pre- sent technology are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present technology. It will be understood that each block of the flowchart illus- trations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block dia- grams, can be implemented by computer program in- structions. These computer program instructionsmay be 5 10 15 20 25 30 35 40 45 50 55 17 31 EP 4 657 140 A2 32 provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing appa- ratus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks, and specified elsewhere in the description.

[0119] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing appa- ratus, or other devices to function in a particular manner, such that the instructionsstored in thecomputer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0120] In some embodiments, method steps. pro- cesses, functions / acts disclosed herein may be per- formed in a different order or combination. In some embodiments, one or more steps of methods processes, functions / acts disclosed herein may be omitted.

[0121] Thecomputer program instructionsmayalsobe loaded onto a computer, other programmable data pro- cessing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instruc- tions which execute on the computer or other program- mableapparatusprovideprocesses for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0122] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems,methodsand com- puter program products according to various embodi- ments of the present technology. In this regard, each block in the flowchart or block diagrams or each method steporprocesssummarized,describedand / or claimed in this application may represent a module, segment, or portion of code,which comprises oneormoreexecutable instructions for implementing the specified logical func- tion(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, twoblocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the spe- cified functions or acts, or combinations of special pur- pose hardware and computer instructions.

[0123] It is to be understood that the described embo- diments of the present technology are illustrative only and that modifications thereof may occur to those skilled in theart. Accordingly, this present technology is not to be regarded as limited to the embodiments disclosed, but is to be limited only as defined by the appended claims herein. It will further be understood that any features described in relation to any particular embodiment may be featured in combinations with other embodiments, for avoidance of doubt.

[0124] While embodiments of optical displays and methods have been described in detail, it should be apparent that modifications and variations thereto are possible, all of which fall within the true spirit and scope of the present technology. With respect to the above de- scription then, it is to be realized that the optimumdimen- sional relationships for the parts of the present technol- ogy, to include variations in size, materials, shape, form, function andmanner of operation, assembly and use, are deemed readily apparent andobvious tooneskilled in the art, and all equivalent relationships to those illustrated in the drawings and described in the specification are in- tended to be encompassed by the present technology.

[0125] Therefore, the foregoing is considered as illus- trative only of the principles of the present technology. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the present technology to the exact construction and operation shown and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope of the present technology. Embodiments of the present disclosure canbe described in view of the following clauses: 1. An optical display system comprising: an optical waveguide combiner; wherein said optical waveguide combiner com- prises: a waveguide optical substrate; an optical input coupler on, in or adjacent said waveguide optical substrate; and an optical output coupler on or in said a waveguide optical substrate, said optical input coupler being coupled via said wave- guide optical substrate to said optical output coupler; and a plurality of cameras; wherein said plurality of cameras comprises a first camera and a second camera; wherein said first camera is optically coupled to the output coupler and has a first field of view of at least one virtual object displayable at an image plane by said op- tical waveguide combiner; the at least one virtual object being in the field of view of an eyebox of the optical waveguide combiner; and wherein said second camera is optically 5 10 15 20 25 30 35 40 45 50 55 18 33 EP 4 657 140 A2 34 coupled to the output coupler and has a secondfieldof viewof theat least onevirtual object displayable by said optical wave- guide combiner; wherein, at the image plane, a combined field of view of the first camera and the second camera comprises the first field of view of the first camera and the second field of view of the second camera and includes anoverlapping regionof thefirst fieldof view and the second field of view; and wherein, at the imageplane, thedisplayable at least one virtual object is within the com- bined field of viewof the first cameraand the second camera. 2. The optical display system of clause 1, wherein said optical waveguide combiner is see through; and wherein at least one real object is viewablewithin the combined field of view of the first camera and the second camera. 3. The optical display system of clause 1, wherein, at the image plane, at least one of the field of viewof the eyebox or a field of view of a projector is within the combined field of view of the first and second cam- eras. 4. The optical display system of clause 2, wherein said plurality of cameras are located in a region outside the field of view of an eyebox of the optical waveguide combiner system. 5. The optical display system of clause 4, wherein said output coupler comprises a pre-exit pupil ex- pansion region and an exit pupil expansion region, and wherein said first camera and said second cam- era are optically coupled to the pre-exit pupil expan- sion region. 6. The optical display system of clause 5, further comprisingaprojector configured toproject a display image of the at least one virtual object into the input coupler. 7. The optical display system of clause 6, further comprising at least one processor operably coupled to said projector and said first camera and said second camera, wherein said at least one processor is configured to cause the projector to project a dis- play image of the at least one virtual object which includes fiducials,wherein the fiducials of the at least one virtual object arewithin the overlapping region of the first field of view of the first camera and the second field of view of the second camera; and wherein said at least one processor is configured to electronically stitch together first and second images captured from the first and second cameras, respectively, by adjusting the first and second images to align the fiducials in the first image with the fiducials in the second image. 8. The optical display system of clause 5, wherein said output coupler comprises any one or combina- tion of prism arrays, cascaded mirrors, diffractive gratings and holograms. 9. The optical display system of clause 5, wherein said output coupler comprises optical diffractive ele- ments. 10. The optical display system of clause 9, wherein said optical diffractive elements comprise two dif- fractive optical elements at least partially overlaid on one another in or on the waveguide substrate. 11. The optical display system of clause 10, wherein said output coupler comprises a combined 2D ex- pansion output diffractive grating. 12. The optical display system of clause 5, wherein eachcameraof saidpluralityof camerascomprisesa wafer level camera; and wherein said wafer level camera is incorporated in or on said optical wave- guide combiner. 13. The optical display system of clause 12, wherein said optical input coupler comprises an optical input diffractive grating, and wherein said optical input diffractive grating is in optical communication with said optical output grating without any intermediate optical diffractive grating therebetween. 14. The optical display system of clause 1, wherein said optical waveguide combiner is incorporated in an eyepiece of a pair of eyeglasses. 15.Amethodof camera calibration for anAR / MR / VR display, the method comprising: projecting into an optical waveguide combiner a dis- play image including fiducials, optically coupling first and second cameras to an output of the optical waveguide combiner, the first and second cameras having a combined field of view comprising a first field of view of the first camera and a second field of view of the second camera and including an over- lapping region of the first field of view and the second field of view; displaying, by the optical waveguide combiner, at an image plane, the display image as a virtual image with the fiducials, wherein the displayed virtual image is within the combined field of view of the first camera and the second camera, wherein the fiducials of the displayed virtual 5 10 15 20 25 30 35 40 45 50 55 19 35 EP 4 657 140 A2 36 image at the image plane are within the over- lapping region of the first field of view and the second field of view; capturing by the first camera a first image of the virtual image; capturingby thesecondcameraasecond image of the virtual image; and electronically stitching the captured first image and second image together; wherein electronically stitching the captured first image and second image together comprises electronically adjusting the captured first image and captured second image by moving the cap- tured first imageand the captured second image to align the fiducials of the first image with the fiducials of the second image. 16. The method of clause 15, wherein electronically stitching the captured first image and second image together further comprises determining the overlap- ping region of the first field of view and the second field of view by comparing the adjusted first and second images with the display image projected into the optical waveguide combiner; and electronically correcting the adjusted first and sec- ond imagesaccording to thedeterminedoverlapping region. 17. The method of clause 16, wherein optically cou- pling the first camera and the second camera to the output of the optical waveguide combiner comprises optically coupling the first camera and the second camera to a pre-exit pupil expansion output of an output coupler of the optical waveguide combiner. 18. Amethod of registering a VR / AR / MRdisplay, the method comprising: providing at least one virtual object for display; projecting into an optical waveguide combiner a display image of the at least one virtual object; displaying, by the optical waveguide combiner, the display image as the at least one virtual object; capturing an image of the displayed at least one virtual object and real-world scene using one or more optical camera(s), the camera(s) being optically coupled to an output of the optical waveguide combiner and having a field of view of the real world scene and of the displayed at least one virtual object; comparing, using the captured image, the at least one virtual object to at least one real object of the real world scene; determining a registration error between the at least one virtual object and the at least one real object based on the comparing; adjusting the virtual object(s) being displayed to substantially correct the registration error. 19. The method of clause 18, wherein comparing, using the captured image, the virtual object(s) with the real object(s) comprises identifying a common feature or fiducial in the virtual object(s) and the real object(s); and comparing said identified common feature or fiducial in the virtual object(s) with said identified common feature or fiducial in the real object(s); wherein determining a registration error between the virtual object(s) and the real object(s) based on said comparison comprises determining a difference in a position of said common feature or fiducial identified in the vir- tual object(s) and the position of said common feature or fiducial identified in the real object(s); and wherein adjusting the position of the virtual ob- ject(s) being displayed to substantially correct the registration error comprises adjusting any one or combination of size, shape and position of the virtual object(s) whereby the common feature or fiducial identified in the virtual ob- ject(s) and the common feature or fiducial iden- tified in the real object(s) substantially coincide. 20. The method of clause 19, wherein providing the virtual object(s) comprisesgeneratingavirtualobject from a real object using external sensors. 21. Amethod of registering a VR / AR / MRdisplay, the method comprising: generatingorprovidingat least onevirtual object of known size; based on an estimated distance of at least one real object from an optical waveguide combiner, scaling the at least one virtual object for display by the optical waveguide combiner; projecting into theopticalwaveguide combiner a display image of the scaled at least one virtual object; displaying, by the optical waveguide combiner, the display image as the at least one virtual object; capturing the displayed at least one virtual ob- ject and at least one real object using one or more optical cameras, each camera having a field of view of the at least one real object and of the displayed at least one virtual object; comparing, using the captured image, the at least one real object to the at least one virtual object; 5 10 15 20 25 30 35 40 45 50 55 20 37 EP 4 657 140 A2 38 determining a registration error between the at least one virtual object and the at least one real object based on the comparing; and adjusting the at least one virtual object being displayed to substantially correct the registra- tion error. 22. The method of clause 21, wherein comparing the captured virtual object(s) to the real object(s) comprises: determining from the captured imageanangular size and / or angle of the at least one real object from a point of view of the one or more optical camera; determining from the captured imageanangular sizeand / or angle of theat least onevirtual object from the point of view of the one or more optical camera; and wherein determining a registration error be- tween the at least one virtual object and the at least one real object based on said comparison comprises determining the difference in at least one of angular size or angle of the at least one virtual object relative to the at least one real object; and wherein adjusting the virtual object being displayed to substantially correct the registration error com- prises adjusting any one or combination of the size, shape and spatial location of the at least one virtual object, wherein the at least one virtual object ap- pears substantially in a same spatial location as the at least one real objectwhen viewed fromaneyebox. 23. A method of registration of a VR / MR / AR display system, the method comprising: capturing stereo images using first and second op- tical cameras of a binocular system, the first and second cameras being optically coupled to respec- tive outputs of first and second optical waveguide combiners of the binocular systemand having a field of view of at least one displayable virtual object and of at least one real object; determining anestimated distanceof the at least one real object from the optical waveguide combiners using at least one of the one ormore optical cameras or an estimated spatial location of the at least one real object using the one or more optical cameras; generating the virtual object(s) for display by the optical waveguide combiners based on the es- timated distance and / or estimated spatial loca- tion; projecting into the optical waveguide combiners a display image of the virtual object(s); displaying, by the optical waveguide combiners, the display image as the virtual object(s); capturing an image of the displayed virtual ob- ject(s) and real object(s) using the optical cam- eras; comparing the displayed virtual object(s) to the real object(s) using the captured image; determining a registration error between the real object(s) and the virtual object(s) based on the comparing; adjusting the virtual object(s) being displayed to substantially correct the registration error. 24. The method of clause 23, wherein determining an estimated distance of the real object(s) from the optical waveguide combiners using the optical cam- eras comprises: extracting from the captured images of the real object(s) a virtual image representing the real object(s); recognizing the extracted virtual object(s); determining depth perception cues from the images captured by the optical cameras; estimating the distance of the real object(s) from the optical waveguide combiners based on the determined depth perception cues and object recognition. 25. The method of clause 23, wherein determining a registration error between the at least one real object and the at least one virtual object based on the comparing comprises: determining a difference in spatial location between the displayed at least one virtual object and the at least one real object from the binocular disparity in the stereo images. 26. A method of determining distance in an AR / MR / VR display, the method comprising: capturing in stereo images using one or more optical camera for a binocular opticalwaveguide combiner system, the one or more optical cam- era having a field of view of at least one display- able virtual object and of at least one real object; extracting, from the captured images of the at least one real object, at least one virtual object representing the at least one real object; recognizing the extracted at least one virtual object; determining depth perception cues from the images captured by the one or more optical cameras; determining the distance of the at least one real object from the optical waveguide combiners based on the determined depth perception cues and object recognition. 27.A method of determining distance in an 5 10 15 20 25 30 35 40 45 50 55 21 39 EP 4 657 140 A2 40 AR / MR / VR display, the method comprising: generatingorprovidingat least onevirtual object for display; projecting into optical waveguide combiners of a binocular optical combiner system a display im- age of the at least one virtual object; displaying in stereo, by the optical waveguide combiners, the display image as the at least one virtual object; capturing stereo imagesof the displayedat least one virtual object and at least one real object using one or more optical camera incorporated into each optical waveguide combiner of the binocular system, each of the one or more op- tical camera having a field of view of the at least one real object and of the displayed at least one virtual object; comparing the displayed at least one virtual object to the at least one real object using the captured stereo images; determining the binocular disparity in the stereo images; and determining the distance of the at least one real object from theat least one virtual object accord- ing to the binocular disparity. Claims 1. An optical display system comprising: an optical waveguide combiner (2), comprising: a waveguide optical substrate (10); and an optical output coupler (8) on or in the waveguide optical substrate (10); and a camera (6); wherein the camera (6) has a field of view of at least one virtual object (13) displayable at an image plane by the optical waveguide combiner (2), the displayable at least one virtual object (13) being in the field of viewof aneyebox (16) of the optical waveguide combiner (2); and wherein, at the image plane, the displayable at least one virtual object (13) is within the field of view of the camera (6). 2. The optical display system of claim 1, wherein: the camera (6) is optically coupled to the optical output coupler (8). 3. The optical display system of claim 1 or 2, further comprising an optical input coupler (3) on, in, or adjacent the waveguide optical substrate (10); wherein the optical input coupler (3) is coupled via the waveguide optical substrate (10) to the optical output coupler (8). 4. The optical display system of claim 3, wherein: the optical input coupler (3) comprises anoptical input diffractive grating; and the optical input diffractive grating is in optical communication with the optical output coupler (8) without any intermediate optical diffractive grating therebetween. 5. The optical display system of claim 3 or 4, further comprisingaprojector configured toproject a display image of the at least one virtual object (13) into the input coupler (3). 6. The optical display system of any preceding claim, wherein: the optical waveguide combiner (2) is transpar- ent; and at least one real object is viewablewithin thefield of view of the camera (6). 7. The optical display system of any preceding claim, wherein, at the image plane, the field of view of the eyebox (16) is within the field of view of the camera (6). 8. The optical display system of any preceding claim, wherein the camera (6) is located in a region outside the field of view of the eyebox (16). 9. The optical display system of claim 8, wherein: the output coupler comprises a pre-exit pupil expansion region and an exit pupil expansion region; and the camera (6) is optically coupled to the pre-exit pupil expansion region. 10. The optical display system of any preceding claim, wherein the output coupler comprises one or more optical diffractive elements; preferably, wherein the one or more optical dif- fractive elements comprise two diffractive opti- cal elements at least partially overlaid on one another in or on the waveguide substrate. more preferably, wherein the output coupler comprises a combined 2D expansion output diffractive grating. 11. The optical display system of any preceding claim, wherein: the camera (6) comprises a wafer level camera; and 5 10 15 20 25 30 35 40 45 50 55 22 41 EP 4 657 140 A2 42 the wafer level camera is incorporated in or on the optical waveguide combiner (2). 12. The optical display system of any preceding claim, wherein: the camera is a first camera (6); the optical display system further comprises a second camera (7); the field of view of the first camera (6) is a first field of view; the second camera (7) is optically coupled to the output coupler and has a second field of view of the at least one virtual object (13) displayable by the optical waveguide combiner (2); and at the image plane, the displayable at least one virtual object (13) is within the combined field of view of the first camera (6) and the second camera (7). 13. The optical display system of claim 12, wherein: at the image plane, a combined field of view of the first camera (6) and the second camera (7) com- prises the first field of view of the first camera (6) and the second field of viewof the second camera (7) and includesanoverlapping regionof thefirst fieldof view and the second field of view. 14. Amethod of registering a VR, AR, or MR display, the method comprising: providing at least one virtual object (13) for dis- play; projecting into an optical waveguide combiner (2) a display image of the at least one virtual object (13); displaying, by the optical waveguide combiner (2), the display image as the at least one virtual object (13); capturing an image of the displayed at least one virtual object (13) and a real world scene using a camera (6), the camera (6) being optically coupled to an output of the optical waveguide combiner (2) andhavinga field of viewof the real world scene and of the displayed at least one virtual object (13); comparing, using the captured image, the at least one virtual object (13) to at least one real object of the real world scene; determining a registration error between the at least one virtual object (13) and the at least one real object based on the comparison; and adjusting the at least one virtual object (13) being displayed to substantially correct the re- gistration error. 15. A non-transitory computer-readable storage med- ium, the computer-readable storagemedium includ- ing instructions thatwhenexecutedbyaprocessor of a system, cause the system to register a VR, AR, or MR display by: providing at least one virtual object (13) for dis- play; projecting into an optical waveguide combiner (2) a display image of the at least one virtual object (13); displaying, by the optical waveguide combiner (2), the display image as the at least one virtual object (13); capturing an image of the displayed at least one virtual object (13) and a real world scene using a camera (6), the camera (6) being optically coupled to an output of the optical waveguide combiner (2) andhavinga field of viewof the real world scene and of the displayed at least one virtual object (13); comparing, using the captured image, the at least one virtual object (13) to at least one real object of the real world scene; determining a registration error between the at least one virtual object (13) and the at least one real object based on the comparison; and adjusting the at least one virtual object (13) being displayed to substantially correct the re- gistration error. 5 10 15 20 25 30 35 40 45 50 55 23 EP 4 657 140 A2 24 EP 4 657 140 A2 25 EP 4 657 140 A2 26 EP 4 657 140 A2 27 EP 4 657 140 A2 28 EP 4 657 140 A2 29 EP 4 657 140 A2 30 EP 4 657 140 A2 31 EP 4 657 140 A2 32 EP 4 657 140 A2 33 EP 4 657 140 A2 34 EP 4 657 140 A2 35 EP 4 657 140 A2 36 EP 4 657 140 A2 37 EP 4 657 140 A2 38 EP 4 657 140 A2 39 EP 4 657 140 A2 40 EP 4 657 140 A2 41 EP 4 657 140 A2 42 EP 4 657 140 A2 43 EP 4 657 140 A2 44 EP 4 657 140 A2 45 EP 4 657 140 A2 46 EP 4 657 140 A2 REFERENCES CITED IN THE DESCRIPTION This list of references cited by the applicant is for the reader’s convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard. Patent documents cited in the description • US 54487021

[0001] • US 10114220 B2

[0055] (19) *EP004657140A3* (11) EP 4 657 140 A3 (12) EUROPEAN PATENT APPLICATION (88) Date of publication A3: 04.03.2026 Bulletin 2026 / 10 (43) Date of publication A2: 03.12.2025 Bulletin 2025 / 49 (21) Application number: 25208906.5 (22) Date of filing: 30.11.2022 (51) International Patent Classification (IPC): G02B 27 / 01 (2006.01) (52) Cooperative Patent Classification (CPC): G02B 27 / 0081; G02B 27 / 0172; G02B 2027 / 0138; G02B 2027 / 0178 (84) Designated Contracting States: AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR (30) Priority: 07.12.2021 US 202117544870 (62) Document number(s) of the earlier application(s) in accordance with Art. 76 EPC: 22840533.8 / 4 445 207 (71) Applicant: Snap Inc. Santa Monica, CA 90405 (US) (72) Inventor: LEIGHTON, James Raymond Santa Monica, 90405 (US) (74) Representative: Mathys & Squire The Shard 32 London Bridge Street London SE1 9SG (GB) (54) OPTICAL WAVEGUIDE COMBINER SYSTEMS AND METHODS (57) An optical display system has an optical wave- guide combiner and one or more cameras, cameras is optically coupled to the optical waveguide combiner and have a field of view of at least one real object and at least one virtual object displayable by the optical display sys- tem. The cameras, whichmay be for example wafer level waveguide cameras, may be disposed outside the usable field of view of an output coupler. The cameras may be self-calibrated electronically using images cap- tured by the cameras of one or more virtual objects displayable by the optical display system. AR / VR / MR registration of devices and / or displayed virtual objects with real objects may be implemented using the images captured by the cameras of the displayed virtual objects and real world objects. Real object distance and / or spa- tial location relative to the optical waveguide combiners may be determined or estimated from the captured images. EP 4 65 7 14 0 A 3 Processed by Luminess, 75001 PARIS (FR) 2 EP 4 657 140 A3 5 10 15 20 25 30 35 40 45 50 55 3 EP 4 657 140 A3 5 10 15 20 25 30 35 40 45 50 55 摘要 CN Title 光波导合束器系统和方法 CN Abstract 一种光学显示系统，具有光波导合束器和一个或多个摄像头，所述摄像头与所述光波导 合束器光学耦合并且其视场覆盖至少一个真实物体和所述光学显示系统可显示的至少 一个虚拟物体。所述摄像头（例如可以是晶圆级波导摄像头）可以设置在输出耦合器的 可用视场之外。所述摄像头可利用由所述摄像头拍摄的所述光学显示系统可显示的一 个或多个虚拟物体的图像进行电子自校准。设备和 / 或所显示的虚拟物体与真实物体之 间的AR / VR / MR配准可利用由所述摄像头拍摄的所显示的虚拟物体和真实物体的图像 实现。真实物体相对于光波导合束器的距离和 / 或空间位置可基于所述拍摄的图像来确 定或估计。

Claims

1. An optical display system comprising: an optical waveguide combiner (2), comprising: a waveguide optical substrate (10); and an optical output coupler (8) on or in the waveguide optical substrate (10); and a camera (6); wherein the camera (6) has a field of view of at least one virtual object (13) displayable at an image plane by the optical waveguide combiner (2), the displayable at least one virtual object (13) being in the field of view of an eyebox (16) of the optical waveguide combiner (2); and wherein, at the image plane, the displayable at least one virtual object (13) is within the field of view of the camera (6).

2. The optical display system of claim 1, wherein: the camera (6) is optically coupled to the optical output coupler (8).

3. The optical display system of claim 1 or 2, further comprising an optical input coupler (3) on, in, or adjacent the waveguide optical substrate (10); wherein the optical input coupler (3) is coupled via the waveguide optical substrate (10) to the optical output coupler (8).

4. The optical display system of claim 3, wherein: the optical input coupler (3) comprises an optical input diffractive grating; and the optical input diffractive grating is in optical communication with the optical output coupler (8) without any intermediate optical diffractive grating therebetween.

5. The optical display system of claim 3 or 4, further comprising a projector configured to project a display image of the at least one virtual object (13) into the input coupler (3).

6. The optical display system of any preceding claim, wherein: the optical waveguide combiner (2) is transparent; and at least one real object is viewable within the field of view of the camera (6).

7. The optical display system of any preceding claim, wherein, at the image plane, the field of view of the eyebox (16) is within the field of view of the camera (6).

8. The optical display system of any preceding claim, wherein the camera (6) is located in a region outside the field of view of the eyebox (16).

9. The optical display system of claim 8, wherein: the output coupler comprises a pre-exit pupil expansion region and an exit pupil expansion region; and the camera (6) is optically coupled to the pre-exit pupil expansion region.

10. The optical display system of any preceding claim, wherein the output coupler comprises one or more optical diffractive elements; preferably, wherein the one or more optical diffractive elements comprise two diffractive optical elements at least partially overlaid on one another in or on the waveguide substrate. more preferably, wherein the output coupler comprises a combined 2D expansion output diffractive grating.

11. The optical display system of any preceding claim, wherein: the camera (6) comprises a wafer level camera; and the wafer level camera is incorporated in or on the optical waveguide combiner (2).

12. The optical display system of any preceding claim, wherein: the camera is a first camera (6); the optical display system further comprises a second camera (7); the field of view of the first camera (6) is a first field of view; the second camera (7) is optically coupled to the output coupler and has a second field of view of the at least one virtual object (13) displayable by the optical waveguide combiner (2); and at the image plane, the displayable at least one virtual object (13) is within the combined field of view of the first camera (6) and the second camera (7).

13. The optical display system of claim 12, wherein: at the image plane, a combined field of view of the first camera (6) and the second camera (7) comprises the first field of view of the first camera (6) and the second field of view of the second camera (7) and includes an overlapping region of the first field of view and the second field of view.

14. A method of registering a VR, AR, or MR display, the method comprising: providing at least one virtual object (13) for display; projecting into an optical waveguide combiner (2) a display image of the at least one virtual object (13); displaying, by the optical waveguide combiner (2), the display image as the at least one virtual object (13); capturing an image of the displayed at least one virtual object (13) and a real world scene using a camera (6), the camera (6) being optically coupled to an output of the optical waveguide combiner (2) and having a field of view of the real world scene and of the displayed at least one virtual object (13); comparing, using the captured image, the at least one virtual object (13) to at least one real object of the real world scene; determining a registration error between the at least one virtual object (13) and the at least one real object based on the comparison; and adjusting the at least one virtual object (13) being displayed to substantially correct the registration error.

15. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor of a system, cause the system to register a VR, AR, or MR display by: providing at least one virtual object (13) for display; projecting into an optical waveguide combiner (2) a display image of the at least one virtual object (13); displaying, by the optical waveguide combiner (2), the display image as the at least one virtual object (13); capturing an image of the displayed at least one virtual object (13) and a real world scene using a camera (6), the camera (6) being optically coupled to an output of the optical waveguide combiner (2) and having a field of view of the real world scene and of the displayed at least one virtual object (13); comparing, using the captured image, the at least one virtual object (13) to at least one real object of the real world scene; determining a registration error between the at least one virtual object (13) and the at least one real object based on the comparison; and adjusting the at least one virtual object (13) being displayed to substantially correct the registration error.

Images