Optical waveguide combiner system and method

Abstract

An optical display system has a light waveguide combiner and one or more cameras optically coupled to the light waveguide combiner and having a field of view of at least one virtual object and at least one real object that can be displayed by the optical display system. The cameras, which can be, for example, wafer-level waveguide cameras, can be disposed outside of the available field of view of the output coupler. The cameras can be electronically self-calibrated using images of one or more virtual objects captured by the cameras that can be displayed by the optical display system. AR / VR / MR registration of the device and / or the displayed virtual objects with real objects can be achieved using images of the displayed virtual objects and real world objects captured by the cameras. Distances and / or spatial locations of real objects relative to the light waveguide combiner can be determined or estimated from the captured images.

Description

[0001] Priority Statement

[0002] This application claims priority to U.S. non-provisional patent application No. 17 / 544,870, filed on December 7, 2021, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to optical display systems, and more particularly, but not exclusively, to optical waveguide combiner systems. Additionally or alternatively, embodiments relate to methods for calibrating imaging devices in optical display systems. Additionally or alternatively, embodiments relate to apparatus and / or methods for registering one or more virtual objects in virtual reality and / or augmented reality display systems. Background Technology

[0004] In recent years, the use of virtual reality and augmented reality display systems has increased significantly, with devices designed for both personal and commercial / industrial purposes. These devices can be used in monocular or binocular formats and for a wide range of applications, from monochrome displays of simple icons to text-based instruction presentations to full-video displays. When these devices are used to supplement real-world scenes with computer-generated imagery, the display system needs to accurately display virtual or augmented reality content within its context. Summary of the Invention

[0005] According to some aspects of the present technology, an optical display system is provided. The optical display system may include an optical waveguide combiner and a plurality of camera devices. The optical waveguide combiner may include: a waveguide optical substrate; an optical input coupler located on, in, or adjacent to the waveguide optical substrate; and an optical output coupler located on or in the waveguide optical substrate. The optical input coupler may be coupled to the optical output coupler via the waveguide optical substrate. The plurality of camera devices may include a first camera device and a second camera device. The first camera device may be optically coupled to the output coupler and has a first field of view capable of displaying one or more virtual objects on an image plane via the optical waveguide combiner. The displayable virtual objects are located within the field of view of the eye-tracking box of the optical waveguide combiner. The second camera device may be optically coupled to the output coupler and has a second field of view capable of displaying virtual objects via the optical waveguide combiner. At the image plane, the combined field of view of the first camera device and the second camera device may include a first field of view of the first camera device and a second field of view of the second camera device, and may include the overlapping area of ​​the first field of view and the second field of view. At the image plane, at least one virtual object that can be displayed may be located within the combined field of view of the first camera device and the second camera device.

[0006] In some implementations, the optical waveguide combiner is see-through. One or more real objects may be visible within the combined field of view of the first and second camera devices.

[0007] In some implementations, at the image plane, the field of view of the eye-tracking box and / or the field of view of the projector are within the combined field of view of the first and second camera devices.

[0008] In some implementations, multiple camera devices are located in an area outside the field of view of the eye-tracking box of the optical waveguide combiner system.

[0009] In some embodiments, the output coupler includes a front exit pupil expansion region and an exit pupil expansion region. A first camera device and a second camera device are optically coupled to the front exit pupil expansion region.

[0010] In some implementations, the system also includes a projector configured to project a display image of a virtual object onto an input coupler.

[0011] In some embodiments, the system further includes one or more processors operatively coupled to a projector and a first and a second camera device. One or more processors are configured to cause the projector to project a display image of a virtual object, the display image including reference points, wherein the reference points of the displayable virtual object are located within the overlapping area of ​​a first field of view of the first camera device and a second field of view of the second camera device. One or more processors are configured to electronically stitch together the first and second images by adjusting the reference points in the first image captured from the first and second camera devices, respectively, to align the reference points in the first image with the reference points in the second image.

[0012] In some embodiments, the output coupler includes any one or a combination of a prism array, a cascaded mirror, a diffraction grating, and a holographic device. The output coupler may include an optical diffraction element.

[0013] In some embodiments, the optical diffraction element may include two diffraction optical elements that are at least partially stacked on top of each other in or on the waveguide substrate.

[0014] In some implementations, the output coupler includes a combined 2D extended output diffraction grating.

[0015] In some embodiments, each of the plurality of camera devices may include a wafer-level camera device. The wafer-level camera device is integrated into or on an optical waveguide assembly.

[0016] In some embodiments, the optical input coupler includes an optical input diffraction grating, and wherein the optical input diffraction grating and the optical output grating communicate optically without any intermediate optical diffraction grating between them.

[0017] In some implementations, the optical waveguide combiner is integrated into the eyepiece of a pair of glasses.

[0018] According to some aspects of this technology, a method for calibrating a camera device for an AR / MR / VR display is provided. The method may include: projecting a display image including a reference point into an optical waveguide combiner; optically coupling a first camera device and a second camera device to the output of the optical waveguide combiner, the first camera device and the second camera device having a combined field of view, the combined field of view including a first field of view of the first camera device and a second field of view of the second camera device, and including an overlapping area of ​​the first field of view and the second field of view; displaying the display image as a virtual image with a reference point at an image plane by the optical waveguide combiner, wherein the displayed virtual image is located within the combined field of view of the first camera device and the second camera device, wherein the reference point of the virtual image displayed at the image plane is located within the overlapping area of ​​the first field of view and the second field of view; capturing a first image of the virtual image by the first camera device; capturing a second image of the virtual image by the second camera device; and electronically stitching the captured first image and the second image together; wherein electronically stitching the captured first image and the second image together includes: electronically adjusting the captured first image and the captured second image by moving the captured first image and the captured second image to align the reference point of the first image with the reference point of the second image.

[0019] In some embodiments, electronically stitching together the captured first and second images further includes: determining the overlap area of ​​the first and second fields of view by comparing the adjusted first and second images with a display image projected into the combiner; and electronically correcting the adjusted first and second images based on the determined overlap area.

[0020] In some embodiments, optically coupling the first camera device and the second camera device to the output of the optical waveguide combiner includes: an extended output of the front exit pupil of the output coupler of the first camera device and the second camera device optically coupled to the optical waveguide combiner.

[0021] According to various aspects of the present technology, a method for registering a VR / AR / MR display is provided. The method may include: providing one or more virtual objects for display; projecting a display image of the virtual objects onto an optical waveguide combiner; displaying the display image as a virtual object by the optical waveguide combiner; capturing images of the displayed virtual objects and a real-world scene using one or more optical cameras, the cameras being optically coupled to the output of the optical waveguide combiner and having a field of view of both the real-world scene and the displayed virtual objects; comparing the captured images with at least one real object in the real-world scene; determining a registration error between the virtual objects and at least one real object based on the comparison; and adjusting the displayed virtual objects to substantially correct the registration error.

[0022] In some implementations, comparing virtual and real objects using the captured images includes: identifying common features or reference points between the virtual and real objects; and comparing the identified common features or reference points in the virtual object with those in the real object; wherein determining the registration error between the virtual and real objects based on the comparison includes: determining the difference between the position of the common features or reference points identified in the virtual object and the position of the common features or reference points identified in the real object; and wherein adjusting the position of the displayed virtual object to substantially correct the registration error includes: adjusting any one or a combination of the size, shape, and position of the virtual object such that the common features or reference points identified in the virtual object substantially coincide with those identified in the real object.

[0023] In some implementations, providing virtual objects includes generating virtual objects based on real objects using external sensors.

[0024] According to some aspects of this technology, a method for registering a VR / AR / MR display is provided. The method may include: generating or providing one or more virtual objects of known size; scaling the virtual objects for display by the optical waveguide combiner based on estimated distances between the one or more real objects and the optical waveguide combiner; projecting a display image of the scaled virtual objects onto the optical waveguide combiner; displaying the display image as virtual objects by the optical waveguide combiner; capturing the displayed virtual objects and real objects using one or more optical imaging devices, each imaging device having a field of view for both the real objects and the displayed virtual objects; comparing the real objects with the virtual objects using the captured images; determining a registration error between the virtual objects and the real objects based on the comparison; and adjusting the displayed virtual objects to substantially correct the registration error.

[0025] In some embodiments, comparing the captured virtual objects with real objects includes: determining from the captured image the angular size and / or angle of one or more real objects viewed from the viewpoint of the camera device; determining from the captured image the angular size and / or angle of one or more virtual objects viewed from the viewpoint of the camera device; and wherein determining the registration error between the virtual objects and real objects based on the comparison includes: determining the difference in angular size and / or angle of one or more virtual objects relative to one or more real objects; and wherein adjusting the displayed virtual objects to substantially correct the registration error includes: adjusting any one or a combination of the size, shape, and spatial position of one or more virtual objects so that, when viewed from the eye-tracking box, one or more virtual objects appear in substantially the same spatial position as real objects.

[0026] According to this technology, a method for registering a VR / MR / AR display system is provided. The method may include: capturing stereoscopic images using a first optical camera device and a second optical camera device of a binocular system, the first and second camera devices being optically coupled to corresponding outputs of a first and second optical waveguide combiner of the binocular system, and having a field of view for displaying one or more virtual objects and one or more real objects; determining an estimated distance between the real object and the optical combiner using the optical camera devices, and / or determining an estimated spatial position of the real object using the optical camera devices; generating virtual objects for display by the optical waveguide combiner based on the estimated distance and / or the estimated spatial position; projecting a display image of the virtual object into the optical waveguide combiner; displaying the display image as a virtual object by the optical waveguide combiner; capturing images of the displayed virtual object and the real object using the optical camera devices; comparing the displayed virtual object with the real object using the captured images; determining a registration error between the real object and the virtual object based on the comparison; and adjusting the displayed virtual object to substantially correct the registration error.

[0027] In some implementations, determining the estimated distance of a real object from an optical combiner using an optical camera device includes: extracting a virtual image representing the real object from an image of the captured real object; identifying the extracted virtual object; determining depth-sensing cues from the image captured by the optical camera device; and estimating the distance of the real object from the optical waveguide combiner based on the determined depth-sensing cues and object identification.

[0028] In some implementations, determining the registration error between the real object and the virtual object based on comparison includes: determining the spatial position difference between the displayed virtual object and the real object based on the binocular parallax in the stereoscopic image.

[0029] According to some aspects of this technology, a method for determining distance in an AR / MR / VR display is provided. The method may include: capturing a stereoscopic image using an optical camera device of a binocular waveguide combiner system, the optical camera device having a field of view for displaying one or more virtual objects and one or more real objects; extracting one or more virtual objects representing the real objects from the captured image of the real objects; identifying the extracted virtual objects; determining depth-sensing cues from the image captured by the optical camera device; and determining the distance between the real objects and the optical combiner based on the determined depth-sensing cues and object recognition.

[0030] According to some aspects of the present technology, a method for determining distance in an AR / MR / VR display is provided. The method may include: generating or providing one or more virtual objects for display; projecting a display image of the virtual objects onto an optical waveguide combiner of a binocular optical combiner system; stereoscopically displaying the display image as virtual objects by the optical combiner; capturing stereoscopic images of the displayed virtual objects and one or more real objects using one or more optical imaging devices incorporated in each optical waveguide combiner of the binocular system, the one or more imaging devices having a field of view for both the real objects and the displayed virtual objects; comparing the displayed virtual objects with real objects using the captured stereoscopic images; determining binocular parallax in the stereoscopic images; and determining the distance between the real objects and the virtual objects based on the binocular parallax.

[0031] In some aspects of this technology, a non-transitory computer-readable medium is provided comprising program instructions that, when executed by one or more processors of a computer system, cause one or more processors and / or an optical display system to perform any one or any combination of the operation and / or method steps described herein.

[0032] In some embodiments, this technology is directed to one or more computing systems that can be configured to perform specific operations or actions by installing software, firmware, hardware, or combinations thereof on the system, which in operation cause the system to perform the operations and / or method steps described herein. Example systems may include one or more processors; and memory, the processors executing instructions stored in the memory to cause one or more processors and / or an optical display system to perform the operations and / or method steps described herein. Attached Figure Description

[0033] Figure 1 A perspective view of an optical waveguide combiner system according to some aspects of the present technology is shown;

[0034] Figure 2 It shows Figure 1A perspective view of an optical waveguide combiner system, showing the overlapping field of view of different optical elements of the system according to some embodiments;

[0035] Figure 3 A perspective view of an optical waveguide combiner system according to some embodiments is shown;

[0036] Figure 4 A rear plan view of an optical waveguide combiner system formed to accommodate an eyepiece, according to some embodiments, is shown.

[0037] Figure 5 The following are illustrated according to some embodiments. Figure 4 Side view of an optical waveguide combiner system;

[0038] Figure 6 A perspective view of a pair of glasses including an eyepiece according to some embodiments is shown;

[0039] Figure 7A and 7B This is a schematic diagram illustrating the theodolite calibration technique of the camera device self-calibration method and system for explaining the implementation method;

[0040] Figure 8 This is a schematic diagram illustrating the inherent reference of the waveguide, used to illustrate a method and system for self-calibration of a camera device for an AR / VR display according to some embodiments;

[0041] Figure 9 This is a flowchart illustrating some aspects of the method according to the present technology;

[0042] Figure 10 This is a schematic diagram showing the field of view of the optical elements of an optical display system according to some embodiments;

[0043] Figure 11 It is a plan view of the field of view of a projector including an alignment reference point, according to some embodiments;

[0044] Figure 12 It is a planar view of a virtual image captured by a waveguide camera device with an misaligned field of view at the image plane of an optical display system;

[0045] Figure 13 Yes Figure 12 The image is electronically adjusted to align the reference point in the plan view;

[0046] Figure 14 Yes Figure 13 The image is electronically corrected to present a planar view of the virtual image displayed by the optical waveguide combiner system without distortion;

[0047] Figure 15 A block diagram of an optical display system according to some embodiments is shown;

[0048] Figure 16 A flowchart of a method according to some aspects of this technology is shown;

[0049] Figure 17 A schematic diagram of an optical display system according to some embodiments is shown, illustrating that the displayed virtual object and the real object are misaligned;

[0050] Figure 18 A block diagram of an optical display system according to some embodiments is shown;

[0051] Figure 19 A block diagram of an optical display system according to some embodiments is shown;

[0052] Figure 20 A flowchart of a method according to some aspects of this technology is shown;

[0053] Figure 21 A schematic diagram of an optical display system according to some embodiments is shown, illustrating the misalignment between the displayed virtual object and the real object;

[0054] Figure 22 A flowchart of a method according to some aspects of this technology is shown;

[0055] Figure 23 A binocular optical display system according to some embodiments is shown;

[0056] Figure 24 A flowchart of a method according to some aspects of this technology is shown;

[0057] Figure 25 A schematic diagram of viewing a real object using a binocular optical display system according to some embodiments is shown;

[0058] Figure 26 A flowchart of a method according to some aspects of this technology is shown;

[0059] Figure 27 The illustration shows hyperstereoscopic vision in a binocular waveguide combiner system according to some embodiments, which is a result of misalignment between the displayed virtual object and the real object;

[0060] Figure 28 The illustration shows hyperstereoscopic vision in a binocular waveguide combiner system according to some embodiments, resulting from misalignment between the displayed virtual and real objects; and

[0061] Figure 29An exemplary computing device is shown that can be used to implement some embodiments of this technology. Detailed Implementation

[0062] In the following description, specific details, such as particular embodiments, processes, techniques, etc., are set forth for purposes of explanation and not limitation, in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that this disclosure may be practiced in other embodiments that depart from these specific details.

[0063] The applicant identifies one or more significant technical problems in optical systems used for virtual reality (VR), mixed reality (MR), and / or augmented reality (AR) displays, such as, but not limited to, head-mounted displays, including near-eye displays. In AR / MR / VR display systems, camera devices and sensors are configured to create a 3D map of the real world, which is then used to calculate images to describe the external world in angular space. However, uncertainties in display angle and position lead to projection axis errors and parallax errors. For these reasons, AR and VR displays require accurate registration of sensors and / or camera devices with the projector / display system. In AR displays, virtual objects need to be registered with real objects in the real world. Therefore, accurate registration techniques are required in AR and VR displays.

[0064] In some applications, accurate mechanical registration may be undesirable or impossible. In applications where the camera / sensor can move independently of the display, physical registration requirements need to be relaxed. One such example is a bomb disposal suit, where the display is attached to the inner helmet and the sensor to an outer visor that can move independently of the inner helmet. Furthermore, in some applications, mechanical tolerances and manufacturing complexities further complicate physical registration. For example, retractable displays or sensors associated with head-mounted displays are designed to be movable to reduce weight on the head, which inherently makes registration difficult.

[0065] Inaccurate registration can lead to poor device performance or even malfunction. As a non-limiting example, a processor connected to an AR display can be configured to display virtual navigation icons overlaid on a real-world view observed through a waveguide, providing the AR display wearer with visual guidance on how to reach their desired destination. The presentation of such navigation icons can have some degree of error in adapting to the scale or size within the AR display, as they are not related to real-world objects but rather indicate whether to turn left, right, or straight when approaching an intersection. However, depending on the environment in which the wearer seeks navigation, GPS may be subject to general positioning accuracy issues, in which case there may be some differences between what the wearer sees through the display and the icons overlaid via the AR display. In other cases, an AR system may be designed to overlay virtual images representing one or more aspects of a real-world object onto the corresponding real-world object viewed through the device. For example, image data from a night vision sensor in poorly lit or dark terrain, seen through the AR display, can be overlaid on the AR display along with terrain information from a depth imaging sensor to allow the wearer to navigate that terrain. Unlike previous examples of GPS icon-driven navigation applications, using data from night vision or depth sensors requires more accurate calibration and registration of AR displays; otherwise, it could lead to the wearer colliding with real-world objects that are incorrectly presented within the AR image. It has been recognized that accurate calibration and registration of AR displays is necessary.

[0066] Various aspects of this technology can alleviate the aforementioned problems. The optical systems and methods of this technology allow for improved or alternative methods for device calibration and virtual model registration in virtual reality, mixed reality, and / or augmented reality display systems. The application of optical systems and methods according to various aspects of this technology in some exemplary augmented reality and virtual reality display systems will now be described in more detail. However, it is understood that in some other aspects and implementations, this technology can be configured for augmented reality, mixed reality, and / or virtual reality systems other than those shown in the examples, and even for other types of optical transmission systems.

[0067] The technical features described in this application can be used to construct various aspects of optical waveguide combiner systems and methods. In some methods, the optical display system has an optical waveguide combiner and one or more imaging devices. One or more imaging devices can be optically coupled to the output of the optical waveguide combiner and have a field of view for one or more virtual objects and / or one or more real objects that can be displayed by the optical display system. Multiple wafer-level waveguide imaging devices can be positioned outside the available field of view of the output coupler. Multiple waveguide imaging devices can perform electronic self-calibration using images of virtual objects captured by the waveguide imaging devices that can be displayed by the optical display system.

[0068] Furthermore, the technical features described in this application can be used to construct various aspects of AR, MR, and / or VR display registration methods and systems. In some methods, AR / VR / MR registration of the device and / or the displayed virtual object with the real object can be achieved using images of a real-world object and a displayed virtual object captured by one or more camera devices optically coupled to a waveguide combiner. The distance and / or spatial position of the real object relative to the optical waveguide combiner can be determined or estimated based on the captured images.

[0069] Referring now to the accompanying drawings, in which reference numerals for various elements of the embodiments will be given, and embodiments will be discussed in the drawings to enable those skilled in the art to make and use the present technology.

[0070] The specific components, process steps, and other elements mentioned are not limiting. Furthermore, it is understood that when referring to alternative drawings, the same components will have the same reference numerals.

[0071] It should be further noted that these diagrams are illustrative and intended for technically skilled readers only, and are not necessarily drawn to scale. On the contrary, the various drawing scales, aspect ratios, and number of parts shown in the diagrams may have been deliberately distorted to make certain features or relationships easier to understand.

[0072] Refer to the attached diagram. Figure 1 According to one aspect of this technology, an optical display system 1 for a near-eye augmented reality display system includes an optical waveguide combiner 2 and a plurality of camera devices 6, 7. Figure 1 In one embodiment, the optical waveguide combiner 2 is a perspective combiner for augmented reality or mixed reality applications. However, in some other embodiments, the optical waveguide combiner may be non-perspective, for example, for VR applications.

[0073] The optical waveguide combiner 2 includes: a waveguide optical substrate 10; an optical input coupler 3 located on, within, or adjacent to the waveguide optical substrate 10; and an optical output coupler 8 located on or within the waveguide optical substrate 10. The optical input coupler 3 is optically coupled to the optical output coupler 8 via a portion of the waveguide optical substrate 10. The optical input coupler 3 can receive optical input, which includes collimated display image light rays 5. The receivable display image light rays 5 are input-coupled into the waveguide optical substrate 10 via the optical input coupler 3. Through total internal reflection in the waveguide, the image light rays 5 are guided along the waveguide to the output coupler 8.

[0074] The waveguide optical substrate 10 is made of a transparent waveguide material (e.g., but not limited to glass, plastic, or other suitable materials). The waveguide optical substrate 10 is a plate-shaped or planar waveguide substrate. In some other embodiments, the waveguide optical substrate 10 may be curved, or may have other shapes and configurations, and may have a bent or flexed axis.

[0075] The progressive extraction of the display image ray 5 into an output image ray 9 along the exit pupil expansion direction 4 of the waveguide optical substrate 10 is achieved through an output coupler 8. The output coupler 8 couples the display image ray 5 to the observer's eye 15.

[0076] Various optical input couplers and coupling methods can be used. Figure 1 In the illustrated embodiment, the optical input coupler 3 is a diffractive input coupler integrated into the waveguide. In some other embodiments, the optical input coupler 3 includes or comprises one or more diffractive optical elements separate from the waveguide substrate. In some other embodiments, the optical input coupler 3 includes a mirror integrated into the waveguide for reflecting and coupling the input light beam injected onto the side of the substrate into the waveguide. In still other embodiments, the optical input coupler 3 includes one or more other optical coupling components integrated or separate from the waveguide optical substrate 10 for coupling the light input 5 into the waveguide. As an example, such optical coupling components may be prisms or lens systems.

[0077] Output coupler 8 is a diffraction output coupler. The diffraction output coupler includes a diffraction element. The diffraction element includes a grating. In some other embodiments, the diffraction element includes a photonic crystal. In some other embodiments, output coupler 8 may be a mirror or a series of mirrors / reflectors or cascaded mirrors / reflectors, a prism array, a holographic device, or a combination thereof.

[0078] The virtual image formed by the output light 9 is a virtual object 13, which can be displayed at a spatial position relative to the real object 14 on the image plane 12, such that the virtual object 13 and the real object 14 can be observed by the eye 15 through the optical waveguide combiner 2 and appear superimposed. In a waveguide-based display, the selected initial focal plane is located at infinity to avoid loss of resolution. This is because the waveguide preserves angular information, not positional information. Typically, the displayed image exists as a virtual image at the image or focal plane 12, approximately three meters in front of the optical waveguide combiner 2. This is conceptually equivalent to existing at infinity, because the eye uses essentially the same angular position when observing an object approximately 3 meters away as it does when observing an object 100 meters or more away. Figure 1In this context, the virtual object 13 shown is a navigation icon, but it is understood that this is only one example of a virtual object. In some implementations, the virtual object 13 represents a real object 14 that is overlaid or superimposed on the virtual object.

[0079] Multiple camera devices 6 and 7 are optically coupled to the output of an optical coupler. Figure 1 In some embodiments, multiple camera devices are integrated within or on an optical waveguide substrate within the region of the output coupler 8. The multiple camera devices include a first camera device 6 and a second camera device 7 spaced apart from the first camera device 6. The camera devices 6 and 7 may be spatially spaced on or near the waveguide surface such that they are within a portion of the field of view (FOV) of the output coupler 8. In some other embodiments, the camera devices may be located in other locations, and / or there may be only one camera device or more than two camera devices. Each camera device is an ultra-miniature camera device with dimensions in millimeters or less, such as a wafer-level camera module. A non-limiting example of such a camera device is the OVM6948 CameraCubeChip (0.65mm x 0.65mm x 1.16mm).

[0080] Figure 2 This is a perspective view of optical display system 1, showing the overlapping fields of view of the different components of the system. Although not drawn to scale, Figure 2The relationship between the overlapping areas of the fields of view of the first camera device 6, the second camera device 7, and the eye-tracking box 16 is shown. The surface area of ​​the output coupler 8 that can be used by the eye-tracking box 16 is smaller than the total surface area of ​​the output coupler 8. The eye-tracking box 15 has a field of view 17 (the usable surface area of ​​the output coupler 8) at the output coupler 8 and a field of view 21 at the image plane 12 of the virtual object 13. Each camera device 6, 7 is positioned outside the eye-tracking box usable area of ​​the output coupler 8, i.e., outside the eye-tracking box field of view 17 of the output coupler 8, so that the camera device cannot be viewed from the eye-tracking box 16. The first camera device 6 has a first field of view 18 at the image plane 12. The second camera device 7 has a second field of view 19 at the image plane 12. Each camera device 6, 7 is arranged with viewpoints spaced apart, such that at the image plane 12, the first field of view 18 is adjacent to and overlaps with the second field of view 19. Therefore, the first camera device 6 and the second camera device 7 have a combined field of view in the image plane, which includes a first field of view 18 and a second field of view 19, and includes an overlapping region 20 of the first field of view 18 and the second field of view 19. The field of view 21 of the eye-tracking box in the image plane 12 is substantially located within the combined field of view of the first camera device and the second camera devices 6, 7. This division and arrangement of the camera device fields of view enables the use of ultra-miniature camera devices (e.g., wafer-level camera devices) for imaging the camera devices in the optical waveguide combiner 2, and allows for electronic self-calibration of the camera devices. Furthermore, since the waveguide camera devices 6, 7 have fields of view for both the virtual object 13 and the real object 14, the virtual object 13 can be registered with the real object 14, and / or the system can be registered with external sensors / camera devices.

[0081] Each camera device 6, 7 is positioned such that it has a field of view greater than 50% of the projector's field of view, which projects the display image beam 5 onto the combiner 2. Although the camera devices 6, 7 are shown in the figures as arranged symmetrically with a similar field of view, in other embodiments, the camera devices may be arranged asymmetrically, and / or may have different fields of view, and / or there may be more than two camera devices (provided that the images captured by the camera devices can be stitched together to form a larger virtual image). In some embodiments, one camera device has a field of view greater than 50% of the projector's field of view, while the other camera device may have a field of view less than 50% of the projector's field of view, provided that there is an overlapping area 20 sufficient for image stitching. As a non-limiting example, in the case of two camera devices, one camera device may have a field of view of 31% of the projector's field of view, and the other camera device may have a field of view of 71%, thus achieving 100% overlap and some degree of overlap to complete the image stitching.

[0082] In some other embodiments, only one camera device is optically coupled to the output coupler 8 of the optical combiner 2, and the field of view 21 of the eye-tracking box 16 at the image plane 12 is substantially within the field of view of the camera device. In still other embodiments, only a portion of the field of view 21 of the eye-tracking box 16 that displays the virtual object 13 at the image plane 12 is within the field of view of the camera device or within the combined field of view of the camera device. In some other embodiments, the field of view of the camera device at the image plane or the combined field of view of the camera device at the image plane substantially matches the field of view of the eye-tracking box.

[0083] In some alternative embodiments, one or more camera devices are optically coupled to the output pupil expansion output of the output coupler. In other embodiments, one or more camera devices optically coupled to the output of the output coupler are located near or away from the waveguide substrate, rather than directly coupled to or on the waveguide substrate. For example, in some embodiments, one or more camera devices are located within the eye-tracking box, or even behind the eye-tracking box.

[0084] like Figure 3 Best shown Figure 3 A perspective view of an optical waveguide combiner 2 according to some embodiments is shown, wherein the output coupler 8 may have a front exit pupil extension region 22 and an exit pupil extension region 23. Figure 3 In the illustrated embodiment, the output coupler 8 is a combined 2D extension and output grating on the main surface or within the waveguide, configured to receive image rays 5 from the input coupler 3 (input grating) and display the image. Figure 3 In this embodiment, eye-tracking box 16 is projected onto an output grating in a given field of view. The output grating is a cross-diffraction grating in which two diffraction gratings overlap. The two diffraction gratings are completely overlapped in or on a waveguide, wherein each of the two diffraction optical elements is configured to receive light from an input diffraction grating and couple light toward the other diffraction grating, which can then act as an output diffraction grating, providing an outer coupling order to the observer. An example of an optical waveguide combiner using such an output grating is described in U.S. Patent 10,114,220B2, filed July 28, 2015, and issued October 10, 2018, in the name of Wave Optics, Inc., entitled “Exit pupil expanding diffractive optical waveguiding device,” the entire contents of which are incorporated herein by reference. In some embodiments, the diffraction optical elements of the diffraction output coupler may be photonic crystals instead of diffraction gratings.

[0085] Cameras 6 and 7 are located outside the normally available area of ​​the output coupler 8, where a complete field of view cannot be obtained. The image rays extracted from the forward extension region are not used to view the virtual object image from the eye-tracking box. The output grating causes the entrance pupil at the input of the waveguide combiner to be replicated into multiple exit pupils. The forward extension region initially diffuses the pupils of the image rays on the surface of the output grating to replicate the entrance pupil, and the extension region expands the exit pupils so that the output grating directs the image rays toward the eye-tracking box for reception by the observer's eye. The forward extension region is defined as the pupil replication not yet completely filling the eye-tracking box, but still having some incomplete image information projected "forward". When the human eye moves through the waveguide within the angular range of the eye-tracking box (e.g., looking up, down, left, right), because the eye-tracking box 16 carries multiple exit pupils of the image rays, the image is perceived throughout the entire area of ​​the eye-tracking box as the human eye scans across the field of view.

[0086] In some implementations, for example, such as Figure 4 and Figure 5 As shown, in any embodiment disclosed herein, the optical waveguide combiner 2 and the imaging devices 6, 7 form the eyepiece of a pair of glasses. The projector 25 is arranged to inject image light into the input coupler 3. A non-limiting example of such a projector is as follows: Figure 6 As shown. In some embodiments, each camera device 6, 7 is a wafer-level camera device. As an example, the first camera device 6 and the second camera device 7 have a volume of approximately 2.5mm x 4mm (assuming the camera devices use the OmniVision Ov01A10 laptop webcam module and are registered trademarks of OmniVision Technologies, Inc.). TM (For Sale). These are just some examples of the dimensions of ultra-compact camera devices. Other sizes and types of known and / or future-developed ultra-compact camera devices are also envisioned.

[0087] exist Figure 4 In this design, the waveguide combiner 2 is oriented such that the input coupler 3 is positioned within the end 42 of the eyepiece 44. The region designed to be outside the field of view of the eye-tracking box 16 (in this case, the forward extension region 22) is located near the end 42 of the eyepiece. The output coupler region designed to be within the usable field of view of the eye-tracking box 16 is spaced apart from the end 42. Depending on the mode of image introduction into the input coupler 3, the projector can be positioned on the opposite side of the eyepiece, facing away from the eye, or it can be positioned on the same side of the eyepiece. Typically, the image pupil is introduced into the input coupler 3 along a direction orthogonal to the surface of the waveguide combiner 2. The projector barrel can be oriented substantially orthogonal to the input coupler 3, or it can be substantially parallel to the input coupler 3, in which case a rotating prism can be used to introduce light from the projector into the input coupler 3. [The last sentence appears to be incomplete and possibly refers to a different design.] Figure 4 Another optical waveguide combiner 2 arranged as shown is configured as the eyepiece of a pair of glasses to provide binocular vision.

[0088] In some aspects of this technology, any of the optical display systems comprising one or more optical waveguide combiners 2 and optionally one or more camera devices 6, 7 described herein can be implemented in a near-eye optical display system having an eyeglass form factor or a head-mounted form factor. In some embodiments, the near-eye optical display system further includes a light engine (projector or other light engine) and a battery. The near-eye optical display system can be an AR or MR optical display system.

[0089] As an example, combined Figure 6 Reference Figure 4 and Figure 5 , Figure 6 This is a perspective view of a pair of eyeglasses 40 according to some embodiments. The eyeglasses 40 includes an eyepiece 44, which includes an optical waveguide combiner 2 and imaging devices 6, 7. A projector 25 is arranged within the temple / arm of the eyeglasses 40, and an input coupler 3 is oriented towards the outermost edge of the eyepiece 44 in the end 42, orthogonal to the corresponding projector 25 located within the arm of the eyeglasses 40. Because the input coupler 3 is located in the end 42 of the eyeglasses, light can be projected from the projector 25 to the corresponding input coupler 3 and can be guided toward the output coupler 8 of the eyepiece 44. The output coupler 8 of the eyepiece 44 extends the light in two dimensions and can couple the light out from the waveguide toward the user's eye. In some other embodiments, the projector can be aligned along the top of the frame supporting the eyepiece 44; in this case, a rotating prism is used to couple the image light into the input coupler 3, which can be oriented below the top of the frame supporting the eyepiece 44.

[0090] The projector 25 shown in any of the embodiments disclosed herein may include any suitable near-eye image projector of any type, which may include optics configured to generate a display image and output the collimated display image at infinity to the optical waveguide combiner 2. In some embodiments, the projector 25 is a wafer-level projector. As a non-limiting example, the projector may be an LED, LCD display, or laser-based device or other light source device for generating the display image. The projector 25 may include LCoS, DMD, OLED, or any other display panel using emissive, transmissive, or reflective display technologies. The projector includes any optics required to collimate the display image to infinity, such as a collimating lens. In some embodiments, the projector does not include any optics (lenses, mirrors), for example, in this case only a phase (holographic) display. In some embodiments, the projector includes a self-emissive micro-LED display.

[0091] The method for camera device self-calibration in VR / AR / MR displays will now be described with reference to some aspects of this technology. To illustrate these methods, first refer to... Figure 7A , Figure 7B 7A and Figure 7B This is a schematic diagram illustrating existing theodolite calibration techniques. When calibrating a physical misalignment system for the line of sight, it is necessary to find a common reference. Typically, a pair of theodolites are used, first referencing each other, and then synchronously providing references for two other objects. As a non-limiting example, theodolite 71 picks up an angular reference of 0°, and theodolite 72 picks up an angular reference of 180° (see...). Figure 7A The theodolite 71 has a turning angle reference of 280°, and the theodolite 72 has a turning angle reference of 280° (see...). Figure 7B ).

[0092] like Figure 8 As shown, the optical waveguide substrate 10 of the optical waveguide combiner 2 itself has multiple references, such that mounting the camera device on the waveguide substrate will align the camera device without the need for calibration techniques (such as the theodolite calibration technique described above) to perform physical reference calibration for each other's visual axes. Any misalignment between the fields of view 18, 19 of the first camera device 6 and the second camera device 7 and the field of view of the eye tracker 16 at the image plane 12 can be corrected electronically.

[0093] Now refer to Figure 9A method 90 for self-calibrating a camera device of a VR / AR / MR display according to one aspect of the present technology is described. A display image including a reference point is projected onto an optical waveguide combiner (step 91). A first camera device and a second camera device are optically coupled to the output of the optical waveguide combiner, the first and second camera devices having a combined field of view including a first field of view of the first camera device and a second field of view of the second camera device, and including an overlapping region of the first and second field of view (step 92). The display image is displayed as a virtual image with a reference point at an image plane by the optical waveguide combiner, the virtual image being located within the overlapping region of the field of view of the first and second camera devices (step 93). A first image of the virtual image is captured by the first waveguide camera device (step 94). A second image of the virtual image is captured by the second waveguide camera device (step 95). The captured first and second images are electronically stitched together. To combine the first and second images into an image corresponding to the displayed virtual image, the captured first and second images are electronically adjusted by moving them to align with a common reference point (step 95). In some embodiments, the combined image is further corrected by determining the overlapping area of ​​the field of view by comparing the adjusted first and second images with the display image projected into the combiner (step 97). Based on the determined overlapping area of ​​the field of view, the adjusted first and second images are electronically corrected (step 98). The corrected and stitched first and second images substantially match the display image projected into the optical combiner.

[0094] By method 90, even when the first field of view of the first camera device and the second field of view of the second camera device are misaligned at the image plane, a substantially complete and distortion-free virtual image at the image plane is derived from the first image captured by the first camera device and the second image captured by the second camera device. Therefore, it is possible to electronically calibrate the first and second camera devices, taking into account camera device misalignment tolerances, without requiring mechanical registration of the camera devices or the use of additional sensors. The calibrated camera device system can then be used to determine and correct registration errors between the displayed virtual and real objects (thus correcting the virtual model of the real world). In some embodiments, method 90 is performed in real time during user use of the optical display system to account for any further misalignment that occurs during use of the display system.

[0095] Method 90 can be implemented in any of the embodiments of the optical display system disclosed herein or in any other optical display system using an optical waveguide combiner, in which multiple camera devices are optically coupled to the output coupler of the optical waveguide combiner. In some embodiments of method 90, the multiple camera devices are optically coupled to the front exit pupil expansion output of the output coupler.

[0096] As an example, now refer to Figures 10 to 15 Some implementations of method 90 in optical display system 1 according to some embodiments are described.

[0097] Figure 10 This is a schematic diagram of an optical display system, showing the fields of view 18 and 19 of the first waveguide camera device 6 and the second waveguide camera device 7 of the optical waveguide combiner 2, as well as the field of view 25 of the projector 25. Figure 11 A non-limiting example of a projector displaying an image with a reference point or common feature 27 is shown. The alignment reference point 27 is set on a vertical line running through the center of the image at both the top and bottom positions. Figure 11 This is merely a non-limiting example of the size and position of the alignment reference point or common feature 27. The position of the alignment reference point or common feature 27 is arranged by the combiner 2 for display within the overlapping area 20 of the first and second fields of view of the imaging devices 6, 7 (e.g., see...). Figure 1 or Figure 2 Therefore, if the overlapping area 20 is located at a different position rather than the center position (due to the camera devices having different FOV sizes, or the use of more than two camera devices), the alignment reference point or common feature 27 will also be located at a different position rather than the center position.

[0098] The minimum size of the alignment reference point 27 is determined by the correction constraints of the camera devices 6 and 7. In some embodiments, method 90 is performed using single-pixel registration, where each reference point 27 is a single pixel in size. As an example, the FOV of the camera device may be 25x30 or 960x1080, and the FOV of the projector may be 50x30 or 1920x1080. In some embodiments, the first camera device 6 and the second camera device 7 use wafer-level camera devices with correction constraints of less than 2 pixels (e.g., see OMNIVISION, a registered trademark of OmniVision Technologies Inc.). TM The OmniVision camera sensor for sale is 0V01A10, 1 / 11” 1.16um 1280x800, 2.5x2 mm package size) or less than 3 pixels (e.g., see the registered trademark OMNIVISION of OmniVision Technologies Inc.). TMThe OmniVision camera sensor for sale is 0H0TA / 31” 1.008um 400x400 550x550um (package).

[0099] The first image 13' captured by camera device 6 at FOV 18 and the second image 13” captured by camera device 7 at FOV 19 are stitched together, and as shown... Figure 12 As shown, this includes dual imaging reference points or common features. For example... Figure 13 As shown, Figure 12 The first image 13' and the second image 13" in the image are moved electronically to align misaligned reference points 27' and 27" and to stitch the images together. In some embodiments, the movement of the first image 13' and the second image 13" is achieved by rotational alignment, which operates on each image to ensure that the reference points become superimposed. The projector's field of view can be compared with the adjusted stitched image to determine the common area of ​​the first image 13' and the second image 13" which represents the overlapping area 20 of the first field of view 18 and the second field of view 19. Figure 14 The diagram shows electronic correction performed based on the determined overlapping region 20. Figure 13 The first image 13' and the second image 13' of the stitched image are used. When this correction is performed on the view of the camera device, a unified "virtual camera device" image can be generated. The corrected image essentially corresponds to the undistorted image that can be observed from the eye-tracking box 16. The misaligned first field of view 18 of the first camera device 6 and the second field of view 19 of the second camera device 7 are effectively corrected without physically moving the camera devices. This corrected image can then be used as computational feedback to determine and correct the registration error between the displayed virtual object and the real object (thereby correcting the virtual model of the real world).

[0100] In some embodiments, the sizes of the first field of view 18 of the first camera device 6 and the second field of view 19 of the second camera device 7 are determined such that the corrected stitched image is within the peripheral margin of the projector's field of view, so as to align and correct the first and second images within the alignment tolerance. As an example, when with... Figure 11 Compared to the projector's field of view in the middle, Figure 14 The corrected image region has an area within the peripheral margin range of the projector's field of view.

[0101] It is understood that the FOVs 18 and 19 of each camera device 6 and 7 are not limited to the FOVs shown in the accompanying drawings. As previously stated, the FOV of camera device 6 may differ from the field of view of camera device 7. Furthermore, as previously stated, the combined FOV of camera devices 6 and 7 may be larger than the projector's field of view. Additionally, the projector's field of view may be substantially located within the combined field of view of the first and second camera devices 6 and 7. In some embodiments, only a portion of the projector's field of view (including the real object and the displayable virtual object used for registration with the real object) is located within the combined field of view of the camera devices.

[0102] Figure 15 This is a block diagram of an optical display system including an optical waveguide combiner system 1 and a projector 25. A processor 302 is operatively connected to camera devices 6, 7 and projector 25. It is understood that the processor described herein and shown in the accompanying drawings may be located locally with the camera device / projector, and / or located outside the camera device / projector, and operatively connected to the camera device / projector via a suitable network (which may be wireless and / or wired).

[0103] When the processor 302 executes computer program instructions, it causes the optical display system to perform method 90 or other camera device calibration methods disclosed herein to take into account any misalignment of the first camera device 6 and the second camera device 7.

[0104] Various methods and systems for VR / AR / MR display registration based on some aspects of this technology will now be discussed.

[0105] Figure 16A method 160 for registering an AR / VR / MR display according to one aspect is illustrated. One or more virtual objects are generated based on one or more real objects using an external sensor, or one or more virtual objects are provided (step 161). A display image of the virtual objects is projected onto an optical waveguide combiner (step 162). The optical waveguide combiner displays the display image as a virtual object (step 163). One or more optical imaging devices optically coupled to the output coupler of the optical waveguide combiner capture images of the displayed virtual objects and real objects. The imaging devices have a field of view of both the real objects and the displayed virtual objects at the image plane of the combiner (step 164). The captured images of the real objects are compared with the captured images of the virtual objects. In some embodiments, the virtual objects displayed in the captured images are compared individually with one or more real objects in the captured images. In some embodiments, at least some or all of the virtual objects in the captured images are grouped together and compared with real objects or a corresponding set of real objects. The registration error between the real objects and the displayed virtual objects is determined based on the comparison (step 165). Adjust the displayed virtual image (step 167) to substantially correct the registration error (step 166).

[0106] Method 160 can be implemented in any embodiment of the optical display system disclosed herein, or in any other optical display system using an optical waveguide combiner in which one or more camera devices are optically coupled to the output of the optical waveguide combiner. Furthermore, according to any embodiment herein, each camera device can be arranged within the optical waveguide combiner. In some embodiments of method 160, one or more camera devices are optically coupled to the front exit pupil expansion output of the output coupler.

[0107] As an example, Figure 17 and Figure 18 An optical display system in which method 160 can be implemented according to some embodiments is shown. Figure 17 This is a schematic diagram of the optical waveguide combiner 2, showing a virtual object that is not aligned with the real object in the image plane. Figure 18 This is a block diagram of an optical display system, which includes an optical waveguide combiner 2, a projector 25, and an external sensor 31. When executed by computer program instructions, the processor 302 causes the optical display system to perform method 160 to account for any misalignment or registration errors between the displayed virtual object 13 and the real object 14.

[0108] The determination of registration error based on the comparison between virtual object 13 and real object 14 can be performed in various ways. In some embodiments, the optical display system or the user of the system first determines a common reference point in the real world that can be identified by external sensor 31 and waveguide camera device 6. As a non-limiting example, the common feature identified as a common reference point could be a detected edge, a projection point, or even a reference LED on a glove or boot. When executed by computer program instructions, processor 302 senses and / or images its reference position (i.e., the common reference point) using external sensors to generate a virtual object for display at the image plane of optical combiner 2. The virtual object exists in a virtual model generated by processor 302. The waveguide camera device system identifies real-world features from the image captured by the waveguide camera device and compares the identified real-world features with the common reference point of the displayed virtual object to determine the registration error between real-world object 13 and the displayed virtual object 14. Then, when executed by computer program instructions, processor 302 can adjust any one or more combinations of the position, size, and shape of the displayed virtual image to substantially correct the registration error. This allows the virtual object 13 displayed by the optical waveguide combiner 2 to be adjusted according to the determined registration error when the processor 302 is executed by the computer program instructions, so that when viewed from the eye-tracking box / camera device, the common features or reference points identified in the real object substantially coincide with the common features or reference points identified in the virtual object.

[0109] In some embodiments of any of the methods and / or systems (including method 160) disclosed herein for AR / VR registration, the generation of the virtual object 13 for display can be performed by an optical display system without the use of external sensors / cameras for the AR display. In some embodiments, a predetermined virtual model of the real world is provided to the system. In some other embodiments, the waveguide camera devices 6, 7 themselves are used to generate the virtual object 13 for display, which is extracted by the system from images of the real world captured by the waveguide camera devices. Figure 19 A block diagram of an optical display system according to some embodiments is shown, in which method 160 is implemented with step 161, which is modified to generate virtual object 13 by providing a virtual object of known size using a predetermined virtual model 32, instead of generating virtual object 13 using an external sensor of the AR display. Processor 302 causes the optical display system to execute method 160 when executed by computer program instructions.

[0110] Figure 20The diagram illustrates a method 200 for registering an AR / VR / MR display according to another aspect. One or more virtual objects of known size for display are provided or generated (step 201). An estimated distance and / or estimated spatial position between the real object and an optical waveguide combiner is determined (step 202). Based on the estimated distance and / or estimated spatial position, the virtual object is scaled for display by the optical waveguide combiner (step 203). A display image of the scaled virtual object is projected onto the optical waveguide combiner (step 204). The display image is displayed as a scaled virtual object by the optical waveguide combiner (step 205). One or more optical imaging devices optically coupled to the output coupler of the optical combiner capture images of the displayed virtual object and the real object. These imaging devices have fields of view for both the real object and the displayed virtual object (step 206). The captured real object is compared with the captured virtual object (step 207). Based on the comparison, a registration error between the real object and the displayed virtual object is determined (step 208). The displayed virtual object is adjusted to substantially correct the registration error (step 209).

[0111] Method 200 can be implemented using any embodiment of the optical display system disclosed herein or any other optical display system using an optical waveguide combiner, in which one or more camera devices are optically coupled to the output coupler of the optical waveguide combiner. Furthermore, according to any embodiment herein, each camera device can be arranged within the optical waveguide combiner. In some embodiments of method 200, multiple camera devices are optically coupled to the front exit pupil expansion output of the output coupler.

[0112] For example, method 200 can be implemented according to some embodiments. Figure 18 or Figure 19 In these embodiments, the processor 302 causes the optical display system to execute method 200 when executed by computer program instructions.

[0113] As an example, Figure 21 The use according to some implementation methods is shown. Figure 19 Example implementation method 200 of the optical display system. Figure 21This is a schematic diagram of an optical waveguide combiner 2 of an optical display system displaying a virtual object 13 that is not aligned with a real object 14. As shown, the system initially displays the virtual object 13 based on the estimated distance and / or spatial position of the real object 14 and the optical waveguide combiner 2. The estimated distance and / or estimated spatial position of the real object 14 and the optical waveguide combiner 2 can be determined using a depth sensor. In some embodiments, the estimated distance and / or estimated spatial position can be determined using the waveguide camera device itself without any external sensors, as will be described in detail below with reference to some embodiments. Waveguide camera devices 6 and 7 are used to compare the angular size of the real object 14 with that of the displayed virtual object 13. The difference in the angular size between the displayed real object 14 and the virtual object 13 reflects a registration error between the virtual object 13 and the real object 14. In this case, the estimation of the distance to the real-world object is incorrect. A correction value for the distance to the real-world object is calculated based on the difference. Since the size of the virtual object 13 is predetermined, the virtual object 13 is scaled relative to the corrected distance of the real object 14 based on the corrected distance and displayed by the optical combiner system. Processor 302 determines and applies a calibration factor so that the image projected into the waveguide is accurately scaled to match the real world. After such calibration, the virtual object 13 and the real object 14 are superimposed, as seen from the eye-tracking box / camera device. The optical display system and / or method effectively achieves the fitting of the real world to a virtual model 32 of known size.

[0114] In some aspects, the method and system for AR / VR display registration according to any embodiment of the single optical combiner 2 described herein can be implemented in a binocular configuration including right and left optical combiners 2. A waveguide camera captures a displayed stereoscopic virtual image and a stereoscopic real-world image, and compares the displayed stereoscopic virtual image with the stereoscopic real-world image to determine a registration error. For example, such a registration error might be caused by misalignment of the optical combiners. The displayed stereoscopic virtual image can be corrected by adjusting the virtual image displayed by the optical combiner according to the registration error, so that the stereoscopic virtual image is displayed correctly.

[0115] It is necessary to determine the actual location of real objects in space in order to perform accurate VR / AR / MR displays. Computers can store 3D models of the world in which all dimensions are known with good accuracy. By making some assumptions about virtual locations in space, this world can be presented in 2D form and provided to users through display means such as computer monitors, virtual reality, or augmented reality displays. In the case of augmented reality, a common problem arises: the displayed 2D image of the world may not match the real world. This is because the assumptions about the virtual locations in space do not match their actual locations. To accurately determine this location, much effort has been made in sensor technology, utilizing a variety of sensors (…). Figure 29 The 318 in the text includes GPS, LIDAR, RADAR, and Time-of-Flight (ToF). Figure 29 (290 in the middle) and many other sensors.

[0116] Figure 22 The diagram illustrates a method 220 for registering a VR / MR / AR display according to one aspect of the present technology. A first and second imaging device capture images using the respective outputs of a first and a second optical waveguide combiner optically coupled to a binocular display system. Each imaging device has a field of view for one or more real objects and one or more displayable virtual objects (step 221). An estimated distance and / or estimated spatial position between the real objects and the optical waveguide combiner is determined using the optical imaging device (step 222). Based on the estimated distance and / or estimated spatial position, a virtual object is generated for display by the optical waveguide combiner (step 223). The display image of the virtual object is projected onto the optical waveguide combiner (step 224). The display image is displayed as a virtual object by the optical waveguide combiner (step 225). Images of the displayed virtual object and the real object are captured using the optical imaging device (step 226). The displayed virtual object is compared with the real object using the captured images (step 227). The registration error between the real object and the displayed virtual object is determined based on the comparison (step 228). The displayed virtual objects are adjusted to substantially correct the registration error (step 229).

[0117] Method 220 can be implemented in any embodiment of the optical display system disclosed herein, arranged in a binocular configuration or any other optical display system using an optical waveguide combiner within a binocular system. The first and / or second optical waveguide combiner can be any of the optical waveguide combiners disclosed herein. Furthermore, according to any embodiment herein, each first camera device can be optically coupled to the first optical waveguide combiner. Furthermore, according to any embodiment herein, each second camera device can be optically coupled to the second optical waveguide combiner. In some embodiments of method 220, for each optical waveguide combiner, one or more camera devices are optically coupled to the front exit pupil of the output coupler of the optical waveguide combiner to expand the output.

[0118] As an example, now refer to Figure 23 Describes the implementation of method 220 according to some embodiments. Figure 23 A block diagram of a binocular optical display system is shown. The binocular system includes optical waveguide combiners 2L and 2R, a projector 25, and a processor 302. When executed by computer program instructions, the processor 302 causes the binocular optical display system to perform method 220.

[0119] Depending on the implementation, the process of using an optical camera to determine the estimated distance and / or estimated spatial position of a real object from an optical combiner (step 222) can be performed in different ways.

[0120] In some embodiments of this technology, the distance between the real object and the optical waveguide combiner estimated using an optical waveguide camera (step 222) is determined based on a depth-sensing method. In other embodiments, the distance between the real object and the optical waveguide combiner estimated using an optical camera (step 222) is determined based on stereo vision.

[0121] Furthermore, when the optical waveguide combiners are used in a binocular configuration (one for the right eye and one for the left eye), the registration and / or calibration methods and / or systems for each optical waveguide combiner, implementing any of the embodiments disclosed herein, enable electronic correction of the image displayed by each waveguide to ensure proper alignment of the combined binocular image. Therefore, a person viewing through this pair of waveguides will correctly perceive the enhanced image; without a self-correction system, a person would not perceive an accurate binocular image, and in fact, a person might experience headaches due to misalignment of the images projected by each waveguide.

[0122] Figure 24The diagram illustrates a method 240 for determining distance in an AR / MR / VR display according to one aspect of the present technology. A first and second imaging device, optically coupled to the respective outputs of a first and a second optical waveguide combiner of a binocular display system, capture images stereoscopically.

[0123] The camera device has a field of view for one or more real objects (step 241). One or more virtual objects are extracted from the captured image of the real objects. The extracted virtual objects are electronically identified (step 242). Depth-sensing cues are determined based on the image captured by the optical camera device (step 243). Based on the determined depth-sensing cues and object identification (step 244), the distance between the real objects and the optical combiner is determined (step 245).

[0124] Method 240 can be implemented in any embodiment of the optical display system disclosed herein (or in any other optical display system using an optical waveguide combiner in a binocular system), in which one or more camera devices are optically coupled to a corresponding output of the optical waveguide combiner. The first and / or second optical waveguide combiner can be any of the optical waveguide combiners disclosed herein. Furthermore, each camera device can be arranged in the optical waveguide combiner according to any embodiment herein. Additionally, in some embodiments, for each optical waveguide combiner, one or more camera devices are optically coupled to the front exit pupil enlargement output of the output coupler of the optical waveguide combiner.

[0125] As an example, according to some implementations, method 240 can be... Figure 23 This is implemented in a dual-eye optical display system. When the processor 302 is executed by computer program instructions, it causes the dual-eye optical display system to execute method 240.

[0126] In step 244, depth-sensing cues can be determined based on the projection angle of the waveguide camera. As an example, Figure 25 The following are illustrated according to some embodiments. Figure 23 The exemplary difference in projection angles of the eye / camera device combined in the left waveguide substrate 2L and right waveguide substrate 2R of the optical display system for viewing the real object 14 is used for the purpose of object recognition and determining depth-sensing cues. The depth-sensing cues determined from the image captured by the waveguide camera device may include or contain the following: a first cue (strongest) (near to far distance), object size versus expected object size (e.g., large = near, small = far); a second cue (near to medium distance), object convergence, the angular difference between the eye / camera device. Always inward – convergence; a third (weak) cue (only near distance), the object accommodation that the eye / camera device must focus on.

[0127] In some aspects of this technology, method 240 is employed in any of the methods for registering AR / VR / MR displays in the embodiments disclosed herein to estimate or determine the distance or spatial position of a real object and an optical combiner. As an example, processing step 222 may include method 240.

[0128] Furthermore, the object recognition technique employed in method 240 can be used in any registration / calibration method disclosed herein to generate a predetermined virtual object for display.

[0129] Figure 26 The diagram illustrates another aspect of the present technology, a method 260 for a stereoscopic vision-based AR / MR / VR display. One or more virtual objects are generated or provided for display (step 261). A display image of the virtual object is projected onto a first and second optical waveguide combiner system of a binocular optical combiner system (step 262). The display image is stereoscopically displayed as a virtual object by the optical waveguide combiner (step 263). A first and second imaging device, optically coupled to the respective outputs of the first and second optical waveguide combiners, capture stereoscopic images. The imaging devices have a field of view for displaying both virtual and real objects (step 264). The displayed virtual object is compared with the real object using the captured stereoscopic image (step 265). Based on the comparison result of comparing the displayed virtual object with the real object using the captured stereoscopic image (step 266), the distance and / or spatial position of the real object relative to the optical waveguide combiner is determined (step 267).

[0130] In some embodiments of step 265, the virtual objects displayed in the captured image are individually compared with real objects or a group of real objects in the captured image. In some embodiments, at least some or all of the virtual objects in the captured image are grouped together and compared with real objects or corresponding groups of real objects.

[0131] In some implementations, step 266 is performed by making a primary estimate of the distance (if the object size is known) by determining the difference in the angular range of the displayed virtual object relative to the real object as viewed from the stereoscopic image. In some implementations, a secondary estimate of the distance to a nearby object may be calculated from the stereoscopic image based on stereoscopic visual disparity (binocular parallax).

[0132] Using the determined distance and / or spatial location, the displayed virtual objects are adjusted by scaling to substantially correct the spatial positional differences between the displayed virtual objects and the real objects.

[0133] Method 260 can be implemented in any embodiment of the optical display system disclosed herein, or in any other optical display system using an optical waveguide combiner in a binocular system, in which one or more camera devices are optically coupled to the output coupler of a respective optical waveguide combiner. The first and / or second optical waveguide combiner can be any of the optical waveguide combiners disclosed herein. Furthermore, each camera device can be optically coupled to an optical waveguide combiner according to any embodiment herein. In some embodiments, for each optical waveguide combiner, one or more camera devices are optically coupled to the front exit pupil enlargement output of the output coupler of the optical waveguide combiner.

[0134] As an example, method 260 according to some implementations can be derived from... Figure 23 The optical display system is implemented. When the processor executes computer program instructions, it causes the dual-optical optical display system to execute method 260.

[0135] In step 266, the binocular parallax in the stereoscopic image is determined based on the presence or absence of hyper-stereoscopic vision or hyper-stereoscopic viewing conditions. As an example, Figure 27 The following are illustrated according to some embodiments. Figure 23 An example difference in the projection lines of the imaging devices combining the right waveguide combiner 2L and the left waveguide combiner 2R in the optical display system for viewing a stereoscopic real object 14 and a virtual object 13. When the displayed virtual object 13 is too close to the waveguide combiners 2L and 2R compared to the real object 14, the angular difference seen by the imaging devices 6 and 7 is larger than expected, such as... Figure 27 As shown (super stereoscopic vision). When the displayed virtual object 13 is too far from the optical waveguide combiner 2L, 2R compared to the real object 14, the angular difference seen by the camera devices 6, 7 is smaller than expected, such as... Figure 28 As shown (hyper-stereo vision). The distance between the real object 14 and the combiner 2 can be estimated based on the virtual stereo error.

[0136] In some aspects of this technology, method 260 is employed in any of the methods for registering AR / VR / MR displays in the embodiments disclosed herein to estimate or determine the distance or spatial position of a real object and an optical combiner. As an example, process step 222 may include method 260.

[0137] An exemplary application of some of the methods and optical display systems disclosed herein is for firefighters navigating poorly lit, smoke-filled rooms. Firefighters can be equipped with protective helmets that include a series of external sensors, including image depth sensors, thermal sensors, night vision cameras, imaging devices, and an augmented reality waveguide on the inner side of the helmet, including an optical waveguide camera. Image data from the external sensors can be combined and presented on the waveguide display, enabling firefighters to navigate safely in environments even in very low light conditions where objects may be indistinguishable to the eye, particularly those low to the ground that could pose a tripping hazard. Because the system continuously updates scaling parameters, the risks of an object appearing to be in a certain location based on the projected image from the waveguide, when in reality it is in a different location, or an object appearing to have a specific size different from reality, are significantly reduced. In such cases, the likelihood of a firefighter colliding with an object is greatly reduced. Given that the imaging device is more accurate than the human eye, the system and method of this embodiment ensure the accuracy of the projected image in both space and size. Therefore, even if the user (rather than the camera device) cannot actually see any objects outside the virtual display, this technology, along with optical and computational feedback, can still function and generate a corrected image of the world.

[0138] When an object is very close (arm's length), its angle differs from when the object is at a distance; therefore, the correction scale and angular position are very important. Another advantage is that some implementations of optical display systems and methods allow for compensation for differences in depth of focus, enabling users to view objects at arm's length as well as objects at greater distances.

[0139] Figure 29This is a schematic representation of an example machine in the form of a computer system 300, which is an example of one or more computers or processors mentioned herein, and within the computer system 300, a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. In various example implementations, the machine operates as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate as a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a module on an AR / VR / MR display, or a remote computing device that interfaces with a module interface electronics device on an AR / VR / MR display. The remote computing device can be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, portable music player (e.g., a portable hard disk audio device such as a Moving Picture Experts Group Audio Layer 3 (MP3) player), network device, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequential or other instructions) that specify the actions the machine should perform. Furthermore, although only a single machine is shown, the term "machine" should also be considered to include any collection of machines that, individually or in combination, execute a set (or more) of instructions to perform any one or more of the methods discussed herein.

[0140] Example computer system 300 includes one or more processors 302 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), main memory 306, and optional static memory 308, which communicate with each other via bus 328. Computer system 300 may also include a video display 314 (e.g., a liquid crystal display (LCD), a touch-sensitive display). Computer system 300 may also include an alphanumeric input device 296 (e.g., a keyboard, keypad, touchpad, touch display, buttons; see also...) Figure 29 The computer system 300 may include a cursor control device (e.g., a mouse), a voice recognition or biometric authentication unit (not shown), a drive unit 320 (also referred to as a disk drive unit), a signal generation device 324 (e.g., a speaker), a motor / servo device 326, and a network interface device 312. The computer system 300 may also include a data encryption module (not shown) for encrypting data. The drive unit 320 includes a computer or machine-readable medium 322 on which one or more sets of instructions and data structures (such as instruction 304) implementing or utilizing any one or more of the methods or functions described herein are stored. Instructions 304 may also reside wholly or at least partially in main memory 306 and / or processor 302 during execution by the computer system 300. Main memory 306 and processor 302 may also constitute a machine-readable medium.

[0141] Instruction 304 can also be sent or received over a network via network interface device 312 using any of several known transport protocols, such as Hypertext Transfer Protocol (HTTP). While machine-readable medium 322 is shown as a single medium in the example embodiment, the term "computer-readable medium" should be considered to include a single or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing one or more sets of instructions. The term "computer-readable medium" should also be considered to include any medium capable of storing, encoding, or carrying a set of instructions executable by a machine and causing the machine to perform any or more methods of this application, or any medium capable of storing, encoding, or carrying data structures used by or associated with such a set of instructions. Therefore, the term "computer-readable medium" should include, but is not limited to, solid-state storage, optical and magnetic media, and carrier signals. Such media may also include, but is not limited to, hard disks, floppy disks, flash memory cards, digital video disks, random access memory (RAM), read-only memory (ROM), etc. The example embodiments described herein can be implemented in an operating environment including software installed on a computer, in hardware, or in a combination of software and hardware.

[0142] For ease of explanation, numerous specific details have been set forth to provide a comprehensive understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure can be practiced without these specific details. In other instances, structures and apparatus are shown only in block diagram form to avoid obscuring the disclosure.

[0143] Throughout this specification, the terms "one embodiment" or "implementation" refer to a specific feature, structure, or characteristic described in relation to an embodiment that is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," or "according to an embodiment" (or other phrases with similar meanings) appearing throughout this specification do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Additionally, depending on the context discussed herein, singular terms may include their plural forms, and plural terms may include their singular forms. Similarly, hyphenated terms (e.g., "on-demand") may occasionally be used interchangeably with their non-hyphenated versions (e.g., "ondemand"), capitalized terms (e.g., "Software") may be used interchangeably with their non-capitalized versions (e.g., "software"), plural terms may be indicated with or without apostrophes (e.g., PE's or PEs), and italicized terms (e.g., "N+1") may be used interchangeably with their non-italicized versions (e.g., "N+1"). Such occasional interchangeability should not be considered contradictory.

[0144] Furthermore, some implementations may be described as "means" for performing a task or a set of tasks. It is understood that "means" here can be represented by a structure, such as a processor, memory, I / O device (e.g., a camera device), or a combination thereof. Alternatively, "means" may also include algorithms describing functional or method steps, while in other implementations, "means" may be represented by mathematical formulas, prose, flowcharts, or signal diagrams.

[0145] The terminology used herein is for descriptive purposes only and is not intended to limit the technology. The singular forms “a,” “an,” and “the” used herein also include the plural forms unless the context clearly indicates otherwise. It should be further understood that the terms “comprising” and / or “including” as used herein specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0146] First, it should be noted that the terms "coupling," "connection," "linkage," and "electrical connection," etc., are used interchangeably in this document to generally refer to the state of an electrical / electronic connection. Similarly, when a first entity electrically sends and / or receives (whether via wired or wireless means) information signals (whether containing data information or non-data / control information) to a second entity, regardless of the type of these signals (analog or digital), the first entity is considered to be in a "communicating" state with the second entity (or entities). It should be further noted that the various figures (including component diagrams) shown and discussed herein are for illustrative purposes only and are not drawn to scale.

[0147] Those skilled in the art will recognize that any one or more processors, camera devices, or other electronic devices disclosed herein can be configured to provide Internet access to one or more computing devices coupled to an Internet service or other network, and the computing devices may include one or more processors, buses, storage devices, display devices, input / output devices, etc. Furthermore, those skilled in the art will understand that the Internet service or other network may be coupled to one or more databases, repositories, servers, etc., and these databases, repositories, servers, etc., may be used to implement any implementation of the present disclosure described herein.

[0148] All means or steps plus functional elements in the appended claims are intended to include any structure, material, action, and equivalent that, in combination with other elements of the specific claims, perform the function. The description of the present technology has been presented for purposes of illustration and description, but such description is not intended to be exhaustive or to limit the technology to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present technology. Exemplary embodiments were chosen and described in order to best explain the principles of the present technology and its practical application, and to enable others skilled in the art to understand various embodiments of the present technology with various modifications suitable for the particular purpose considered. Various aspects of the present technology have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present technology. It will be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram, and elsewhere in the specification.

[0149] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus or other device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing comprising instructions that implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0150] In some embodiments, the method steps, processes, functions / actions disclosed herein may be performed in different orders or combinations. In some embodiments, one or more of the methods, processes, functions / actions disclosed herein may be omitted.

[0151] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operational steps to be performed on the computer, other programmable apparatus or other equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide a process that implements the function / action specified in one or more boxes of a flowchart and / or block diagram.

[0152] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present technology. In this regard, each block in the flowcharts or block diagrams, or each method step or process summarized, described, and / or claimed in this application, may represent a module, code segment, or code portion comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative embodiments, the functions marked in the blocks may not appear in the order indicated in the figures. For example, two blocks shown consecutively in the figures may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified function or action.

[0153] It should be understood that the embodiments described herein are merely illustrative, and modifications can be made by those skilled in the art. Therefore, this technology should not be considered limited to the disclosed embodiments, but rather to the embodiments defined in the appended claims. It should also be understood that, for the avoidance of doubt, any feature associated with any particular embodiment may be used in combination with other embodiments.

[0154] While embodiments of the optical display and method have been described in detail, it should be apparent that modifications and alterations can be made to these embodiments, all of which fall within the true spirit and scope of the present invention. Based on the foregoing description, it should be understood that optimal dimensional relationships of the components of the present invention, including variations in size, material, shape, form, function, and mode of operation, assembly, and use, are apparent to those skilled in the art, and all equivalent relationships shown in the figures and described in the specification are intended to be included within the present invention.

[0155] Therefore, the above content is intended only to illustrate the principles of this technology. Furthermore, since those skilled in the art can easily make many modifications and changes, it is not intended to limit this technology to the exact structures and operations shown and described; therefore, all appropriate modifications and equivalent methods can be employed within the scope of this technology.

Claims

1. An optical display system, comprising: an optical waveguide combiner; wherein the optical waveguide combiner comprises: a waveguide optical substrate; an optical input coupler on, in, or adjacent to the waveguide optical substrate; and an optical output coupler on or in the waveguide optical substrate, the optical input coupler coupled to the optical output coupler via the waveguide optical substrate; and a plurality of cameras; wherein the plurality of cameras comprises a first camera and a second camera; wherein the first camera is optically coupled to the optical output coupler and has a first field of view of at least one virtual object that can be displayed at an image plane through the optical waveguide combiner; the virtual object that can be displayed is located within a field of view of an eyebox of the optical waveguide combiner; wherein the second camera is optically coupled to the optical output coupler and has a second field of view of the virtual object that can be displayed through the optical waveguide combiner; wherein at the image plane, a combined field of view of the first camera and the second camera comprises the first field of view of the first camera and the second field of view of the second camera, and comprises an overlap region of the first field of view and the second field of view; and wherein at the image plane, the at least one virtual object that can be displayed is located within the combined field of view of the first camera and the second camera.

2. The optical display system of claim 1, wherein, the optical waveguide combiner is see-through; and wherein at least one real object is viewable within the combined field of view of the first camera and the second camera.

3. The optical display system of claim 2, wherein, the plurality of cameras are located in a region outside of a field of view of an eyebox of the optical waveguide combiner system.

4. The optical display system of claim 3, wherein, the optical output coupler comprises a front exit pupil expansion region and an exit pupil expansion region, and wherein the first camera and the second camera are optically coupled to the front exit pupil expansion region.

5. The optical display system of claim 4, further comprising a projector configured to project a display image of the virtual object into the input coupler.

6. The optical display system of claim 5, wherein, at the image plane, a field of view of the eyebox and / or a field of view of the projector is within the combined field of view of the first camera and the second camera.

7. The optical display system of claim 5, further comprising at least one processor operably coupled to the projector and the first camera and the second camera, wherein, the at least one processor is configured to cause the projector to project a display image of the virtual object, the display image comprising a fiducial point, wherein the fiducial point of the virtual object that can be displayed is located within the overlap region of the first field of view of the first camera and the second field of view of the second camera; and wherein the at least one processor is configured to electronically stitch together a first image and a second image respectively captured from the first camera and the second camera by adjusting the first image and the second image to align the fiducial point in the first image with the fiducial point in the second image.

8. The optical display system of claim 4, wherein, The light output coupler includes any one or a combination of a prism array, a cascaded mirror, a diffraction grating, and a holographic device.

9. The optical display system of claim 4, wherein, The light output coupler includes an optical diffraction element.

10. The optical display system of claim 9, wherein, The optical diffraction element includes two diffractive optical elements at least partially superimposed on each other in or on the waveguide optical substrate.

11. The optical display system of claim 10, wherein, The light output coupler includes a combined 2D expanded output diffraction grating.

12. The optical display system of claim 4, wherein, Each of the plurality of cameras includes a wafer-level camera; and wherein the wafer-level cameras are incorporated in or on the light waveguide combiner.

13. The optical display system of claim 12, wherein, The light input coupler includes a light input diffraction grating, and wherein the light input diffraction grating is in optical communication with a light output grating without any intermediate light diffraction grating between the light input diffraction grating and the light output grating.

14. The optical display system of claim 1, wherein, The light waveguide combiner is incorporated in an eyepiece of a pair of glasses.

15. A method for camera calibration for AR / MR / VR displays, the method comprising: projecting a display image including a fiducial point into a light waveguide combiner, optically coupling a first camera and a second camera to an output of the light waveguide combiner, the first camera and the second camera having a combined field of view, the combined field of view including a first field of view of the first camera and a second field of view of the second camera and including an overlap region of the first field of view and the second field of view; displaying, by the light waveguide combiner, the display image as a virtual image having the fiducial point at an image plane, wherein the displayed virtual image is located within the combined field of view of the first camera and the second camera, wherein the fiducial point of the displayed virtual image at the image plane is located within the overlap region of the first field of view and the second field of view; capturing, by the first camera, a first image of the virtual image; capturing, by the second camera, a second image of the virtual image; and electronically stitching together the captured first image and the captured second image; wherein electronically stitching together the captured first image and the captured second image includes electronically adjusting the captured first image and the captured second image by shifting the captured first image and the captured second image to align the fiducial point of the first image with the fiducial point of the second image.

16. The method of claim 15, wherein, electronically stitching together the captured first image and the captured second image further includes determining the overlap region of the first field of view and the second field of view by comparing the adjusted first image and the adjusted second image to the display image projected into the light waveguide combiner; and electronically correcting the adjusted first image and the adjusted second image according to the determined overlap region.

17. The method of claim 16, wherein, optically coupling the first camera and the second camera to the output of the light waveguide combiner includes optically coupling the first camera and the second camera to a front exit pupil expanded output of an output coupler of the light waveguide combiner.

18. A method of registering a VR / AR / MR display, the method comprising: providing at least one virtual object for display; projecting a display image of the virtual object into an optical waveguide combiner; displaying, by the optical waveguide combiner, the display image as the virtual object; capturing, using one or more optical cameras optically coupled to an output of the optical waveguide combiner, images of the displayed virtual object and a real-world scene, the cameras having a field of view of the real-world scene and the displayed virtual object; comparing, using the captured images, the virtual object to at least one real object of the real-world scene; determining, based on the comparison, a registration error between the virtual object and the real object; adjusting the displayed virtual object to substantially correct for the registration error.

19. The method of claim 18, wherein comparing, using the captured images, the virtual object and the real object includes: identifying a common feature or fiducial point of the virtual object and the real object; and comparing the identified common feature or fiducial point in the virtual object to the identified common feature or fiducial point in the real object; wherein determining, based on the comparison, a registration error between the virtual object and the real object includes: determining a difference between a location of the identified common feature or fiducial point in the virtual object and a location of the identified common feature or fiducial point in the real object; and wherein adjusting a position of the displayed virtual object to substantially correct for the registration error includes adjusting any one or a combination of a size, shape, and position of the virtual object such that the identified common feature or fiducial point in the virtual object substantially coincides with the identified common feature or fiducial point in the real object.

20. The method of claim 19, wherein, providing the virtual object includes generating the virtual object from a real object using an external sensor.

21. A method of registering a VR / AR / MR display, the method comprising: generating or providing at least one virtual object of a known size; scaling the virtual object, based on an estimated distance of at least one real object from an optical waveguide combiner, for display by the optical waveguide combiner; projecting a display image of the scaled virtual object into the optical waveguide combiner; displaying, by the optical waveguide combiner, the display image as the virtual object; capturing, using one or more optical cameras, the displayed virtual object and the real object, the or each camera having a field of view of the real object and the displayed virtual object; comparing, using the captured images, the real object to the virtual object; determining, based on the comparison, a registration error between the virtual object and the real object; and adjusting the displayed virtual object to substantially correct for the registration error.

22. The method of claim 21, wherein comparing the captured virtual object to the real object includes: determining, from the captured images, an angle and / or an angle size of the at least one real object from a viewpoint of the camera; ​ determining, from the captured images, an angle and / or an angle size of the at least one virtual object from a viewpoint of the camera; wherein determining, based on the comparison, a registration error between the virtual object and the real object comprises determining a difference in angle and / or angle size of the at least one virtual object relative to the at least one real object; and wherein adjusting the displayed virtual object to substantially correct for the registration error comprises adjusting any one or a combination of size, shape, and spatial position of the at least one virtual object, whereby, when viewed from an eyebox of the optical waveguide combiner, the at least one virtual object appears in substantially the same spatial position as the real object.

23. A method of registering a VR / MR / AR display system, the method comprising: capturing stereoscopic images using optical cameras of a binocular system, the optical cameras comprising a first optical camera and a second optical camera, the first and second optical cameras being optically coupled to respective outputs of first and second optical waveguide combiners of the binocular system and having a field of view of at least one displayable virtual object and at least one real object; determining, using the optical cameras, an estimated distance of the real object from the optical waveguide combiners, and / or determining, using the optical cameras, an estimated spatial position of the real object; generating, based on the estimated distance and / or the estimated spatial position, a virtual object for display by the optical waveguide combiners; projecting an image of the virtual object into the optical waveguide combiners; displaying, by the optical waveguide combiners, the image as the virtual object; capturing, using the optical cameras, images of the displayed virtual object and real object; comparing, using the captured images, the displayed virtual object to the real object; determining, based on the comparison, a registration error between the real object and the virtual object; adjusting the displayed virtual object to substantially correct for the registration error.

24. The method of claim 23, wherein, determining, using the optical cameras, an estimated distance of the real object from the optical waveguide combiners comprises: extracting, from the captured images of the real object, a virtual image representing the real object; identifying the extracted virtual object; determining, from the images captured by the optical cameras, depth perception cues; estimating, based on the determined depth perception cues and object identification, a distance of the real object from the optical waveguide combiners.

25. The method of claim 23, wherein, determining, based on the comparison, a registration error between the real object and the virtual object comprises: determining, from binocular disparity in the stereoscopic images, a difference in spatial position between the displayed virtual object and the real object.

26. A method of determining distance in an AR / MR / VR display, the method comprising: capturing stereoscopic images using optical cameras of a binocular optical waveguide combiner system, the optical cameras having a field of view of at least one displayable virtual object and at least one real object; extracting, from the captured images of the real object, at least one virtual object representing the real object; identifying the extracted virtual object; determining depth perception cues from images captured by the optical camera; determining a distance of the real object from the optical waveguide combiner based on the determined depth perception cues and object identification.

27. A method of determining distance in an AR / MR / VR display, the method comprising: generating or providing at least one virtual object for display; projecting a display image of the virtual object into an optical waveguide combiner of a binocular optical combiner system; stereoscopically displaying the display image as the virtual object by the optical waveguide combiner; capturing stereoscopic images of the displayed virtual object and at least one real object using one or more optical cameras incorporated in each optical waveguide combiner of the binocular optical combiner system, the one or more cameras having a field of view of the real object and the displayed virtual object; comparing the displayed virtual object to the real object using the captured stereoscopic images; determining binocular disparity in the stereoscopic images; and determining a distance of the real object from the virtual object from the binocular disparity. ​

Images