Augmented reality display system

Abstract

An augmented reality display system is configured to align 3D content with a real object using fiducial markers. The augmented reality display system can optionally include a depth sensor configured to detect a position of the real object. The augmented reality display system can also include a light source configured to illuminate at least a portion of the object with non-visible light, and a light sensor configured to form an image using a reflected portion of the non-visible light. Processing circuitry of the display system can identify a position marker based on a difference between the emitted light and the reflected light, and determine an orientation of the real object based on the position of the real object and the position of the position marker.

Description

[0001] This application is a divisional application of the patent application with international application number PCT / US2017 / 065850, international application date December 12, 2017, Chinese national application number 201780086089.X, entitled "Augmented Reality Display System".

[0002] Cross-references to related applications

[0003] This application claims priority from U.S. Provisional Application No. 62 / 433,767, filed December 13, 2016, entitled “3D OBJECT RENDERING USING DETECTED FEATURES”, under 35 U.S.SC §119(e), the entire contents of which are incorporated herein by reference and for all purposes. The entire contents of the following patent applications are incorporated herein by reference: U.S. Application No. 14 / 555,585, filed November 27, 2014; U.S. Application No. 14 / 690,401, filed April 18, 2015; U.S. Application No. 14 / 212,961, filed March 14, 2014; U.S. Application No. 14 / 331,218, filed July 14, 2014; and U.S. Application No. 15 / 072,290, filed March 16, 2016. Technical Field

[0004] This disclosure relates to optical devices, including virtual reality and augmented reality imaging and visualization systems. Background Technology

[0005] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to the user in a way that appears real or can be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information that is opaque to other actual real-world visual inputs; augmented reality or "AR" scenarios typically involve presenting digital or virtual image information as an enhancement of the visualization of the real world surrounding the user. Mixed reality or "MR" scenarios are a type of AR scenario and typically involve virtual objects integrated into and responding to the natural world. For example, in an MR scenario, AR image content can be obscured by real-world objects or otherwise perceived as interacting with real-world objects.

[0006] refer to Figure 1The image depicts an augmented reality scene 10, in which an AR user sees a real-world park-like setting 20 characterized by people, trees, and buildings in the background, as well as a concrete platform 30. In addition to these elements, the AR user also perceives that he "sees" "virtual content," such as a robot statue 40 standing on the real-world platform 30, and a flying cartoonish avatar 50 that appears to be anthropomorphized Bumblebee, even though these elements 40 and 50 do not exist in the real world. Because the human visual perception system is complex, producing AR technology that provides a comfortable, natural, and richly presented experience of virtual image elements within other virtual or real-world image elements is challenging.

[0007] The systems and methods disclosed in this paper address a variety of challenges related to AR and VR technologies. Summary of the Invention

[0008] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither the summary of the invention nor the following detailed description is intended to limit or restrict the scope of the inventive subject matter.

[0009] Example

[0010] 1. An augmented reality display system configured to align 3D content with real-world objects, the system comprising:

[0011] A frame that is configured to be mounted on the wearer;

[0012] Augmented reality display, which is attached to a frame and configured to direct images to the wearer's eyes;

[0013] A light source configured to illuminate at least a portion of an object by emitting invisible light;

[0014] A light sensor configured to image the portion of the object illuminated by the light source using the invisible light; and

[0015] The processing circuit is configured to determine information about the location, orientation, or both of an object based on one or more characteristics of features in an image formed using a reflective portion of invisible light.

[0016] 2. The augmented reality display system according to Example 1, wherein features in an image formed using a reflective portion of invisible light are invisible to the eye.

[0017] 3. The augmented reality display system according to any one of Examples 1-2, wherein invisible light includes infrared light.

[0018] 4. An augmented reality display system according to any one of Examples 1-3, wherein invisible light emitted by a light source comprises a beam forming spots on said portion of the object.

[0019] 5. An augmented reality display system according to any one of Examples 1-3, wherein the invisible light emitted by the light source comprises a light pattern.

[0020] 6. An augmented reality display system according to any one of Examples 1-5, wherein the characteristic includes the location of the feature.

[0021] 7. An augmented reality display system according to any one of Examples 1-6, wherein the characteristic includes the shape of the feature.

[0022] 8. An augmented reality display system according to any one of Examples 1-7, wherein the characteristic includes the orientation of the feature.

[0023] 9. The augmented reality display system according to any one of Examples 1-8 further includes a depth sensor configured to detect the position of a real object in the world.

[0024] 10. An augmented reality display system according to any one of Examples 1-9, wherein the processing circuitry is configured to determine the difference between the distribution of emitted invisible light and the distribution of the reflected portion of the invisible light.

[0025] 11. The augmented reality display system according to Example 10, wherein the processing circuitry is configured to identify a difference signature based on the determined differences.

[0026] 12. The augmented reality display system according to Example 11, wherein the processing circuitry is configured to determine the orientation of the real object based on the location of the real object and the location of the difference signature.

[0027] 13. The augmented reality display system according to Example 12, wherein the processing circuitry is configured to determine the location of a real object based at least in part on the location of a difference signature.

[0028] 14. The augmented reality display system according to any one of Examples 1-13 further includes an eyepiece disposed on a frame, at least a portion of the eyepiece being transparent and positioned in front of the wearer's eyes when the wearer wears the display system, such that the transparent portion transmits light from the environment in front of the wearer to the wearer's eyes to provide a view of the environment in front of the wearer.

[0029] 15. An augmented reality display system, comprising:

[0030] A frame that is configured to be mounted on the wearer;

[0031] Augmented reality display, which is attached to a frame and configured to direct images to the wearer's eyes;

[0032] A depth sensor, which is configured to map the surface of real objects within the wearer's field of view;

[0033] A light source configured to project light of at least a first wavelength onto the surface of a real object;

[0034] A light detector, configured to form an image using a portion of the light reflected from the surface of a real object; and

[0035] The processing circuit is configured to determine the optical difference mark based at least in part on the reflected portion of the optical pattern, and to present a virtual object at a fixed displacement relative to the optical difference mark.

[0036] 16. The augmented reality display system according to Example 15, wherein presenting a virtual object at a fixed displacement relative to a light difference marker includes:

[0037] Receive the initial position of the virtual object relative to the real object at the initial time;

[0038] The fixed displacement is determined based on the distance between the initial position and the optical difference marker;

[0039] Detect the subsequent position of the optical difference marker at a time after the initial time; and

[0040] A virtual object is presented at a fixed displacement relative to the detected subsequent position of the optical difference marker.

[0041] 17. The augmented reality display system according to Example 15 or 16 further includes an eyepiece disposed on a frame, at least a portion of which is transparent and is positioned in front of the wearer's eyes when the wearer wears the display system, such that the transparent portion transmits light from the environment in front of the wearer to the wearer's eyes to provide a view of the environment in front of the wearer.

[0042] 18. An augmented reality display system configured to align 3D content with real-world objects, the system comprising:

[0043] A frame that is configured to be mounted on the wearer;

[0044] Augmented reality display, which is attached to a frame and configured to direct images to the wearer's eyes;

[0045] A depth sensor, which is configured to map the surface of real objects within the wearer's field of view;

[0046] A light source configured to illuminate at least a portion of an object by emitting light;

[0047] A light sensor configured to image the portion of the object illuminated by the light source using the emitted light; and

[0048] The processing circuitry is configured to determine information about the location, orientation, or both of the real object based on one or more characteristics of features in the image of the object.

[0049] 19. The augmented reality display system according to Example 18, wherein the light emitted by the light source comprises a beam that forms spots on said portion of the object.

[0050] 20. The augmented reality display system according to Example 18, wherein the light emitted by the light source includes a light pattern.

[0051] 21. An augmented reality display system according to any one of Examples 18-20, wherein the characteristic includes the location of the feature.

[0052] 22. An augmented reality display system according to any one of Examples 18-21, wherein the characteristic includes the shape of the feature.

[0053] 23. An augmented reality display system according to any one of Examples 18-22, wherein the characteristic includes the orientation of the feature.

[0054] 24. An augmented reality display system according to any one of Examples 18-23, wherein the processing circuitry is configured to determine the difference between the distribution of emitted light and the distribution of emitted light reflected from the object.

[0055] 25. The augmented reality display system according to Example 24, wherein the processing circuitry is configured to identify optical difference markers based on the determined differences.

[0056] 26. The augmented reality display system according to Example 25, wherein the processing circuitry is configured to determine the orientation of the real object based on the position of the real object and the position of the optical difference marker.

[0057] 27. The augmented reality display system according to Example 25 or 26, wherein the processing circuitry is configured to determine the position of a real object based at least in part on the position of a light difference marker.

[0058] 28. An augmented reality display system according to any one of Examples 18-27, wherein the depth sensor comprises a laser or an ultrasonic rangefinder.

[0059] 29. The augmented reality display system according to any one of Examples 18-28, wherein the depth sensor includes a camera.

[0060] 30. The augmented reality display system according to any one of Examples 18-29 further includes an eyepiece disposed on a frame, at least a portion of the eyepiece being transparent and positioned in front of the wearer's eyes when the wearer wears the display system, such that the transparent portion transmits light from the environment in front of the wearer to the wearer's eyes to provide a view of the environment in front of the wearer.

[0061] 31. An augmented reality display system according to any of the examples above, wherein the display is configured to present image content as if different image content were positioned at different depths.

[0062] 32. The augmented reality display system according to Example 31, wherein the display includes a plurality of refractive optical elements having different refractive powers to provide for the different depths. Attached Figure Description

[0063] Figure 1 The illustration shows a user's view through an augmented reality (AR) device.

[0064] Figure 2 An example of a wearable display system is illustrated.

[0065] Figure 3 The illustration shows a conventional display system used to simulate 3D images for users.

[0066] Figure 4 The illustration shows aspects of a method for simulating 3D images using multiple depth planes.

[0067] Figures 5A-5C The diagram illustrates the relationship between the radius of curvature and the focal radius.

[0068] Figure 6 The illustration shows an example of waveguide stacking used to output image information to a user.

[0069] Figure 7 The illustration shows an example of an output beam from a waveguide.

[0070] Figure 8 The illustration shows an example of a stacked waveguide assembly in which each depth plane includes an image formed using multiple different component colors.

[0071] Figure 9A The illustration shows a cross-sectional side view of an example of a set of stacked waveguides, each including coupled optical elements.

[0072] Figure 9B The diagram shows... Figure 9A A perspective view of an example of multiple stacked waveguides.

[0073] Figure 9C The diagram shows... Figure 9A and Figure 9B A top-down planar view of an example of multiple stacked waveguides.

[0074] Figure 10 An augmented reality display system configured to use markers to track the orientation of objects in the world is illustrated schematically.

[0075] Figure 11 The illustration shows an example method for using markers to track the orientation of objects in the world. Detailed Implementation

[0076] Reference will now be made to the accompanying drawings, wherein similar reference numerals refer to similar parts throughout. It will be understood that the embodiments disclosed herein include optical systems and generally include display systems. In some embodiments, the display system is wearable, which can advantageously provide a more immersive VR or AR experience. For example, a display comprising one or more waveguides (e.g., a stack of waveguides) can be configured to be worn and positioned in front of the eyes of a user or observer. In some embodiments, two stacks of waveguides (one for each eye of the observer) can be used to provide a different image to each eye.

[0077] Example display system

[0078] Figure 2An example of a wearable display system 60 is illustrated. The display system 60 includes a display 70, and various mechanical and electronic modules and systems supporting the operation of the display 70. The display 70 may be coupled to a frame 80, which is wearable by a user or observer 90 of the display system, and the frame 80 is configured to position the display 70 in front of the user 90's eyes. In some embodiments, the display 70 may be considered as eyeglasses. In some embodiments, a speaker 100 is coupled to the frame 80 and configured to be positioned near the user 90's ear canal (in some embodiments, another speaker, not shown, is positioned near the user's other ear canal to provide stereo / shapeable sound control). In some embodiments, the display system may also include one or more microphones 110 or other devices for detecting sound. In some embodiments, the microphones are configured to allow the user to provide input or commands to the system 60 (e.g., voice menu commands, natural language questions, etc.), and / or may allow audio communication with other people (e.g., with other users of similar display systems). The microphones may also be configured as peripheral sensors to acquire audio data (e.g., sound from the user and / or environment). In some embodiments, the display system may further include a peripheral sensor 120a, which may be detachable from the frame 80 and attached to the body of the user 90 (e.g., on the user 90's head, torso, limbs, etc.). In some embodiments, the peripheral sensor 120a may be configured to collect data characterizing the physiological state of the user 90. For example, the sensor 120a may be an electrode.

[0079] Continue to refer to Figure 2The display 70 is operatively coupled to the local data processing module 140 via a communication link 130 (e.g., via a wired or wireless connection), which can be mounted in various configurations, such as being fixedly attached to the frame 80, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 90 (e.g., in a backpack configuration or a belt-coupled configuration). Similarly, the sensor 120a can be operatively coupled to the local processor and data module 140 via a communication link 120b (e.g., via a wired or wireless connection). The local processing and data module 140 may include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory or a hard disk drive), both of which can be used for auxiliary data processing, caching, and storage. The data includes: a) data acquired from sensors (which may be operatively coupled to frame 80 or otherwise attached to user 90, such as image acquisition devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, gyroscopes, and / or other sensors disclosed herein); and / or b) data acquired and / or processed using remote processing module 150 and / or remote data repository 160 (including data relating to virtual content), which may be used to transmit to display 70 after such processing or retrieval. Local processing and data module 140 may be operatively coupled to remote processing module 150 and remote data repository 160 via communication links 170, 180 (e.g., via wired or wireless communication links), such that these remote modules 150, 160 are operatively coupled to each other and can be used as resources of local processing and data module 140. In some embodiments, the local processing and data module 140 may include one or more of the following: an image acquisition device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be attached to the frame 80, or may be a separate structure that communicates with the local processing and data module 140 via a wired or wireless communication path.

[0080] Continue to refer to Figure 2In some embodiments, the remote processing module 150 may include one or more processors configured to analyze and process data and / or image information. In some embodiments, the remote data repository 160 may include a digital data storage facility, which may be available via an internet or other network configuration in a "cloud" resource configuration. In some embodiments, the remote data repository 160 may include one or more remote servers that provide information, such as information for generating augmented reality content, to the local processing and data module 140 and / or the remote processing module 150. In some embodiments, all data is stored and all computations are performed in the local processing and data module, allowing for fully autonomous use from the remote modules.

[0081] Perceiving an image as “three-dimensional” or “3-D” can be achieved by providing a slightly different presentation of the image to each of the observer’s eyes. Figure 3 The illustration depicts a conventional display system used to simulate 3D images for a user. Two distinct images 190 and 200—one for each eye 210 and 220—are output to the user. Images 190 and 200 are spaced 230 from the eyes 210 and 220 along an optical axis or z-axis parallel to the observer's line of sight. Images 190 and 200 are planar, and the eyes 210 and 220 can be focused on the images by assuming a single accommodation state. Such a 3D display system relies on the human visual system to combine images 190 and 200 to provide depth and / or scale perception for the combined image.

[0082] However, it will be understood that the human visual system is far more complex, and providing realistic depth perception is far more challenging. For example, many observers of conventional “3-D” display systems find such systems uncomfortable or may not perceive depth at all. Without being bound by theory, it is believed that an observer of an object can perceive it as “three-dimensional” due to a combination of convergence and accommodation. The convergence of the two eyes relative to each other (i.e., the rotation of the eyes, causing the pupils to move toward or away from each other to converge the line of sight to fix it on the object) is closely related to the focusing (or “accommodation”) of the eye’s lens and pupil. Under normal circumstances, changing the focus of the eye’s lens or adjusting the eye to shift the focus from one object to another at a different distance will automatically cause a matching change in convergence for the same distance, known as the “accommodation-convergence reflex” and the relationship of pupillary dilation or constriction. Similarly, under normal circumstances, a change in convergence will trigger a matching change in accommodation of the lens shape and pupil size. As this paper points out, many stereoscopic or “3-D” display systems use slightly different presentations (and therefore slightly different images) for each eye to display a scene, allowing the three-dimensional perspective to be perceived by the human visual system. However, such systems are uncomfortable for many observers because, among other things, they simply provide different presentations of the scene, but in which the eye observes all image information at a single accommodative state, which contradicts the “accommodation-verbose” principle. Display systems that provide a better match between accommodation and convergence can create more realistic and comfortable simulations of three-dimensional images, which contributes to increased durability and, consequently, compliance with diagnostic and treatment protocols.

[0083] Figure 4 The illustration depicts aspects of a method for simulating 3D imagery using multiple depth planes. (Reference) Figure 4Objects at various distances from eyes 210 and 220 along the z-axis are adjusted by eyes 210 and 220 to bring those objects into focus. Eyes 210 and 220 adopt specific adjustment states to focus objects at different distances along the z-axis. Thus, a specific adjustment state can be said to be associated with a particular one of the depth planes 240 and has an associated focal length such that when the eye is in an adjustment state for that depth plane, an object or part of an object in that depth plane is in focus. In some embodiments, the three-dimensional image can be simulated by providing different renderings of the image for each of eyes 210 and 220 and also by providing different renderings of the image corresponding to each of the depth planes. Although shown separately for clarity of illustration, it will be understood that the fields of view of eyes 210 and 220 can overlap, for example, as distance along the z-axis increases. Additionally, although shown flat for clarity of illustration, it will be understood that the contours of the depth planes can be curved in physical space such that all features in the depth plane are in focus with the eye in a specific adjustment state.

[0084] The distance between the object and the eye 210 or 220 can also change the amount of light emanating from the object as observed by the eye. Figures 5A-5C The diagram illustrates the relationship between distance and light divergence. The distances between the object and the eye 210 are represented by R1, R2, and R3 in decreasing order. (As shown in...) Figures 5A-5C As shown, the light rays become more divergent as the distance to the object decreases. As the distance increases, the light rays become more collimated. In other words, the light field generated by a point (the object or part of the object) can be said to have a spherical wavefront curvature, which is a function of how far that point is from the user's eye. The curvature increases as the distance between the object and the eye 210 decreases. Therefore, the divergence of light rays is also different at different depth planes, where the divergence increases as the distance between the depth plane and the observer's eye 210 decreases. Although in Figures 5A-5C For clarity, only a single eye 210 is illustrated in the other accompanying figures in this article, but it will be understood that the discussion of eye 210 can be applied to both of the observer's eyes 210 and 220.

[0085] Without being bound by theory, it is believed that the human eye can generally interpret a finite number of depth planes to provide depth perception. Therefore, a highly reliable simulation of depth perception can be achieved by providing the eye with different presentations of images corresponding to each of these finite number of depth planes. These different presentations can be individually focused by the observer's eye, thus contributing to providing the user with depth cues based on the eye's accommodation required to focus on different image features of a scene located on different depth planes and / or based on observing different image features on different defocused depth planes.

[0086] Figure 6 An example of a waveguide stack for outputting image information to a user is illustrated. The display system 250 includes a stack of waveguides or a stacked waveguide assembly 260, which can be used to provide three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310. In some embodiments, the display system 250 is... Figure 2 The system 60, and Figure 6 Some parts of the system 60 are shown schematically in more detail. For example, the waveguide assembly 260 may be... Figure 2 It is part of the display 70. It will be understood that in some embodiments, the display system 250 may be considered a light field display.

[0087] Continue to refer to Figure 6Waveguide assembly 260 may further include multiple features 320, 330, 340, 350 between waveguides. In some embodiments, features 320, 330, 340, 350 may be one or more lenses. Waveguides 270, 280, 290, 300, 310 and / or multiple lenses 320, 330, 340, 350 may be configured to send image information toward the eye at different levels of wavefront curvature or light divergence. Each waveguide level may be associated with a specific depth plane and may be configured to output image information corresponding to that depth plane. Image injection devices 360, 370, 380, 390, 400 may serve as light sources for the waveguides and may be used to inject image information into waveguides 270, 280, 290, 300, 310, as described herein, wherein each waveguide may be configured to distribute incident light across each respective waveguide for output toward the eye 210. Light exits the output surfaces 410, 420, 430, 440, and 450 of the image injection devices 360, 370, 380, 390, and 400 and is injected into the corresponding input surfaces 460, 470, 480, 490, and 500 of the waveguides 270, 280, 290, 300, and 310. In some embodiments, each of the input surfaces 460, 470, 480, 490, and 500 may be an edge of the corresponding waveguide or a portion of the main surface of the corresponding waveguide (i.e., one of the waveguide surfaces directly facing the world 510 or the observer's eye 210). In some embodiments, a single beam of light (e.g., a collimated beam) may be injected into each waveguide to output the entire field of a cloned collimated beam directed toward the eye 210 at a specific angle (and divergence) corresponding to the depth plane associated with the particular waveguide. In some embodiments, one of the image injection devices 360, 370, 380, 390, 400 may be associated with a plurality of (e.g., three) waveguides 270, 280, 290, 300, 310 and inject light into the plurality of (e.g., three) waveguides 270, 280, 290, 300, 310.

[0088] In some embodiments, image injection devices 360, 370, 380, 390, and 400 are discrete displays, each generating image information for injection into corresponding waveguides 270, 280, 290, 300, and 310, respectively. In some other embodiments, image injection devices 360, 370, 380, 390, and 400 are outputs of a single multiplexed display, the output of which may, for example, deliver image information to each of image injection devices 360, 370, 380, 390, and 400 via one or more optical conduits (such as fiber optic cables). It will be understood that the image information provided by image injection devices 360, 370, 380, 390, and 400 may include light of different wavelengths or colors (e.g., different component colors, as discussed herein).

[0089] In some embodiments, light injected into waveguides 270, 280, 290, 300, 310 is provided by an optical projector system 520, which includes an optical module 530 that may include a light emitter, such as a light-emitting diode (LED). Light from the optical module 530 may be directed via a beam splitter 550 to an optical modulator 540 (e.g., a spatial light modulator) and modified by the optical modulator 540. The optical modulator 540 may be configured to alter the perceived intensity of light injected into waveguides 270, 280, 290, 300, 310. Examples of spatial light modulators include liquid crystal displays (LCDs), including liquid crystal on silicon (LCOS) displays.

[0090] In some embodiments, the display system 250 may be a scanning fiber optic display, including one or more scanning fibers configured to project light in various patterns (e.g., raster scanning, spiral scanning, Lissajous patterns, etc.) into one or more waveguides 270, 280, 290, 300, 310 and ultimately to an observer's eye 310. In some embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may schematically represent a single scanning fiber or a bundle of scanning fibers configured to inject light into one or more waveguides 270, 280, 290, 300, 310. In some other embodiments, the illustrated image injection devices 360, 370, 380, 390, 400 may schematically represent multiple scanning fibers or multiple bundles of scanning fibers, each configured to inject light into an associated one of the waveguides 270, 280, 290, 300, 310. It will be understood that one or more optical fibers can be configured to transmit light from optical module 530 to one or more waveguides 270, 280, 290, 300, 310. It will be understood that one or more intermediate optical structures may be provided between the scanning fiber or the fiber and one or more waveguides 270, 280, 290, 300, 310 to, for example, redirect light leaving the scanning fiber to one or more waveguides 270, 280, 290, 300, 310.

[0091] Controller 560 controls the operation of one or more of the stacked waveguide assemblies 260, including the operation of image injection devices 360, 370, 380, 390, 400, light source 530, and optical modulator 540. In some embodiments, controller 560 is part of local data processing module 140. Controller 560 includes timing and provided programming (e.g., instructions in a non-transient medium) of image information modulated to waveguides 270, 280, 290, 300, 310 according to, for example, any of the various schemes disclosed herein. In some embodiments, the controller may be a single integrator or a distributed system connected by wired or wireless communication channels. In some embodiments, controller 560 may be processing module 140 or 150. Figure 2 Part of ).

[0092] Continue to refer to Figure 6Waveguides 270, 280, 290, 300, and 310 can be configured to propagate light within each respective waveguide via total internal reflection (TIR). Waveguides 270, 280, 290, 300, and 310 can each be planar or have another shape (e.g., curved), having a primary top surface and a primary bottom surface, and an edge extending between those primary top and bottom surfaces. In the illustrated configuration, waveguides 270, 280, 290, 300, and 310 can each include coupling optics 570, 580, 590, 600, and 610, which are configured to extract light from the waveguide by redirecting light propagating within each respective waveguide, thereby extracting light from the waveguide to output image information to eye 210. The extracted light can also be referred to as the coupled light, and the coupled optics light can also be referred to as the light extraction optics. The extracted light beam can be output by striking the light extraction optics at the location of the light propagating in the waveguide. The coupling optics 570, 580, 590, 600, and 610 can be, for example, gratings including diffractive optical features, as further discussed herein. Although illustrated on the bottom main surface of waveguides 270, 280, 290, 300, and 310 for ease of description and clarity of the figures, in some embodiments, the coupling optics 570, 580, 590, 600, and 610 can be disposed on the top and / or bottom main surfaces, and / or can be disposed directly within the volume of waveguides 270, 280, 290, 300, and 310, as further discussed herein. In some embodiments, the coupling optics 570, 580, 590, 600, and 610 can be formed in a material layer attached to a transparent substrate to form waveguides 270, 280, 290, 300, and 310. In some other embodiments, waveguides 270, 280, 290, 300, and 310 may be monolithic materials, and coupled optical elements 570, 580, 590, 600, and 610 may be formed on the surface and / or inside the monolithic material.

[0093] Continue to refer to Figure 6As discussed herein, each waveguide 270, 280, 290, 300, 310 is configured to output light to form an image corresponding to a specific depth plane. For example, the waveguide 270 closest to the eye can be configured to deliver collimated light (injected into such a waveguide 270) to the eye 210. The collimated light may represent the optical infinity focal plane. The next up waveguide 280 can be configured to emit collimated light that passes through a first lens 350 (e.g., a negative lens) before reaching the eye 210; such a first lens 350 can be configured to produce a slight concave wavefront curvature, such that the eye / brain interprets the light from the next up waveguide 280 as coming from a first focal plane that is closer inward from optical infinity toward the eye 210. Similarly, the third upward waveguide 290 causes its output light to pass through the first lens 350 and the second lens 340 before reaching the eye 210; the combined refractive power of the first lens 350 and the second lens 340 can be configured to produce another increase in wavefront curvature, such that the eye / brain interprets the light from the third waveguide 290 as a second focal plane that is closer to the person than the light from the first upward waveguide 280, which is from optical infinity toward the person.

[0094] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, wherein the highest waveguide 310 in the stack sends its output through all the lenses between it and the eye, representing the total optical power to the nearest focal plane of the person. To compensate for the stacking of lenses 320, 330, 340, 350 when viewing / interpreting light from the world 510 on the other side of the stacked waveguide assembly 260, a compensation lens layer 620 can be disposed on top of the stack to compensate for the total optical power of the lens stack 320, 330, 340, 350 below. Such a configuration provides as many focal planes as are available waveguide / lens pairs. Both the decoupled optics of the waveguides and the focusing aspects of the lenses can be static (i.e., non-dynamic or electrically active). In some alternative embodiments, one or both may be dynamically dynamic using electrically active features.

[0095] In some embodiments, two or more of waveguides 270, 280, 290, 300, and 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, and 310 may be configured to output image sets to the same depth plane, or multiple subsets of waveguides 270, 280, 290, 300, and 310 may be configured to output image sets to the same multiple depth planes, wherein there is one set for each depth plane. This can provide the advantage of forming stitched images to provide an extended field of view at those depth planes.

[0096] Continue to refer to Figure 6The coupling optical elements 570, 580, 590, 600, and 610 can be configured to redirect light outside their respective waveguides and output the light with an appropriate amount of divergence or collimation for a specific depth plane associated with the waveguide. Thus, waveguides with different associated depth planes can have different configurations of the coupling optical elements 570, 580, 590, 600, and 610, which, depending on the associated depth plane, output light with different amounts of divergence. In some embodiments, the light extraction optical elements 570, 580, 590, 600, and 610 can be volumetric or surface features that can be configured to output light at a specific angle. For example, the light extraction optical elements 570, 580, 590, 600, and 610 can be volumetric holograms, surface holograms, and / or diffraction gratings. In some embodiments, features 320, 330, 340, and 350 may not be lenses; instead, they may simply be spacers (e.g., cladding and / or structures for forming gaps).

[0097] In some embodiments, the coupling optical elements 570, 580, 590, 600, and 610 are diffraction features that form a diffraction pattern, or "diffraction optical elements" (also referred to herein as "DOEs"). Preferably, the DOE has a sufficiently low diffraction efficiency such that only a portion of the beam is deflected toward the eye 210 at each intersection of the DOE, while the remainder continues to move through the waveguide via TIR. The light carrying image information is thus split into many associated outgoing beams exiting the waveguide at many locations, and the result is a fairly uniform pattern of outgoing emission toward the eye 210 for that particular collimated beam bouncing throughout the waveguide.

[0098] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may include a polymer-dispersed liquid crystal layer, wherein the droplets comprise a diffraction pattern in a host medium, and the refractive index of the droplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light) or the droplets may be switched to a refractive index that does not match the refractive index of the host medium (in which case the pattern actively diffracts incident light).

[0099] In some embodiments, camera assembly 630 (e.g., a digital camera, including visible light and infrared light cameras) may be provided to acquire images of eye 210 and / or tissues surrounding eye 210, for example, to detect user input and / or monitor the user's physiological state. As used herein, the camera can be any image acquisition device. In some embodiments, camera assembly 630 may include an image acquisition device and a light source that projects light (e.g., infrared light) onto the eye, which can then be reflected by the eye and detected by the image acquisition device. In some embodiments, camera assembly 630 may be attached to frame 80 (… Figure 2 It can also be electrically communicated with processing modules 140 and / or 150, which can process image information from camera assembly 630 to make various determinations regarding, for example, the user's physiological state, as discussed herein. It will be understood that information regarding the user's physiological state can be used to determine the user's behavioral or emotional state. Examples of such information include the user's movements and / or facial expressions. The user's behavioral or emotional state can then be triangulated using acquired environmental and / or virtual content data to determine the relationship between the behavioral or emotional state, the physiological state, and the environmental or virtual content data. In some embodiments, a separate camera assembly 630 can be used for each eye to monitor each eye individually.

[0100] Now for reference Figure 7 An example of an output beam from a waveguide is shown. A single waveguide is illustrated, but it will be understood that in the case where waveguide assembly 260 comprises multiple waveguides, waveguide assembly 260 ( Figure 6 Other waveguides in the waveguide can operate similarly. Light 640 is injected into waveguide 270 at input surface 460 and propagates within waveguide 270 via TIR. At the point where light 640 is incident on DOE 570, a portion of the light exits the waveguide as an outgoing beam 650. Outgoing beam 650 is illustrated as substantially parallel, but as discussed herein, it can also be redirected to propagate at an angle to eye 210 (e.g., forming a diverging outgoing beam), depending on the depth plane associated with waveguide 270. It will be understood that a substantially parallel outgoing beam can indicate a waveguide with a coupling optics element that couples the light to form an image that appears to be set on a depth plane at a large distance (e.g., optical infinity) from eye 210. Other waveguides or other sets of coupled optical elements can output a more divergent beam pattern that would require the eye 210 to adjust to a closer distance to focus on the retina and be interpreted by the brain as light coming from a distance closer to the eye 210 than optical infinity.

[0101] In some embodiments, a panchromatic image can be formed at each depth plane by an image of each of the overlapping component colors (e.g., three or more component colors). Figure 8 An example of a stacked waveguide assembly, where each depth plane includes an image formed using multiple different component colors, is illustrated. The illustrated embodiment shows depth planes 240a–240f, but more or fewer depths are also contemplated. Each depth plane may have three or more component color images associated with it, including: a first image of a first color G; a second image of a second color R; and a third image of a third color B. Different depth planes are indicated in the figures by different numbers following the letters G, R, and B for diopter (dpt). By way of example only, the numbers following each of these letters indicate diopter (1 / m), or the inverse distance of the depth plane from the observer, and each box in the figures represents a separate component color image. In some embodiments, the exact positioning of the depth planes for different color components can vary to account for differences in eye focusing on different wavelengths of light. For example, different component color images for a given depth plane may be placed on depth planes corresponding to different distances from the user. Such an arrangement can increase visual sensitivity and user comfort and / or can reduce chromatic aberration.

[0102] In some embodiments, light of each component color can be output by a single dedicated waveguide, and therefore, each depth plane can have multiple waveguides associated with it. In such embodiments, each box in the figure including the letters G, R, or B can be understood to represent a single waveguide, and each depth plane can provide three waveguides, wherein each depth plane provides three color component images. Although the waveguides associated with each depth plane are shown as adjacent to each other in the figures, it will be understood that in a physical device, the waveguides can all be arranged in a stack with one waveguide per layer. In some other embodiments, multiple component colors can be output by the same waveguide, such that, for example, each depth plane can provide only a single waveguide.

[0103] Continue to refer to Figure 8 In some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light (including magenta and cyan) may be used additionally or may replace one or more of red, green, or blue. In some embodiments, features 320, 330, 340, and 350 may be active or passive optical filters configured to block or selectively block light from the surrounding environment to the observer's eye.

[0104] It will be understood that references to a given color of light throughout this disclosure will be interpreted as encompassing light of one or more wavelengths within a range of wavelengths perceived by an observer to have that given color. For example, red light may include light of one or more wavelengths in the range of approximately 620–780 nm, green light may include light of one or more wavelengths in the range of approximately 492–577 nm, and blue light may include light of one or more wavelengths in the range of approximately 435–493 nm.

[0105] In some embodiments, the light source 530 ( Figure 6 The display 250 can be configured to emit light at one or more wavelengths outside the observer's visual perception range, such as infrared and / or ultraviolet wavelengths. Additionally, the waveguide's coupling-in, coupling-out, and other light redirection structures of the display 250 can be configured to direct the light toward the user's eye 210 and emit it outside the display, for example, for imaging and / or user stimulation applications.

[0106] Now for reference Figure 9A In some embodiments, light incident on a waveguide may need to be redirected to couple the light into the waveguide. Coupling optics can be used to redirect the light and couple it into its corresponding waveguide. Figure 9A The illustration shows a cross-sectional side view of an example of multiple stacked waveguides or stacked waveguide sets 660, each including coupled optical elements. Each waveguide can be configured to output light of one or more different wavelengths or one or more different wavelength ranges. It will be understood that stack 660 can correspond to stack 260 (…). Figure 6 Furthermore, the illustrated waveguide of stack 660 may correspond to a portion of multiple waveguides 270, 280, 290, 300, 310, with the exception that light from one or more of the image injection devices 360, 370, 380, 390, 400 is injected into the waveguide from a location where light redirection is required for coupling.

[0107] The illustrated stacked waveguide assembly 660 includes waveguides 670, 680, and 690. Each waveguide includes an associated coupling optical element (which may also be referred to as a light input region on the waveguide), wherein, for example, a coupling optical element 700 is disposed on the main surface (e.g., the upper main surface) of waveguide 670, a coupling optical element 710 is disposed on the main surface (e.g., the upper main surface) of waveguide 680, and a coupling optical element 720 is disposed on the main surface (e.g., the upper main surface) of waveguide 690. In some embodiments, one or more of the coupling optical elements 700, 710, and 720 may be disposed on the bottom main surface of the respective waveguide 670, 680, and 690 (particularly wherein one or more coupling optical elements are reflection deflection optical elements). As illustrated, coupling optical elements 700, 710, and 720 may be disposed on the upper main surface of their respective waveguides 670, 680, and 690 (or on top of the next lower waveguide), particularly wherein those coupling optical elements are transmission deflection optical elements. In some embodiments, coupling optical elements 700, 710, and 720 may be disposed within the body of the respective waveguides 670, 680, and 690. In some embodiments, as discussed herein, coupling optical elements 700, 710, and 720 are wavelength selective, such that they selectively redirect one or more wavelengths of light while transmitting other wavelengths of light. Although illustrated on one side or corner of their respective waveguides 670, 680, and 690, it will be understood that in some embodiments, coupling optical elements 700, 710, and 720 may be disposed in other regions of their respective waveguides 670, 680, and 690.

[0108] As illustrated, the coupling optical elements 700, 710, and 720 can be laterally offset from each other. In some embodiments, each coupling optical element can be offset such that it receives light without the light passing through another coupling optical element. For example, each coupling optical element 700, 710, and 720 can be configured to receive light from, for example,... Figure 6 The different image injection devices 360, 370, 380, 390 and 400 shown receive light and can be separated from other coupled optical elements 700, 710, 720 (e.g., laterally spaced) such that they substantially do not receive light from other coupled optical elements among the coupled optical elements 700, 710, 720.

[0109] Each waveguide also includes associated light distribution elements, such as light distribution element 730 disposed on the main surface (e.g., top main surface) of waveguide 670, light distribution element 740 disposed on the main surface (e.g., top main surface) of waveguide 680, and light distribution element 750 disposed on the main surface (e.g., top main surface) of waveguide 690. In some other embodiments, light distribution elements 730, 740, and 750 may be disposed on the bottom main surface of associated waveguides 670, 680, and 690, respectively. In some other embodiments, light distribution elements 730, 740, and 750 may be disposed on the top and bottom main surfaces of associated waveguides 670, 680, and 690, respectively; or light distribution elements 730, 740, and 750 may be disposed on different main surfaces of the top and bottom main surfaces of different associated waveguides 670, 680, and 690.

[0110] Waveguides 670, 680, and 690 can be separated and isolated by layers of, for example, gaseous, liquid, and / or solid materials. For example, as illustrated, layer 760a can separate waveguides 670 and 680; and layer 760b can separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed of a low-refractive-index material (i.e., a material having a lower refractive index than the material forming one of the directly adjacent waveguides of waveguides 670, 680, and 690). Preferably, the refractive index of the material forming layers 760a and 760b is 0.05 or more, or 0.10 or less, less than the refractive index of the material forming waveguides 670, 680, and 690. Advantageously, the lower-refractive-index layers 760a and 760b can be used as cladding layers that facilitate the TIR (e.g., the TIR between the top and bottom principal surfaces of each waveguide) of light passing through waveguides 670, 680, and 690. In some embodiments, layers 760a and 760b are formed of air. Although not illustrated, it will be understood that the top and bottom of the illustrated waveguide assembly 660 may include directly adjacent cladding layers.

[0111] Preferably, for ease of manufacturing and other considerations, the materials forming waveguides 670, 680, and 690 are similar or identical, and the materials forming layers 760a and 760b are similar or identical. In some embodiments, the materials forming waveguides 670, 680, and 690 may be different between one or more waveguides, and / or the materials forming layers 760a and 760b may be different, while still maintaining the various refractive index relationships indicated above.

[0112] Continue to refer to Figure 9ALight rays 770, 780, and 790 are incident on waveguide assembly 660. It will be understood that light rays 770, 780, and 790 can be injected into waveguides 670, 680, and 690 via one or more image injection devices 360, 370, 380, 390, and 400. Figure 6 ).

[0113] In some embodiments, the light rays 770, 780, and 790 have different properties, such as different wavelengths or different wavelength ranges, which may correspond to different colors. The light rays 770, 780, and 790 can also be laterally shifted to different positions corresponding to the lateral positions of the coupled optical elements 700, 710, and 720. The coupled optical elements 700, 710, and 720 each deflect the incident light, causing the light to propagate via TIR through a corresponding waveguide 670, 680, and 690.

[0114] For example, the coupling optical element 700 can be configured to deflect light 770 having a first wavelength or wavelength range. Similarly, transmitted light 780 is incident on and deflected by the coupling optical element 710, which is configured to deflect light of a second wavelength or wavelength range. Likewise, light 790 is deflected by the coupling optical element 720, which is configured to selectively deflect light of a third wavelength or wavelength range.

[0115] Continue to refer to Figure 9A The deflected rays 770, 780, and 790 are deflected so that they propagate through the corresponding waveguides 670, 680, and 690; that is, the coupling optical elements 700, 710, and 720 of each waveguide deflect the light into that corresponding waveguide 670, 680, and 690 to couple the light into that corresponding waveguide. The rays 770, 780, and 790 are deflected at an angle so that the light propagates through the corresponding waveguides 670, 680, and 690 via TIR, and are thus guided therein. For example, the deflection of rays 770, 780, and 790 can be caused by one or more reflective, diffractive, and / or holographic optical elements, such as holographic, diffractive, and / or reflective tuning features, reflectors, or mirrors. In some cases, the deflection can be caused by microstructures (such as diffractive features in one or more gratings) and / or holographic and / or diffractive optical elements configured to tune or redirect the light, for example, to be guided by light guides. Light rays 770, 780, and 790 propagate via TIR through corresponding waveguides 670, 680, and 690, where they are guided until they are incident on the corresponding light distribution elements 730, 740, and 750 of the waveguides.

[0116] Now for reference Figure 9B The illustration shows Figure 9AA perspective view of an example of multiple stacked waveguides. As described above, coupled rays 770, 780, and 790 are deflected by coupled optical elements 700, 710, and 720, respectively, and then propagate via TIR and are guided within waveguides 670, 680, and 690, respectively. Guided rays 770, 780, and 790 are then incident on light distribution elements 730, 740, and 750, respectively. Light distribution elements 730, 740, and 750 may include one or more reflective, diffractive, and / or holographic optical elements, such as holographic, diffractive, and / or reflective tuning features, reflectors, or mirrors. In some cases, deflection may be caused by microstructures (such as diffractive features in one or more gratings) and / or configured to tune or redirect light, for example, to utilize holographic and / or diffractive optical elements guided by light guides. Light rays 770, 780, and 790 propagate via TIR through corresponding waveguides 670, 680, and 690, where they are guided until they are incident on corresponding optical distribution elements 730, 740, and 750 of the waveguides. However, they are deflected in such a way that light rays 770, 780, and 790 are still guided within the waveguides. Optical distribution elements 730, 740, and 750 deflect light rays 770, 780, and 790 so that they propagate toward coupling optical elements 800, 810, and 820, respectively.

[0117] The coupling optical elements 800, 810, and 820 are configured to direct light guided within the waveguide (e.g., rays 770, 780, and 790) out of the waveguide and toward the observer's eye. The coupling optical elements 800, 810, and 820 can therefore be configured to deflect and redirect the light guided within the waveguide, such as rays 770, 780, and 790, at a more normal angle relative to the surface of the waveguide to reduce the effects of total internal reflection (TIR), so that the light is not guided within the waveguide but instead exits from it. Furthermore, these coupling optical elements 800, 810, and 820 can be configured to deflect and redirect the light, such as rays 770, 780, and 790, toward the observer's eye. Therefore, the coupling optical elements 800, 810, and 820 may include one or more reflective, diffractive, and / or holographic optical elements, such as holographic, diffractive, and / or reflective tuning features, reflectors, or mirrors. In some cases, deflection can be caused by microstructures (such as diffraction features in one or more gratings) and / or by being configured to tune or redirect light, for example, to utilize holographic and / or diffractive optical elements guided by a light guide. Optical elements 800, 810, 820 can be configured to reflect, deflect, and / or diffract light rays 770, 780, 790 such that they propagate toward the user's eye outside the waveguide.

[0118] In some embodiments, light distribution elements 730, 740, and 750 are orthogonal pupil dilators (OPEs). In some embodiments, the OPE deflects or distributes light to output optics 800, 810, and 820 and also replicates (one or more) the beam to form a larger number of beams propagating to the output optics. As the beams propagate along the OPE, a portion of the beam may be split from the beam and propagate in a direction orthogonal to the beam (in the direction of the output optics 800, 810, and 820). Orthogonal splitting of the beams in the OPE can occur repeatedly along the path of the beams through the OPE. For example, the OPE may include an increased reflectivity along the beam path, such that a series of substantially uniform sub-beams are generated from a single beam. In some embodiments, output optics 800, 810, and 820 are exit pupils (EPs) or exit pupil dilators (EPEs) that direct light into the observer's eye 210. Figure 7 The OPE can be configured to increase the size of the eye box (e.g., along the x-direction), and the EPE can increase the size of the eye box in an axis that spans (e.g., orthogonal to) the OPE (e.g., along the y-direction).

[0119] Therefore, refer to Figure 9A and 9BIn some embodiments, waveguide assembly 660 includes waveguides 670, 680, 690; coupling optics 700, 710, 720; light distribution elements (e.g., OPE) 730, 740, 750; and coupling out optics (e.g., EPE) 800, 810, 820 for each component color. Waveguides 670, 680, 690 may be stacked by gaps and / or cladding between each. Coupling optics 700, 710, 720 redirect or deflect incident light (where different coupling optics receive light of different wavelengths) into their respective waveguides. The light then propagates at an angle that will result in a TIR within the respective waveguide 670, 680, 690, and is guided therein. In the example shown, ray 770 (e.g., blue light) is deflected by the first coupled-in optics 700 and then continues to propagate within the waveguide, where it is guided and interacts with the light distribution element (e.g., OPE) 730, where it is replicated in the manner previously described as multiple rays propagating to the coupled-out optics (e.g., EPE) 800. Rays 780 and 790 (e.g., green and red light, respectively) will pass through waveguide 670, where ray 780 is incident on and deflected by coupled-in optics 710. Ray 780 then bounces down along waveguide 680 via TIR and continues to its light distribution element (e.g., OPE) 740, where it is replicated in multiple rays propagating to the coupled-out optics (e.g., EPE) 810. Finally, ray 790 (e.g., red light) passes through waveguide 690 to be incident in the light coupled-in optics 720 of waveguide 690. The light-coupled optics 720 deflects the light ray 790, causing it to propagate via TIR to a light distribution element (e.g., OPE), where it is replicated into multiple light rays that propagate via TIR to an output optics 820 (e.g., EPE). The output optics 820 then further replicates the light ray 790 and couples it out to an observer, who also receives the coupled light from other waveguides 670, 680.

[0120] Figure 9C The diagram shows... Figure 9A and Figure 9BA top-down plan view (front view) of an example of multiple stacked waveguides. As illustrated, waveguides 670, 680, 690, along with associated light distribution elements 730, 740, 750 and associated coupling optics 800, 810, 820 for each waveguide, can be vertically aligned (e.g., along the x and y directions). However, as discussed herein, the coupling optics 700, 710, 720 are not vertically aligned; instead, the coupling optics are preferably non-overlapping (e.g., laterally spaced along the x direction, as seen in the top-down view of the front view in this example). Offsets in other directions, such as the y direction, can also be employed. This non-overlapping spatial arrangement facilitates the injection of light from different resources (such as different light sources and / or displays) into different waveguides on a one-to-one basis, thereby allowing a specific light source to be uniquely coupled to a specific waveguide. In some embodiments, an arrangement including non-overlapping, laterally separated coupling optics can be referred to as an offset pupil system, and the coupling optics within these arrangements can correspond to sub-pupils.

[0121] In addition to coupling light out of the waveguide, the coupling optics 800, 810, 820 can collimate or diverge light as if it originated from an object at a distance or closer, at depth or a depth plane. For example, collimated light is aligned with light from an object far from the viewer. Increased divergence aligns light from an object closer, for example, 5-10 feet or 1-3 feet in front of the observer. The eye's natural lens will adjust when viewing an object closer to the eye, and the brain can sense this adjustment, which then serves as a depth cue. Similarly, by diverging light by a certain amount, the eye will adjust and perceive the object as being closer. Therefore, the coupling optics 800, 810, 820 can be configured to collimate or diverge light as if it were emitted from a distance or closer, at depth or a depth plane. In doing so, the coupling optics 800, 810, 820 can include refractive power. For example, the coupled optical elements 800, 810, and 820 may include holographic, diffractive, and / or reflective optical elements, such that, in addition to deflecting or redirecting light out of the waveguide, these holographic, diffractive, and / or reflective optical elements may also include refractive power to collimate or diverge light. The coupled optical elements 800, 810, and 820 may optionally include a refractive surface that includes refractive power to collimate or diverge light. Therefore, for example, in addition to diffractive or holographic tuning features, the coupled optical elements 800, 810, and 820 may include a refractive surface that provides refractive power. In addition to the coupled optical elements 800, 810, and 820, such a refractive surface may also be included, for example, on top of the coupled optical elements 800, 810, and 820. In some embodiments, for example, optical elements (e.g., diffractive optical elements, holographic optical elements, refractive lens surfaces, or other structures) may be disposed relative to the coupling optical elements 800, 810, 820 to provide refractive power that causes collimation or divergence of light. Layers with refractive power (such as layers with refractive surfaces or layers with diffractive and / or holographic features) may be disposed relative to the coupling optical elements 800, 810, 820, for example, to additionally provide refractive power. Combinations of contributions from both the refractive optical elements 800, 810, 820 and additional layers with refractive power (such as layers with refractive surfaces or layers with diffractive and / or holographic features) are also possible.

[0122] Example of a 3D content rendering system and method

[0123] In various implementations, the augmented reality systems and methods described herein can be used to render virtual content, such as virtual objects, that appears to interact with real objects in the world surrounding the wearer. In some embodiments, depth sensors can be used to map the world around the wearer and / or parts of the wearer's body, and the augmented reality system can render 3D virtual content, such as objects or graphics, onto real objects detected in the world. In one example, the virtual content could be a virtual watch rendered on the wearer's wrist. Thus, although the wearer of the head-mounted augmented reality device is not wearing a real watch, the virtual watch displayed to the wearer via the device's display may appear to be on the wearer's wrist. In another example, the virtual content could be a graphic design for display on a real object, such as a logo or advertising content to be displayed on the rim of a coffee cup.

[0124] The position of real-world objects associated with virtual content can be tracked, for example, using depth sensors. Virtual content can then be displayed in a location determined based on the position of the real-world object. For instance, when a wearer moves their wrist associated with a virtual watch, the device can change the position of the virtual watch displayed to the wearer so that it continues to appear on the wearer's wrist. However, existing depth sensors may not be able to detect the orientation of real-world objects associated with virtual content. For example, if the wearer's wrist or the coffee cup described above rotates, the information from the depth sensor may be insufficient to allow the system to distinguish between different symmetrical or nearly symmetrical orientations of the real-world object. Therefore, while depth sensors can detect the position of real-world objects, the system described herein can use secondary tracking of features of the real-world object (sometimes referred to herein as landmarks or reference features) to identify more precise positional and / or orientation information that depth sensors cannot detect.

[0125] The various systems and methods described herein allow augmented reality display systems to track the position and orientation of an object based on the light reflection and / or scattering properties of visible or invisible markings on or near the object's surface. In some example embodiments, the augmented reality display system may use a depth sensor to track the object's position, identify features on the object's surface, and determine the object's orientation based on the position of the features relative to the object. The features (sometimes referred to herein as reference features or benchmarks) may be pre-existing features of the object that allow the object's orientation to be tracked without applying additional markings for orientation tracking purposes. As will be described in more detail, features or benchmarks may be detected using visible or invisible light, such as in the infrared or ultraviolet range. Benchmarks or features may be background features (such as a birthmark on the wrist or a seam in a coffee cup) or invisible features (such as one or more veins on the wrist, arm, or hand).

[0126] Now will be on Figure 10For reference, Figure 10 A schematic diagram of various components of an augmented reality display system 2010 configured to track the position and orientation of real-world objects as described herein is shown. In some embodiments, the augmented reality display system may be a mixed reality display system. As shown, the augmented reality display system 2010 includes a frame 64 that at least partially encloses left and right waveguide stacks 2005, 2006 configured to deliver augmented reality content to the left and right eyes 2001, 2002 of a wearer of the augmented reality display system. The system also includes a depth sensor 28, a light source 26, and a light detector 24. The tracking system 22 may include a processing module 70 that can control and / or analyze data received from the depth sensor 28, the light detector 24, and / or the light source 26. The depth sensor 28, the light detector 24, and / or the light source 26 may communicate with the processing module 70 via data links 76, 78.

[0127] Depth sensor 28 can be configured to detect the shape and position of various objects in the world surrounding the wearer. For example, objects detected by depth sensor 28 may include walls, furniture, and other items in a room surrounding the wearer; other people or animals near the wearer; outdoor objects such as trees, bushes, cars, buildings, etc.; and / or parts of the wearer's body such as arms, hands, legs, and feet. In various embodiments, the depth sensor may be effective at mapped objects at distances between 0.5 meters and 4 meters from the wearer, 1 meter to 3 meters, up to 5 meters, or any other range. The depth sensor may be an optical depth sensor configured to determine depth using infrared light, visible light, etc. In various embodiments, the depth sensor may include one or more of the following: a laser source, a laser rangefinder, a camera, an ultrasonic rangefinder, or other distance sensing, imaging, and / or mapping devices.

[0128] The light detector 24 can be configured to detect one or more of infrared light, visible light, ultraviolet light, or other ranges of electromagnetic radiation. Similarly, the light source 26 can be configured to emit one or more of infrared light, visible light, ultraviolet light, or other ranges of electromagnetic radiation. In some embodiments, at least a portion of the spectrum emitted by the light source 26 will be detectable by the light detector 24. In some designs, the light source 26 can be mounted on a gimbal or other movable mount such that the direction of emitted radiation can be controlled independently of the orientation of the augmented reality display device 2010. The light detector 24 can be an imaging device (such as a camera) configured to acquire an image of light in at least a portion of the wearer's field of view. In various embodiments, each light detector 24 can be a camera and can include a two-dimensional array of light sensors. In some example embodiments, the light source 26 is configured to emit infrared light within a specified wavelength range, and the light detector 24 includes an infrared sensor or infrared light detector configured to acquire an infrared image using infrared light reflected from objects within the field of view of the light detector 24.

[0129] In some cases, features that are not prominent in visible light, such as infrared or ultraviolet light, can become distinguishable when illuminated with invisible light. For example, veins that may not be resolvable to the eye can be clearly resolved by an infrared camera under infrared illumination. Such veins can be used as a reference to identify and track the movement, translation, and / or changes in orientation of an object. Thus, illuminating an object with light, such as invisible light, can otherwise make invisible features detectable by a camera or imaging sensor. The movement of these features, which may be referred to as a reference (or a differential signature as described more fully below), can allow the movement of the object to be tracked, for example, enabling the proper positioning of virtual content relative to a moving object. While veins are used as an example, other features can be observable using illumination, such as infrared illumination using an infrared detector. For example, other features of skin can reflect or absorb IR (or UV) light to produce features, markers, or references that can be tracked using a camera sensitive to one or more suitable wavelengths (e.g., an IR camera) to track the movement of the object, including changes in rotation or orientation. Using known changes in the object's movement and orientation, virtual content can be accurately positioned and oriented. Virtual content designed to follow an object or have a fixed position and / or orientation relative to an object can be appropriately presented. Similarly, an appropriate viewpoint can be provided for the virtual content. While infrared or ultraviolet light has been discussed in the examples above, other wavelengths of visible and invisible light or electromagnetic radiation can be used.

[0130] In some cases, illumination may include patterns, such as grids or arrays. Furthermore, an image of a pattern projected onto a surface can be compared by processing module 70 with the emission pattern of light to determine the difference between emitted and reflected light. Similarly, processing module 70 can be configured to identify locally unique differences within an image. Locally unique differences can be caused by a portion of a real object having a reflectivity different from its surrounding area. For example, a birthmark, mole, vein, scar tissue, or other structure on an arm or hand may have a reflectivity different from that of surrounding tissue in the imaging wavelength range (e.g., infrared or UV). Therefore, if a region of scar tissue is present in the area of ​​illumination and imaging, the light difference between the emitted and reflected radiation distributions can include an anomalous region shaped like the scar tissue region. Such a feature, referred to herein as a difference label, can be used to track objects, allowing virtual content to be appropriately positioned and oriented relative to the object.

[0131] Joint Reference Figure 10 and 11 An example method 1100 for tracking the position and / or orientation of a 3D object using detected features will now be described. Method 1100 can be implemented by any of the systems described herein, such as in... Figure 2 and Figure 10 The wearable augmented reality display system 60, 2010 is depicted in the diagram. Method 1100 begins at box 1110, where virtual content is received. In one example, the virtual content may be a 3D representation of a watch, which can be used as a virtual watch visible to the wearer of the augmented reality display system 2010. The system 2010 also receives the position of the virtual content relative to a detected real object. For example, the system 2010 may automatically or based on instructions made by the wearer using depth sensor 28 to detect the wearer's wrist. The position for the virtual watch may be determined automatically or by instructions from the wearer, such as through gestures or using any suitable input device. For example, the wearer may select the position of the virtual watch so that the virtual watch appears to be set around the wearer's wrist as a real-world watch. In another example, the virtual content may be a virtual name tag or item identifier, advertising graphics to be displayed on a real object (such as a coffee cup), or any other type of virtual content intended to appear as if attached to a real object. Once the desired position of the virtual content relative to the real object is determined, the relative position can be selected, registered, or otherwise completed. After receiving the virtual content and its position relative to the real object, method 1100 continues to box 1120.

[0132] At frame 1120, system 2010 emits a radiation pattern and determines differences between images projected onto the object. This difference may depend on and indicate structural features of the real object. The difference may, for example, indicate changes in absorption, reflectivity, and / or scattering attributable to structural variations in the object. This difference can be used as a benchmark, difference label, or marker to track changes in motion and / or orientation. The radiation pattern may be emitted by light source 26, a light pattern such as a textured light field, a grid, or a series of dots, intersections, circles (e.g., concentric circles or contours), or other patterns. While patterns such as grids have been discussed above, illumination may also include substantially uniform illumination or spots of any shape (e.g., circles, squares, etc.), and difference labels or markers may still be obtained. The emitted radiation pattern may include visible or invisible light, such as infrared, ultraviolet, or any other suitable wavelength or wavelength range. In some embodiments, an invisible wavelength range such as infrared or ultraviolet may be desirable to avoid distracting the wearer or others by projecting a visible light pattern or visible light. The orientation of the radiation pattern may be selected such that at least a portion of the radiation pattern is incident on the surface of the real object. For example, in some configurations, the light source is movable, such as via a gimbal or other rotating platform and / or a tiltable platform. In some embodiments, radiation can be projected onto a location directly adjacent to the receiving location of the virtual object. In an example implementation of a virtual watch, radiation can be projected onto the back of the wearer's hand or onto the wearer's forearm adjacent to the location of the virtual watch. In other embodiments, radiation can be projected onto a location spaced away from the location of the virtual object.

[0133] After the radiated pattern is emitted, a portion of the emitted radiation can be reflected back to the augmented reality display system 2010 by a real object. Some of the emitted radiation can be reflected by external surfaces or internal structures beneath the surface of an object. For example, if the radiated pattern is an infrared light pattern guided at the user's arm or hand, a portion of the infrared light can be reflected by external skin surfaces, internal areas of the skin and / or veins, or other structures beneath the skin. The reflected portion of the radiated pattern can be detected at the light detector 24. For example, the light source 26 can be an infrared source emitting the infrared radiated pattern, and the light detector 24 can be an infrared camera configured to acquire an image in the infrared spectrum. Therefore, when a portion of the emitted radiated pattern is reflected back to the display system 2010, the light detector 24 can acquire an image of the reflected light pattern.

[0134] To determine the optical difference signature, an image of the reflected light pattern can be compared to the distribution of the emitted radiation pattern. The determination of the optical difference signature can occur at processing module 70 or any other local or remote processing circuitry associated with the augmented reality display system 2010. Processing module 70 can search for a unique difference between the emitted and reflected light patterns that can be used as landmarks, markers, or references. If a unique difference is found between the emitted and reflected light patterns, the difference can be stored and the optical difference signature or marker can be recorded. In some cases, the unique difference can be caused by a portion of a real object having a different reflectivity than the surrounding area, such as a birthmark, mole, vein, scar tissue, or other structure on an arm or hand. For example, if an area of ​​scar tissue is present in the area being illuminated and imaged, the optical difference between the emitted and reflected radiation distributions can include an anomalous region shaped like the scar tissue area that can be used as a landmark or difference signature. (Since various types of biological tissues (such as skin, fat, oxygenated blood, deoxygenated blood, etc.) can have different infrared absorption rates and scattering characteristics, the radiation pattern can include multiple light wavelengths if a unique difference cannot be detected at the first wavelength.) If a detectable and / or locally unique sub-region of optical difference is found between the emitted and reflected light patterns, this difference can be stored as an optical difference signature. If no unique difference that can be used as a landmark or reference is found after comparing the emitted and reflected light patterns, box 1120 can be repeated by radiating the radiation pattern onto different locations on the real object, for example, different portions adjacent to the virtual object location and / or locations slightly further away from the virtual object location. If the unique difference that can be used as a landmark is identified and stored as an optical difference signature or marker, method 1100 continues to box 1130.

[0135] At box 1130, the position of the virtual content is determined relative to the optical signature, landmark, or "difference marker." The position of the optical signature, landmark, or difference marker can be used as a reference point for rendering the virtual content. Displacement can then be determined relative to the reference point for the virtual object. For example, displacement can include coordinates in one or more dimensions. In some embodiments, displacement can be a position in a two-dimensional or three-dimensional coordinate system with the origin at the reference point. Therefore, when the same optical signature, marker, or landmark is detected again, the virtual content can be rendered at the same position relative to the optical signature, marker, or landmark.

[0136] In some implementations, the optical signature, marker, or landmark may also be associated with a real-world physical reference point detectable by the depth sensor 28. The physical reference point may be a feature of a real object, such as a finger or wrist bone in the virtual watch example described herein. Similar to the position of the virtual object relative to the optical signature, marker, or landmark, the position of the optical signature, marker, or landmark relative to the physical reference point may include coordinates in one or more dimensions. Displacements between the virtual object and the optical marker or landmark, and between the physical reference point and the optical marker or landmark, may be recorded in the same or different reference frames or coordinate systems.

[0137] After determining the positions of the virtual content and physical reference points relative to optical markers or landmarks, system 2010 can use depth sensor 28 to monitor the position of the physical reference points intermittently or continuously. In some embodiments, depth sensor 28 may require less power and / or processing power than light detector 24 and light source 26 to continuously monitor the position of the real object and / or physical reference points. Therefore, depth sensor 28 can continuously monitor the position of the physical reference points while utilizing the emission and detection of radiation or radiation patterns by light source 26 and light detector 24 at a lower frequency, such as when a change in the position of the physical reference point is detected, when a frame of the virtual content needs to be refreshed, or at any other regular or irregular interval. When the position of the virtual content is to be refreshed, method 1100 continues to block 1140.

[0138] At box 1140, radiation or a radiation pattern is emitted again. The radiation emitted at box 1140 can have the same distribution (e.g., uniform distribution, speckle, pattern, texture, etc.) as the radiation emitted at box 1120. The direction of the emitted radiation pattern can be determined based on the position of a physical reference point, such as that tracked by depth sensor 28, such that the emitted radiation pattern is at least partially incident on the area of ​​the real object where a photometric signature, mark, or landmark is expected to be found. The direction of emission can be selected by pivoting or otherwise moving one or more light sources 24 on a gimbal or other movable mounting platform. For example, in the example of a wearer's arm, the emitted radiation pattern can be directed to the location of a birthmark, vein, or other feature associated with a photometric signature, mark, or landmark based on a calculated displacement from the physical reference point. The radiation or radiation pattern can be emitted by light source 24 in the same manner as described above with reference to box 1120. After the radiation or radiation pattern is emitted, method 1100 continues to box 1150.

[0139] At box 1150, system 2010 obtains the reflected radiation distribution or pattern and locates a difference signature, marker, or landmark within the reflected radiation distribution or pattern. The reflected radiation or pattern can be detected and / or imaged by photodetector 26, which is configured to detect light of one or more wavelengths emitted by light source 24. Differences can then be determined and / or calculated between the emitted radiation pattern and the reflected radiation pattern. The stored optical difference signature, marker, or landmark can be compared with the differences between the emitted radiation distribution or pattern and the reflected radiation distribution or pattern to verify its existence and determine the position of the optical difference signature, marker, or landmark in the new reflected radiation distribution. The position of the optical difference signature, marker, or landmark can then be recorded as a reference point for rendering the virtual content when it is refreshed. After the optical difference marker is located within the reflected radiation distribution or difference, method 1100 continues to box 1160.

[0140] At frame 1160, the virtual content is rendered relative to the position of an optical signature, mark, or landmark. The position for rendering the virtual content can be determined based on the displacement between the optical signature, mark, or landmark and the virtual content calculated at frame 1130. By using the same previously calculated displacement, the augmented reality display system 2010 can achieve the advantage of displaying virtual content that appears to move along with the associated real object. For example, by repeatedly rendering a virtual watch at the same displacement relative to an optical signature, mark, or landmark caused by a birthmark or group of veins on the wearer's hand, the face of the virtual watch can appear to remain near the top side of the wearer's wrist, even if the wearer rotates their wrist or changes the orientation of their wrist in a manner that cannot be reliably detected by the depth sensor 28 alone. After the virtual content is rendered relative to the optical signature, mark, or landmark, method 1100 can return to frame 1140, wherein frames 1140-1160 can be repeated indefinitely at regular or irregular intervals whenever the virtual content frame is to be refreshed. For example, method 1100 may return to box 1140 at any time when a change in the position of the physical reference point is detected or at regular intervals (such as every second, five times per second, ten times per second, fifty times per second, one hundred times per second, or any other suitable interval). In some embodiments, light source 24 may be configured to continuously project radiation such as infrared light, rather than sending discrete pulses of light each time the difference marker is repositioned.

[0141] As discussed above, the systems, apparatus, methods, and processes discussed in conjunction with the illumination of light patterns are also applicable to uniformly illuminating or projecting spots of any shape, such as circular spots. Furthermore, while the above discussion involves the identification and use of markers or landmarks, multiple markers or landmarks can be identified and used to provide tracking and displacement of virtual image content.

[0142] It is anticipated that innovations can be realized in or associated with a wide range of applications and therefore include a broad range of variations. For example, variations in the shape, number, and / or refractive power of the EPE are anticipated. The structures, devices, and methods described herein can be found in particular for use in displays such as wearable displays (e.g., head-mounted displays), which can be used for augmented reality and / or virtual reality. More generally, the described embodiments can be implemented in any device, apparatus, or system that can be configured to display images (whether moving (e.g., video) or fixed (e.g., still images), and whether text, graphics, or drawings). However, it is anticipated that the described embodiments can be included in or associated with a variety of electronic devices, such as, but not limited to: mobile phones, cellular phones with multimedia internet enabled, mobile TV receivers, wireless devices, smartphones, etc. Devices, personal digital assistants (PDAs), wireless email receivers, handheld or portable computers, netbooks, notebooks, smartbooks, tablets, printers, copiers, scanners, fax machines, GPS receivers / navigators, cameras, digital media players (such as MP3 players), camcorders, game consoles, watches, clocks, calculators, television monitors, flat panel displays, electronic reading devices (e.g., e-readers), computer monitors, automatic displays (including odometer and speedometer displays, etc.), cockpit controls and / or displays, camera view displays (such as displays for rear-view cameras in vehicles), electronic photographs, electronic billboards or signs, projectors, architectural structures, microwaves, refrigerators, stereo systems, cassette recorders or players, DVD players, CD players, VCRs, radios, portable memory chips, washing machines, dryers, washer / dryer units, parking timers, head-mounted displays, and various imaging systems. Therefore, the teachings are not intended to be limited to the embodiments depicted only in the accompanying drawings, but rather have a wide applicability as will readily become apparent to those skilled in the art.

[0143] Various modifications to the embodiments described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded the widest scope consistent with the disclosure, principles, and novel features disclosed herein. The word “exemplary” is used exclusively herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous relative to other embodiments. Furthermore, those skilled in the art will readily understand that the terms “above” and “below,” “above” and “below,” etc., are sometimes used to facilitate the description of the drawings and to indicate the relative positions of the orientations of figures corresponding to appropriate orientations on pages, and may not reflect the appropriate orientation of the structures described herein, as those structures are implemented.

[0144] In the context of separate embodiments, certain features described in this specification can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually or in any suitable sub-combination in multiple embodiments. Moreover, although features may be described above as acting in certain combinations and even so initially claimed, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0145] Similarly, although operations are depicted in the accompanying drawings in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or that all illustrated operations be performed to achieve the desired result. Furthermore, the drawings may schematically depict one or more example processes in the form of flowcharts. However, other operations not depicted may be included within the schematically illustrated example processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any illustrated operation. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Furthermore, other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result.

[0146] Various exemplary embodiments of the invention are described herein. These examples are referred to in a non-limiting sense, provided to illustrate a wider range of applicability of the invention. Various changes may be made to the described invention and equivalents may be substituted without departing from the true spirit and scope of the invention. Furthermore, numerous modifications may be made to adapt particular circumstances, materials, composition of substances, processes, process actions(s) or steps(s) to the purposes, spirit, or scope of the invention(s). Moreover, as will be understood by those skilled in the art, each individual variant described and illustrated herein has discrete components and features that can be readily separated from or combined with features of any of the other several embodiments without departing from the scope or spirit of this disclosure. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0147] This invention includes methods that can be performed using a subject device. The methods may include actions of providing such a suitable device. Such provision can be performed by an end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set up, activate, power on, or otherwise perform actions to provide the necessary device in this method. The methods described herein can be performed in any logically possible order of events and in the order in which events are recorded.

[0148] The exemplary aspects of the invention, along with details regarding material selection and manufacturing, have been described above. Other details of the invention can be understood in conjunction with the patents and disclosures mentioned above and are generally known or understood by those skilled in the art. The same applies to the method-based aspects of the invention, as is typically or logically used.

[0149] Furthermore, although the invention has been described with reference to several examples that optionally include various features, the invention is not limited to the invention as described or indicated with respect to each variation thereof. Various changes may be made to the described invention and equivalents may be substituted (whether set forth herein or not included for some reason of brevity) without departing from the true spirit and scope of the invention. Additionally, where a range of values ​​is provided, it should be understood that every intermediate value between the upper and lower limits of the range, and any other stated or intermediate value within the declared range, is covered in the invention.

[0150] Furthermore, it should be anticipated that any optional feature of the described inventive variant may be formulated and claimed independently or in combination with any one or more of the features described herein. References to singular items include the possibility of multiple identical items. More specifically, as used herein and in the claims associated therewith, unless otherwise stated, the singular forms “a,” “an,” “said,” and “the” include plural indicators. In other words, the use of articles allows for “at least one” of the subject matter items described above and in the claims associated with this disclosure. It should also be noted that such claims may be drafted without any optional elements. Thus, the statement is intended to serve as a prior basis for using specialized terms such as “solely,” “only,” or using a “negative” restriction in conjunction with the elements of the claim.

[0151] Without using such specialized terminology, the term "comprising" in claims associated with this disclosure shall allow for the inclusion of any additional elements—regardless of whether a given number of elements are enumerated in such a claim, or whether the addition of a feature may be considered as a transformation of the nature of an element set forth in such a claim. Except as specifically defined herein, all technical and scientific terms used herein shall be given the broadest possible meaning as commonly understood, while maintaining the validity of the claims.

[0152] The scope of this invention will not be limited to the examples provided and / or this specification, but rather is defined only by the scope of the claims associated with this disclosure.

Claims

1. An augmented reality display system configured to align 3D content with real-world objects, the system comprising: A frame that is configured to be mounted on the wearer; An augmented reality display is attached to the frame and configured to direct images to the wearer's eyes; A light source configured to illuminate at least a portion of the object by emitting invisible light; A light sensor configured to image the portion of the object illuminated by the light source using the invisible light; A depth sensor configured to detect the position of the object; as well as The processing circuit is configured as follows: The depth sensor is used to periodically monitor the position of the object at a first frequency; as well as The orientation of the object is periodically monitored at a second frequency less than the first frequency, based on one or more characteristics of features in an image formed using the reflective portion of the invisible light.

2. The augmented reality display system according to claim 1, wherein, The processing circuitry is further configured to render augmented reality content onto a view of the object via the augmented reality display.

3. The augmented reality display system according to claim 2, wherein, The processing circuitry is further configured to orient the presented augmented reality content based at least in part on the orientation of the object's monitoring.

4. The augmented reality display system according to claim 1, wherein, The processing circuit is further configured to determine the difference between the distribution of the emitted invisible light and the distribution of the reflected portion of the invisible light.

5. The augmented reality display system according to claim 4, wherein, The processing circuitry is further configured to identify a difference signature based on the determined differences.

6. The augmented reality display system according to claim 5, wherein, The processing circuitry is further configured to detect the features in the image based at least in part on the difference signature.

7. The augmented reality display system according to claim 6, wherein, The processing circuitry is further configured to select and store the features as markers for rendering augmented reality content onto a view of the object via the augmented reality display.

8. The augmented reality display system according to claim 7, wherein, The object includes the wearer's skin or the skin of a person other than the wearer.

9. The augmented reality display system according to claim 8, wherein, The features include pre-existing surface features of the skin.

10. The augmented reality display system according to claim 1, wherein, The augmented reality display is configured to transmit light from the environment in front of the wearer to the wearer's eyes to provide a view of the environment in front of the wearer, while the augmented reality display directs the image to the wearer's eyes.

11. A method for aligning 3D content with a real object, the method comprising, under the control of processing circuitry: The light source of the augmented reality display system emits invisible light to illuminate at least a portion of the object; The augmented reality display system uses a light sensor to image the portion of the object illuminated by the light source; The position of the object is periodically monitored at a first frequency using the depth sensor of the augmented reality display system. The orientation of the object is periodically monitored at a second frequency less than the first frequency, based on one or more characteristics of features in an image formed using the reflective portion of the invisible light. as well as Augmented reality content is presented onto a view of the object by the augmented reality display of the augmented reality display system to the wearer of the augmented reality display system.

12. The method according to claim 11, wherein, The rendered augmented reality content is oriented at least in part based on the orientation of the monitored object.

13. The method according to claim 11, wherein, The virtual location of the rendered augmented reality content is determined at least in part based on the monitored location of the object.

14. The method of claim 11, further comprising determining the difference between the distribution of the emitted invisible light and the distribution of the reflected portion of the invisible light.

15. The method of claim 14, further comprising identifying a difference signature based on the determined differences.

16. The method of claim 15, further comprising detecting the features in the image at least in part based on the differential signature.

17. The method of claim 16, further comprising selecting and storing the features for use as markers for rendering the augmented reality content onto a view of the object.

18. The method according to claim 17, wherein, The object includes the wearer's skin or the skin of a person other than the wearer.

19. The method according to claim 18, wherein, The features include pre-existing surface features of the skin.

20. The method according to claim 19, wherein, The features include birthmarks, moles, or scar tissue.

Images