Single pupil RGB light source

Abstract

To provide a single pupil RGB light source. [Solution] Embodiments of the present disclosure are systems and methods for displays. In embodiments, the display system includes a light source configured to emit a first light, a lens configured to receive the first light, and an image generator configured to receive the first light and emit a second light. The display system further includes a plurality of waveguides, at least two of which include an internal coupling grating configured to selectively couple the second light. In some embodiments, the light source may comprise a single pupil light source having a reflector and a micro-LED array disposed within the reflector.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Application No. 63 / 135,533, filed on January 8, 2021, and Provisional Application No. 63 / 165,663, filed on March 24, 2021, the contents of both of which are incorporated herein by reference in their entirety.

[0002] The present disclosure generally relates to systems and methods for displaying visual information, and more particularly, to systems and methods for displaying visual information within a mixed - reality environment.

Background Art

[0003] Virtual environments are ubiquitous in computing environments and are found in use in video games (where the virtual environment can represent a game world), maps (where the virtual environment can represent terrain to be navigated), simulations (where the virtual environment can simulate a real - world environment), digital storytelling (where virtual characters can interact with each other within the virtual environment), and many other applications. Modern computer users are generally comfortable perceiving and interacting with virtual environments. However, a user's experience with a virtual environment can be limited by the technology used to present the virtual environment. For example, conventional displays (e.g., 2D display screens) and audio systems (e.g., fixed speakers) may not be able to realize a virtual environment in a way that attracts people, creates a realistic, and immersive experience.

[0004] Virtual reality ("VR"), augmented reality ("AR"), mixed reality ("MR"), and related technologies (collectively, "XR") share the ability to present users of XR systems with sensory information corresponding to a virtual environment represented by data within a computer system. This disclosure assumes specificity between VR, AR, and MR systems (however, some systems may be categorized as VR in one aspect (e.g., a visual aspect) and simultaneously as AR or MR in another aspect (e.g., an audio aspect)). As used herein, a VR system presents a virtual environment that replaces the user's real environment in at least one aspect. For example, a VR system may present the user with a view of the virtual environment while simultaneously obscuring that view of the real environment, such as by using a light-blocking head-mounted display. Similarly, a VR system may present the user with audio corresponding to the virtual environment while simultaneously blocking (attenuating) audio from the real environment.

[0005] VR systems can suffer from various drawbacks stemming from their replacement of the user's real environment with a virtual one. One drawback is motion sickness, which can occur when the user's field of vision in the virtual environment no longer corresponds to the state of their inner ear that detects their balance and orientation in the real environment (rather than the virtual one). Similarly, users can suffer disorientation in the VR environment if their own body and limbs (on which they rely to feel "grounded" in the real environment) are not directly visible. Another drawback is the computational burden (e.g., memory, processing power) placed on VR systems, especially in real-time applications where the goal is to immerse the user in the virtual environment, as they must present a fully 3D virtual environment. Likewise, such environments may need to reach a very high level of reality in order to be considered immersed, as users tend to be sensitive to even the slightest imperfections in the virtual environment, any of which can disrupt the user's sense of immersion. Furthermore, another drawback of VR systems is that such applications of the system cannot utilize the wide range of sensory data found in real environments, such as various sights and sounds, that are experienced in the real world. A related drawback is that VR systems may struggle to create shared environments in which multiple users can interact, as it may be impossible for users sharing the same physical space in the real environment to directly see or interact with each other within the virtual environment.

[0006] As used herein, an AR system presents a virtual environment that overlaps or overlays the real environment in at least one aspect. For example, an AR system may present a view of the virtual environment overlaid on the user's view of the real environment, for instance, by using a transparent head-mounted display that presents images while allowing light to pass through the display into the user's eyes. Similarly, an AR system may present audio corresponding to the virtual environment while simultaneously mixing it with audio from the real environment. Likewise, as used herein, an MR system, like an AR system, presents a virtual environment that overlaps or overlays the real environment in at least one aspect, and in addition, may allow the virtual environment within the MR system to interact with the real environment in at least one aspect. For example, a virtual character in the virtual environment may toggle a light switch in the real environment, turning a corresponding light bulb in the real environment on or off. In another embodiment, a virtual character may react to audio signals in the real environment (e.g., using facial expressions). By maintaining the presentation of the real environment, AR and MR systems can avoid some of the aforementioned shortcomings of VR systems. For example, motion sickness in users is reduced because visual cues from the real environment (including the user's own body) can remain visible, and such systems, being immersive, do not need to present the user with a fully realized 3D environment. Furthermore, AR and MR systems can create new applications by leveraging and extending real-world sensory input (e.g., scenery, objects, and the views and sounds of other users).

[0007] Presenting a virtual environment in a realistic manner to create an immersive experience for the user can be challenging. For example, head-mounted displays are designed to be lightweight and compact to minimize user burden while presenting high-quality images with a wide field of view ("FOV"). However, real-world design constraints can lead to trade-offs between the size and weight of the optical system in a head-wearable device and the quality of the displayed image. Image quality can depend, for example, on the efficiency of the optical system, such as the uniformity of light and color, and the ability to maintain the field of view. Therefore, there is a need for compact optical systems that are efficient and have a wide field of view, such as those used to present virtual environments in VR, AR, or MR systems. [Overview of the project] [Means for solving the problem]

[0008] Disclosed herein are systems and methods for displays, such as for head-wearable devices. An exemplary display system may include a light source configured to emit a first light, a lens configured to receive the first light, and an image generator configured to receive the first light and emit a second light. The display system may further include a plurality of waveguides, at least one of which may include an internal coupling grating configured to selectively couple the second light. In some embodiments, the light source may comprise a single pupil light source having a reflector and a micro-LED array disposed within the reflector. Embodiments disclosed herein may provide compact and efficient display systems. The present invention provides, for example, the following: (Item 1) It is a display system, A light source configured to emit a first light, A lens configured to receive the first light, An image generator configured to receive the first light and emit the second light, A plurality of waveguides, wherein at least two of the plurality of waveguides each include an internal coupling grating configured to selectively couple the second light, and Equipped with, The display system comprises a single pupil light source, which includes a reflector and a micro-LED array disposed within the reflector. (Item 2) The display system according to item 1, wherein the microLED array includes at least a plurality of first microLED sources configured to emit at a first wavelength and a plurality of second microLED sources configured to emit at a second wavelength. (Item 3) The display system according to item 2, wherein at least a first internal coupling grating is configured to internally couple light corresponding to the first wavelength, and a second internal coupling grating is configured to internally couple light corresponding to the second wavelength. (Item 4) The display system according to item 2, wherein the light source is configured to emit a first wavelength at a first time and a second wavelength at a second time, the second time being different from the first time. (Item 5) The internal coupling grids of at least two of the waveguides are matched, as in the display system described in item 1. (Item 6) The display system according to item 1, wherein the light source is offset from the image generator. (Item 7) The display system according to item 1, wherein the reflector comprises a composite parabolic condenser reflector. (Item 8) It is a head-mounted wearable device, It is a display system, A light source configured to emit a first light, A lens configured to receive the first light, An image generator configured to receive the first light and emit the second light, A plurality of waveguides, wherein at least two of the plurality of waveguides each include an internal coupling grating configured to selectively couple the second light, and Equipped with, The light source comprises a single pupil light source including a reflector and a micro-LED array disposed within the reflector, in a display system A head-mounted wearable device equipped with [feature / feature]. (Item 9) The head wearable device according to item 8, wherein the microLED array includes at least a plurality of first microLED sources configured to emit at a first wavelength and a plurality of second microLED sources configured to emit at a second wavelength. (Item 10) The head-wearable device according to item 9, wherein at least a first internal coupling grating is configured to internally couple light corresponding to the first wavelength, and a second internal coupling grating is configured to internally couple light corresponding to the second wavelength. (Item 11) The light source is configured to emit a first wavelength at a first time and a second wavelength at a second time, wherein the second time is different from the first time, as described in item 9 of the head wearable device. (Item 12) The internal coupling grids of at least two of the waveguides are matched in the head-wearable device described in item 8. (Item 13) The light source is offset from the image generator, and the head wearable device is as described in item 8. (Item 14) The head-wearable device according to item 8, wherein the reflector comprises a composite parabolic condenser reflector. (Item 15) It is a display system, A single pupil light source, A reflector having an inlet opening and an outlet opening, A microLED array positioned in close proximity to the aforementioned entrance opening and A single-pupil light source comprising A display system comprising (Item 16) The display system according to item 15, wherein the reflector comprises a compound parabolic concentrator reflector. (Item 17) The display system according to item 15, wherein the micro-LED array comprises a plurality of micro-LEDs arranged in at least one selected from the group consisting of a square configuration, a rectangular configuration, a hexagonal configuration, a radial configuration, and a stripe configuration. (Item 18) The display system according to item 15, wherein the micro-LED array comprises a plurality of micro-LED sources, and the micro-LED sources have at least one of a circular shape, a rectangular shape, and a square shape. (Item 19) The display system according to item 15, wherein the micro-LED array is configured to emit light at at least three wavelengths. (Item 20) The display system according to item 15, wherein the micro-LED array is disposed across the incident aperture.

Brief Description of the Drawings

[0009] [Figure 1A] Figures 1A-1C illustrate an exemplary mixed reality environment according to one or more embodiments of the present disclosure. [Figure 1B] Figures 1A-1C illustrate an exemplary mixed reality environment according to one or more embodiments of the present disclosure. [Figure 1C] Figures 1A-1C illustrate an exemplary mixed reality environment according to one or more embodiments of the present disclosure.

[0010] [Figure 2A] Figures 2A-2D illustrate components of an exemplary mixed reality system that can be used to generate and interact with a mixed reality environment according to one or more embodiments of the present disclosure. [Figure 2B] Figures 2A-2D illustrate components of an exemplary mixed reality system that may be used to generate and interact with a mixed reality environment, according to one or more embodiments of the present disclosure. [Figure 2C] Figures 2A-2D illustrate components of an exemplary mixed reality system that may be used to generate and interact with a mixed reality environment, according to one or more embodiments of the present disclosure. [Figure 2D] Figures 2A-2D illustrate components of an exemplary mixed reality system that may be used to generate and interact with a mixed reality environment, according to one or more embodiments of the present disclosure.

[0011] [Figure 3A] Figure 3A illustrates an exemplary mixed reality handheld controller that may be used to provide input to a mixed reality environment according to one or more embodiments of the present disclosure.

[0012] [Figure 3B] Figure 3B illustrates an exemplary auxiliary unit that may be used in conjunction with an exemplary mixed reality system according to one or more embodiments of the present disclosure.

[0013] [Figure 4] Figure 4 illustrates an exemplary functional block diagram relating to an exemplary mixed reality system according to one or more embodiments of the present disclosure.

[0014] [Figure 5] Figure 5 illustrates an exemplary optical system for an exemplary mixed reality system according to one or more embodiments of the present disclosure.

[0015] [Figure 6] Figure 6 illustrates an exemplary optical system for an exemplary mixed reality system according to one or more embodiments of the present disclosure.

[0016] [Figure 7] Figure 7 illustrates an exemplary optical system for an exemplary mixed reality system according to one or more embodiments of the present disclosure.

[0017] [Figure 8A] Figures 8A–8C illustrate exemplary optical systems for exemplary mixed reality systems according to one or more embodiments of the present disclosure. [Figure 8B] Figures 8A–8C illustrate exemplary optical systems for exemplary mixed reality systems according to one or more embodiments of the present disclosure. [Figure 8C] Figures 8A–8C illustrate exemplary optical systems for exemplary mixed reality systems according to one or more embodiments of the present disclosure.

[0018] [Figure 9] Figure 9 illustrates an exemplary reflector for an exemplary optical system according to one or more embodiments of the present disclosure.

[0019] [Figure 10] Figure 10 illustrates an exemplary light source for an exemplary optical system according to one or more embodiments of the present disclosure.

[0020] [Figure 11A] Figures 11A-11B illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 11B] Figures 11A-11B illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0021] [Figure 12A] Figures 12A–12B illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 12B] Figures 12A–12B illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0022] [Figure 13A] Figures 13A–13B illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 13B] Figures 13A–13B illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0023] [Figure 14A] Figures 14A-14B illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 14B] Figures 14A-14B illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0024] [Figure 15A] Figures 15A–15H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 15B] Figures 15A–15H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 15C] Figures 15A–15H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 15D] Figures 15A–15H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 15E] Figures 15A–15H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 15F] Figures 15A–15H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 15G] Figures 15A–15H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 15H] Figures 15A–15H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0025] [Figure 16A] Figures 16A–16H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 16B] Figures 16A–16H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 16C] Figures 16A–16H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 16D] Figures 16A–16H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 16E] Figures 16A–16H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 16F] Figures 16A–16H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 16G] Figures 16A–16H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 16H] Figures 16A–16H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0026] [Figure 17A] Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 17B] Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 17C] Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 17D] Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 17E] Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 17F] Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 17G] Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 17H]Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 17I] Figures 17A-17I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0027] [Figure 18A] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 18B] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 18C] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 18D] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 18E] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 18F] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 18G] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 18H] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 18I] Figures 18A-18I illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0028] [Figure 19A] Figures 19A–19H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 19B] Figures 19A–19H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 19C] Figures 19A–19H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 19D] Figures 19A–19H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 19E] Figures 19A–19H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 19F] Figures 19A–19H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 19G] Figures 19A–19H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure. [Figure 19H] Figures 19A–19H illustrate exemplary illuminances of exemplary micro-LED arrays for exemplary optical systems according to one or more embodiments of the present disclosure.

[0029] [Figure 20]Figure 20 illustrates an exemplary micro-LED array for an exemplary optical system according to one or more embodiments of the present disclosure. [Modes for carrying out the invention]

[0030] Detailed explanation In the following description of the embodiments, accompanying drawings, which form part of this specification and illustrate specific embodiments that can be put into practice, are referenced. It should be understood that other embodiments may also be used, and structural modifications may be made without departing from the scope of the disclosed embodiments. Mixed reality environment

[0031] Like all people, users of a mixed reality system perceive the three-dimensional parts of the real environment, i.e., the "real world," and all of its contents. For example, users perceive the real environment using their normal human senses, namely sight, hearing, touch, taste, and smell, and interact with the real environment by moving their bodies within it. Locations within the real environment can be described as coordinates in coordinate space, for example, coordinates can comprise latitude, longitude, and altitude relative to sea level, distance in three orthogonal dimensions from a reference point, or other preferred values. Similarly, vectors can describe quantities that have direction and magnitude in coordinate space.

[0032] A computing device can maintain a representation of a virtual environment, for example, in memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. A virtual environment can include representations of any object, action, signal, parameter, coordinate, vector, or other properties associated with that space. In some embodiments, the circuitry of the computing device (e.g., a processor) can maintain and update the state of the virtual environment; that is, the processor can determine the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or inputs provided by the user at a first time t0. For example, if an object in the virtual environment is located at a first coordinate at time t0, has some programmed physical parameters (e.g., mass, coefficient of friction), and inputs received from the user indicate that a force should be applied to the object in a certain direction vector, the processor can determine the object's location at time t1 by applying the laws of kinematics and using basic mechanics. The processor can determine the state of the virtual environment at time t1 using any known suitable information about the virtual environment and / or any suitable inputs. When maintaining and updating the state of a virtual environment, the processor may run any suitable software, including software related to creating and deleting virtual objects within the virtual environment, software for defining the behavior of virtual objects or characters within the virtual environment (e.g., scripts), software for defining the behavior of signals within the virtual environment (e.g., audio signals), software for creating and updating parameters associated with the virtual environment, software for generating audio signals within the virtual environment, software for handling inputs and outputs, software for implementing network operations, software for applying asset data (e.g., animation data for moving virtual objects over time), or many other possibilities.

[0033] Output devices such as displays or speakers can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects, people, animals, light, etc.) that can be presented to the user. The processor can determine a view of the virtual environment (e.g., corresponding to a "camera" with origin coordinates, visual axes, and frustum) and render a viewable scene of the virtual environment corresponding to that view on the display. Any suitable rendering technique may be used for this purpose. In some embodiments, the viewable scene may include only some virtual objects in the virtual environment and exclude some other virtual objects. Similarly, the virtual environment may include an audio aspect that can be presented to the user as one or more audio signals. For example, virtual objects in the virtual environment may generate sounds resulting from the object's location coordinates (e.g., a virtual character may speak or produce sound effects), or the virtual environment may be associated with musical cues or ambient sounds, which may or may not be associated with a specific location. The processor can determine an audio signal that corresponds to the "listener" coordinates, for example, the synthesis of sound in a virtual environment, which is mixed and processed to simulate the audio signal that would be heard by the listener at the listener coordinates, and present the audio signal to the user through one or more speakers.

[0034] Because a virtual environment exists only as a computational structure, users cannot directly perceive it using their normal senses. Instead, users can only perceive the virtual environment indirectly, for example, by being presented to them through displays, speakers, haptic output devices, etc. Similarly, users cannot directly touch, manipulate, or otherwise interact with the virtual environment, but they can provide input data via input devices or sensors to a processor that can update the virtual environment using device or sensor data. For example, a camera sensor may provide optical data indicating that the user is attempting to move an object in the virtual environment, and the processor can use this data to cause the object to respond appropriately within the virtual environment.

[0035] A mixed reality system can present a user with a mixed reality environment ("MRE") that combines aspects of the real and virtual environments, for example, using a transparent display and / or one or more speakers (which may be incorporated in a wearable head device, for example). In some embodiments, one or more speakers may be located outside the head-mounted wearable unit. As used herein, the MRE is a simultaneous representation of the real environment and the corresponding virtual environment. In some embodiments, the corresponding real and virtual environments share a single coordinate space, and in some embodiments, the real coordinate space and the corresponding virtual coordinate space are related by a transformation matrix (or other preferred representation). Thus, a single coordinate (in some embodiments, together with the transformation matrix) can define a first location in the real environment and a second corresponding location in the virtual environment, and vice versa.

[0036] In MRE, virtual objects (for example, in a virtual environment associated with the MRE) can correspond to real objects (for example, in a real environment associated with the MRE). For example, if the real environment of the MRE has a real lamppost (real object) at a certain location coordinate, the virtual environment of the MRE may have a virtual lamppost (virtual object) at the corresponding location coordinate. As used herein, real objects, together with their corresponding virtual objects, constitute a “mixed reality object.” It is not necessary for the virtual object to perfectly match or be consistent with the corresponding real object. In some embodiments, the virtual object may be a simplified version of the corresponding real object. For example, if the real environment includes a real lamppost, the corresponding virtual object may have a cylindrical shape with approximately the same height and radius as the real lamppost (reflecting that the lamppost may be roughly cylindrical in shape). Simplifying the virtual object in this way can enable computational efficiency and simplify the calculations that would have to be performed on such virtual objects. Furthermore, in some embodiments of the MRE, not all real objects in the real environment have to be associated with corresponding virtual objects. Similarly, in some embodiments of MRE, not all virtual objects within the virtual environment may be associated with corresponding real-world objects. That is, some virtual objects may exist only within the MRE's virtual environment without any real-world counterparts.

[0037] In some embodiments, virtual objects may have characteristics that are sometimes significantly different from those of their corresponding real objects. For example, a real environment within an MRE might comprise a cactus with two green branches, i.e., a thorny, inanimate object, while a corresponding virtual object within an MRE might have the characteristics of a virtual character with two green arms, accompanied by human facial features and an unfriendly demeanor. In this embodiment, the virtual object is similar to its corresponding real object in some characteristics (color, number of arms), but differs from the real object in other characteristics (facial features, personality). Thus, virtual objects have the potential to represent real objects in a creative, abstract, exaggerated, or fictional way, or to confer behavior (e.g., human personality) to real objects that are otherwise inanimate. In some embodiments, virtual objects may be purely fictional creations without real-world counterparts (e.g., a virtual monster in a virtual environment, perhaps in a place corresponding to emptiness in a real environment).

[0038] Compared to VR systems, which present a virtual environment while obscuring the real environment, mixed reality systems that present MRE offer the advantage that the real environment remains perceptible while the virtual environment is presented. Therefore, users of mixed reality systems can experience and interact with the corresponding virtual environment using visual and audio cues associated with the real environment. For example, users of VR systems may struggle to perceive or interact with virtual objects displayed within the virtual environment because, as described above, the user cannot directly perceive or interact with the virtual environment. In contrast, users of MR systems may find it intuitive and natural to interact with virtual objects because they can see, hear, and touch the corresponding real objects in their own real environment. This level of interaction can enhance the user's sense of immersion, connection, and engagement with the virtual environment. Similarly, by presenting the real and virtual environments simultaneously, mixed reality systems can reduce the negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with VR systems. Mixed reality systems also offer many possibilities for applications that can extend or modify our real-world experiences.

[0039] Figure 1A illustrates an exemplary real environment 100 in which a user 110 uses a mixed reality system 112. The mixed reality system 112 may comprise, for example, a display (e.g., a transmissive display) and one or more speakers, and one or more sensors (e.g., a camera), as described below. The illustrated real environment 100 comprises a rectangular room 104A in which the user 110 stands, and real objects 122A (a lamp), 124A (a table), 126A (a sofa), and 128A (a painting). Room 104A further comprises a location coordinate 106, which can be considered the origin of the real environment 100. As shown in Figure 1A, an environment / world coordinate system 108 (with x-axis 108X, y-axis 108Y, and z-axis 108Z) with its origin at point 106 (world coordinate) can define the coordinate space for the real environment 100. In some embodiments, the origin 106 of the environment / world coordinate system 108 may correspond to the location where the mixed reality system 112 is powered on. In some embodiments, the origin 106 of the environment / world coordinate system 108 may be reset during operation. In some embodiments, the user 110 may be considered a real object in the real environment 100, and similarly, the body parts of the user 110 (e.g., hands, feet) may be considered real objects in the real environment 100. In some embodiments, a user / listener / head coordinate system 114 (with x-axis 114X, y-axis 114Y, and z-axis 114Z), with its origin at point 115 (e.g., user / listener / head coordinates), can define a coordinate space for the user / listener / head on which the mixed reality system 112 is located. The origin 115 of the user / listener / head coordinate system 114 may be defined with respect to one or more components of the mixed reality system 112. For example, the origin 115 of the user / listener / head coordinate system 114 may be defined relative to the display of the mixed reality system 112, such as during the initial calibration of the mixed reality system 112. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can characterize the transformation between the user / listener / head coordinate system 114 space and the environment / world coordinate system 108 space.In some embodiments, the left ear coordinates 116 and the right ear coordinates 117 may be defined relative to the origin 115 of the user / listener / head coordinate system 114. Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other preferred representations can characterize the transformation between the left ear coordinates 116 and the right ear coordinates 117 and the user / listener / head coordinate system 114 space. The user / listener / head coordinate system 114 can simplify the representation of location relative to the user's head or head-mounted device, for example, the environment / world coordinate system 108. The transformation between the user coordinate system 114 and the environment coordinate system 108 can be determined and updated in real time using simultaneous localization and mapping (SLAM), visual odometry, or other techniques.

[0040] Figure 1B illustrates an exemplary virtual environment 130 corresponding to the real environment 100. The shown virtual environment 130 comprises a virtual rectangular room 104B corresponding to the real rectangular room 104A, a virtual object 122B corresponding to the real object 122A, a virtual object 124B corresponding to the real object 124A, and a virtual object 126B corresponding to the real object 126A. The metadata associated with the virtual objects 122B, 124B, and 126B may include information derived from the corresponding real objects 122A, 124A, and 126A. In addition, the virtual environment 130 comprises a virtual monster 132, which does not correspond to any real object in the real environment 100. The real object 128A in the real environment 100 does not correspond to any virtual object in the virtual environment 130. A persistent coordinate system 133 (with x-axis 133X, y-axis 133Y, and z-axis 133Z), with its origin at point 134 (persistent coordinate), can define a coordinate space for virtual content. The origin 134 of the persistent coordinate system 133 may be defined relative to / with respect to one or more real objects such as real object 126A. Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other preferred representations can characterize transformations between the persistent coordinate system 133 space and the environment / world coordinate system 108 space. In some embodiments, virtual objects 122B, 124B, 126B, and 132 may each have their own persistent coordinate points relative to the origin 134 of the persistent coordinate system 133. In some embodiments, there may be multiple persistent coordinate systems, and the virtual objects 122B, 124B, 126B, and 132 may each have their own persistent coordinate points relative to one or more persistent coordinate systems.

[0041] Persistent coordinate data can be coordinate data that persists in relation to the physical environment. Persistent coordinate data may be used by an MR system (e.g., MR systems 112, 200) to place persistent virtual content, which may not be tied to the movement of a display on which virtual objects are displayed. For example, a two-dimensional screen may display only virtual objects relative to a certain position on the screen. As the two-dimensional screen moves, the virtual content may move with the screen. In some embodiments, persistent virtual content may be displayed in the corner of a room. When an MR user looks at the corner, they may see the virtual content, and when they look away from the corner (the virtual content may no longer be visible because it has moved from within the user's field of view to outside the user's field of view due to the movement of the user's head), and look back, the virtual content may be visible in the corner (similar to how a real object may behave).

[0042] In some embodiments, persistent coordinate data (e.g., persistent coordinate system and / or persistent coordinate frame) may include an origin and three axes. For example, the persistent coordinate system may be assigned by the MR system to the center of a room. In some embodiments, a user may move around the room, leave the room, and re-enter the room, but the persistent coordinate system may remain at the center of the room (e.g., so that it persists relative to the physical environment). In some embodiments, virtual objects may be displayed using a transformation to persistent coordinate data, which may enable the display of persistent virtual content. In some embodiments, the MR system may use simultaneous localization and mapping to generate persistent coordinate data (e.g., the MR system may assign the persistent coordinate system to a point in space). In some embodiments, the MR system may map the environment by generating persistent coordinate data at regular intervals (e.g., the MR system may assign the persistent coordinate system to a grid, where the persistent coordinate system may be within at least 5 feet of another persistent coordinate system).

[0043] In some embodiments, persistent coordinate data may be generated by the MR system and transmitted to a remote server. In some embodiments, the remote server may be configured to receive persistent coordinate data. In some embodiments, the remote server may be configured to synchronize persistent coordinate data from multiple observation instances. For example, multiple MR systems may map the same room with persistent coordinate data and transmit that data to the remote server. In some embodiments, the remote server may use this observation data to generate reference persistent coordinate data, which may be based on one or more observations. In some embodiments, the reference persistent coordinate data may be more accurate and / or reliable than a single observation of persistent coordinate data. In some embodiments, the reference persistent coordinate data may be transmitted to one or more MR systems. For example, an MR system may use image recognition and / or location data to recognize that it is located in a room with corresponding reference persistent coordinate data (for example, because another MR system has previously mapped that room). In some embodiments, an MR system may receive reference persistent coordinate data corresponding to its location from the remote server.

[0044] With respect to Figures 1A and 1B, the environment / world coordinate system 108 defines a shared coordinate space for both the real environment 100 and the virtual environment 130. In the examples shown, the coordinate space has its origin at point 106. Furthermore, the coordinate space is defined by three identical orthogonal axes (108X, 108Y, 108Z). Thus, a first location in the real environment 100 and a second corresponding location in the virtual environment 130 can be described with respect to the same coordinate space. This simplifies the identification and representation of corresponding locations in the real and virtual environments, as the same coordinates can be used to identify both locations. However, in some examples, the corresponding real and virtual environments do not need to use a shared coordinate space. For example, in some examples (not shown), a matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other preferred representation can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0045] Figure 1C illustrates an exemplary MRE 150 that simultaneously presents aspects of the real environment 100 and the virtual environment 130 to the user 110 via the mixed reality system 112. In the shown embodiment, the MRE 150 simultaneously presents to the user 110 real objects 122A, 124A, 126A, and 128A from the real environment 100 (e.g., via the transparent portion of the display of the mixed reality system 112) and virtual objects 122B, 124B, 126B, and 132 from the virtual environment 130 (e.g., via the active display portion of the display of the mixed reality system 112). As described above, the origin 106 acts as the origin for the coordinate space corresponding to the MRE 150, and the coordinate system 108 defines the x-axis, y-axis, and z-axis for the coordinate space.

[0046] In the embodiments shown, a mixed reality object comprises corresponding pairs of real and virtual objects (i.e., 122A / 122B, 124A / 124B, 126A / 126B) occupying corresponding locations in coordinate space 108. In some embodiments, both the real and virtual objects may be visible to the user 110 simultaneously. This may be desirable, for example, in cases where information is presented in which the virtual object is designed to extend the view of the corresponding real object (e.g., in a museum setting where the virtual object presents a missing part of an ancient, damaged statue). In some embodiments, the virtual objects (122B, 124B, and / or 126B) may be displayed in such a way that they occlude the corresponding real objects (122A, 124A, and / or 126A) (e.g., via active pixelated occlusion, using a pixelated occlusion shutter). This can be desirable, for example, in cases where a virtual object acts as a visual replacement for a corresponding real object (such as in interactive storytelling applications where an inanimate real object becomes a "living" character).

[0047] In some embodiments, real objects (e.g., 122A, 124A, 126A) may be associated with virtual content or helper data, which may not necessarily constitute a virtual object. The virtual content or helper data can facilitate the processing or handling of virtual objects within a mixed reality environment. For example, such virtual content may include a two-dimensional representation of a corresponding real object, a custom asset type associated with the corresponding real object, or statistical data associated with the corresponding real object. This information can enable or facilitate calculations involving real objects without incurring unnecessary computational overhead.

[0048] In some embodiments, the presentations described above may also incorporate an audio aspect. For example, in MRE150, the virtual monster 132 may be associated with one or more audio signals, such as footsteps, generated as the monster walks around MRE150. As further described below, the processor of the mixed reality system 112 may compute an audio signal corresponding to the mixture and processed synthesis of all such sounds within MRE150 and present the audio signal to the user 110 via one or more speakers contained within the mixed reality system 112 and / or one or more external speakers. Exemplary Mixed Reality System

[0049] An exemplary mixed reality system 112 may include a wearable head device (e.g., a wearable augmented reality or mixed reality head device) comprising: a display (which may comprise left and right transmissive displays, which may be eyepiece displays, and associated components for coupling light from the displays to the user's eyes); left and right speakers (e.g., positioned adjacent to the user's left and right ears, respectively); an inertial measuring unit (IMU) (e.g., mounted on the temple arms of the head device); an orthogonal coil electromagnetic receiver (e.g., mounted on the left temple component); left and right cameras (e.g., depth (time-of-flight) cameras) oriented away from the user; and left and right eye cameras oriented towards the user (e.g., for detecting the user's eye movements). However, the mixed reality system 112 may incorporate any suitable display technology and any suitable sensors (e.g., optical, infrared, acoustic, LiDAR, EOG, GPS, magnetic). In addition, the mixed reality system 112 may incorporate networking features (e.g., Wi-Fi capability) to communicate with other devices and systems, including other mixed reality systems. The mixed reality system 112 may further include a battery (which may be housed in an auxiliary unit such as a belt pack designed to be worn around the user's waist), a processor, and memory. The wearable head device of the mixed reality system 112 may include a tracking component, such as an IMU or other suitable sensor, configured to output a set of coordinates of the wearable head device relative to the user's environment. In some embodiments, the tracking component may provide input to the processor and perform simultaneous localization and mapping (SLAM) and / or visual odometry algorithms. In some embodiments, the mixed reality system 112 may also include an auxiliary unit 320, which may be a handheld controller 300 and / or a wearable belt pack, as further described below.

[0050] Figures 2A-2D illustrate components of an exemplary mixed reality system 200 (which may correspond to mixed reality system 112) that may be used to present an MRE (which may correspond to MRE150) or other virtual environment to a user. Figure 2A shows a perspective view of a wearable head device 2102 included in the exemplary mixed reality system 200. Figure 2B shows a top view of the wearable head device 2102 worn on the user's head 2202. Figure 2C shows a front view of the wearable head device 2102. Figure 2D shows an edge view of an exemplary eyepiece 2110 of the wearable head device 2102. As shown in Figures 2A-2C, the exemplary wearable head device 2102 includes an exemplary left eyepiece (e.g., a left transparent waveguide set eyepiece) 2108 and an exemplary right eyepiece (e.g., a right transparent waveguide set eyepiece) 2110. Each eyepiece 2108 and 2110 may include a transmissive element through which the real environment may be visible, and a display element for presenting a display that overlaps with the real environment (e.g., via image-modulated light). In some embodiments, such a display element may include a surface diffractive optical element for controlling the flow of image-modulated light. For example, the left eyepiece 2108 may include a left internally coupled grating set 2112, a left orthogonal pupil extension (OPE) grating set 2120, and a left exit (output) pupil extension (EPE) grating set 2122. As used herein, the pupil may refer to the emission of light from an optical element such as a grating set or reflector. Similarly, the right eyepiece 2110 may include a right internally coupled grating set 2118, a right OPE grating set 2114, and a right EPE grating set 2116. The light modulated for each image can be transferred to the user's eye via internally coupled gratings 2112 and 2118, OPE 2114 and 2120, and EPE 2116 and 2122. Each internally coupled grating set 2112, 2118 can be configured to deflect the light toward its corresponding OPE grating set 2120, 2114.Each OPE grating set 2120, 2114 can be designed to gradually deflect light downward toward its associated EPE 2122, 2116, thereby extending the formed exit pupil horizontally. Each EPE 2122, 2116 can be configured to gradually redirect at least a portion of the light received from its corresponding OPE grating set 2120, 2114 outward toward a user eyebox position (not shown) defined behind the eyepieces 2108, 2110, thereby extending the formed exit pupil vertically toward the eyebox. Alternatively, instead of the internally coupled grating sets 2112 and 2118, OPE grating sets 2114 and 2120, and EPE grating sets 2116 and 2122, the eyepieces 2108 and 2110 may include other arrays of gratings and / or refractive and reflective features to control the coupling of image-modulated light to the user's eye.

[0051] In some embodiments, the wearable head device 2102 may include a left lance arm 2130 and a right lance arm 2132, the left lance arm 2130 including a left speaker 2134 and the right lance arm 2132 including a right speaker 2136. The orthogonal coil electromagnetic receiver 2138 may be located in the left lance component or in another preferred location within the wearable head unit 2102. The inertial measuring unit (IMU) 2140 may be located in the right lance arm 2132 or in another preferred location within the wearable head device 2102. The wearable head device 2102 may also include a left depth (e.g., time-of-flight) camera 2142 and a right depth camera 2144. The depth cameras 2142 and 2144 may preferably be oriented in different directions to cover a wider field of view together.

[0052] In the embodiment shown in Figures 2A-2D, the left source 2124 of the per-image modulated light can be optically coupled into the left eyepiece 2108 through the left internal coupling grating set 2112, and the right source 2126 of the per-image modulated light can be optically coupled into the right eyepiece 2110 through the right internal coupling grating set 2118. The per-image modulated light sources 2124 and 2126 may include, for example, an optical fiber scanning device, an electronic optical modulator including a digital light processing (DLP) chip or a liquid crystal on silicon (LCoS) modulator, a projector, or a light-emitting display such as a microlight-emitting diode (μLED) or microorganic light-emitting diode (μOLED) panel coupled into the internal coupling grating sets 2112 and 2118 using one or more lenses per side. The input coupling grid sets 2112 and 2118 can deflect the light from the image-modulated light sources 2124 and 2126 to an angle above the critical angle for total internal reflection (TIR) ​​for the eyepieces 2108 and 2110. The OPE grid sets 2114 and 2120 gradually deflect the propagating light downwards toward the EPE grid sets 2116 and 2122 by TIR. The EPE grid sets 2116 and 2122 gradually combine the light toward the user's face, including the pupil of the user's eye.

[0053] In some embodiments, as shown in Figure 2D, the left eyepiece 2108 and the right eyepiece 2110 each include a plurality of waveguides 2402. For example, each eyepiece 2108, 2110 may include a plurality of individual waveguides, each dedicated to a separate color channel (e.g., red, blue, and green). In some embodiments, each eyepiece 2108, 2110 may include a plurality of sets of such waveguides, each set configured to impart a different wavefront curvature to the emitted light. The wavefront curvature may be convex with respect to the user's eye, for example, to present a virtual object positioned at a certain distance in front of the user (e.g., only a distance corresponding to the reciprocal of the wavefront curvature). In some embodiments, the EPE grating sets 2116, 2122 may include curved grating grooves to produce a convex wavefront curvature by modifying the Poynting vector of the light emitting across each EPE.

[0054] In some embodiments, stereoscopically adjusted left and right eye images can be presented to the user through image-perfect optical modulators 2124, 2126 and eyepieces 2108, 2110 to create the perception that the displayed content is three-dimensional. The perceived reality of the presentation of the three-dimensional virtual object can be enhanced by selecting waveguides (and therefore corresponding wavefront curvatures) such that the virtual object is displayed at a distance approximating the distance indicated by the stereoscopic left and right images. This technique can also reduce motion sickness experienced by some users, which may be caused by the difference between the depth perception cues provided by the stereoscopic left and right eye images and the automatic near and far accommodation of the human eye (e.g., object distance-dependent focus).

[0055] Figure 2D illustrates a top-edge view of the right eyepiece 2110 of an exemplary wearable head device 2102. As shown in Figure 2D, the plurality of waveguides 2402 may include a first subset 2404 of three waveguides and a second subset 2406 of three waveguides. The two subsets of waveguides 2404 and 2406 can be distinguished by different EPE gratings, each featuring different grating line curvatures to impart different wavefront curvatures to the emitted light. Within each of the waveguide subsets 2404 and 2406, each waveguide can be used to couple different spectral channels (e.g., one of the red, green, and blue spectral channels) to the user's right eye 2206. (Although not shown in Figure 2D, the structure of the left eyepiece 2108 is similar to that of the right eyepiece 2110.)

[0056] Figure 3A illustrates an exemplary handheld controller component 300 of the mixed reality system 200. In some embodiments, the handheld controller 300 includes a gripping portion 346 and one or more buttons 350 positioned along the upper surface 348. In some embodiments, the buttons 350 may be configured for use as optical tracking targets for tracking the 6-degree-of-freedom (6DOF) motion of the handheld controller 300, in conjunction with, for example, a camera or other optical sensor (which may be mounted within the head unit of the mixed reality system 200 (e.g., a wearable head device 2102)). In some embodiments, the handheld controller 300 includes a tracking component (e.g., an IMU or other suitable sensor) for detecting position or orientation, such as position or orientation relative to the wearable head device 2102. In some embodiments, such a tracking component may be positioned within the handle of the handheld controller 300 and / or mechanically coupled to the handheld controller. The handheld controller 300 can be configured to provide one or more output signals corresponding to a button press state, or the position, orientation, and / or movement of the handheld controller 300 (e.g., via the IMU), or more than one of these. Such output signals may be used as inputs to the processor of the mixed reality system 200. Such inputs may correspond to the position, orientation, and / or movement of the handheld controller (by extension, the position, orientation, and / or movement of the user's hand holding the controller). Such inputs may also correspond to the user pressing button 350.

[0057] Figure 3B illustrates an exemplary auxiliary unit 320 of the mixed reality system 200. The auxiliary unit 320 may include a battery to provide energy and operate the system 200, and may include a processor to run programs and operate the system 200. As shown, the exemplary auxiliary unit 320 includes a clip 2128 for attaching the auxiliary unit 320 to the user's belt. It will also become apparent that other shape factors are suitable for the auxiliary unit 320 and may include shape factors that do not involve mounting the unit on the user's belt. In some embodiments, the auxiliary unit 320 is coupled to the wearable head device 2102 through a multi-tube cable, which may include, for example, electrical wires and optical fibers. Wireless connectivity between the auxiliary unit 320 and the wearable head device 2102 can also be used.

[0058] In some embodiments, the mixed reality system 200 may include one or more microphones that can detect sound and provide corresponding signals to the mixed reality system. In some embodiments, the microphones may be attached to or integrated with a wearable head device 2102 and may be configured to detect the user's voice. In some embodiments, the microphones may be attached to or integrated with a handheld controller 300 and / or an auxiliary unit 320. Such microphones may be configured to detect ambient sounds, background noise, the voice of the user or a third party, or other sounds.

[0059] Figure 4 shows an exemplary functional block diagram that may correspond to an exemplary mixed reality system, such as the mixed reality system 200 described above (which may correspond to the mixed reality system 112 relating to Figure 1). As shown in Figure 4, the exemplary handheld controller 400B (which may correspond to the handheld controller 300 ("Totem")) includes a totem / wearable head device 6-degree-of-freedom (6DOF) totem subsystem 404A, and the exemplary wearable head device 400A (which may correspond to the wearable head device 2102) includes a totem / wearable head device 6DOF subsystem 404B. In the embodiment, the 6DOF totem subsystem 404A and the 6DOF subsystem 404B cooperate to determine the six coordinates of the handheld controller 400B relative to the wearable head device 400A (e.g., offset in three translation directions and rotation along three axes). The six degrees of freedom may be expressed relative to the coordinate system of the wearable head device 400A. The three translation offsets may be represented as X, Y, and Z offsets in such a coordinate system, as a translation matrix, or as some other representation. The rotational degrees of freedom may be represented as a sequence of yaw, pitch, and roll rotations, as a rotation matrix, as a quaternion, or as some other representation. In some embodiments, a wearable head device 400A, one or more depth cameras 444 (and / or one or more non-depth cameras) contained within the wearable head device 400A, and / or one or more optical targets (e.g., a button 350 on a handheld controller 400B as described above, or a dedicated optical target contained within the handheld controller 400B) can be used for 6DOF tracking. In some embodiments, the handheld controller 400B may include a camera as described above, and the wearable head device 400A may include an optical target for optical tracking in conjunction with the camera.In some embodiments, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids, which are used to wirelessly transmit and receive three distinguishable signals. The 6DOF of the wearable head device 400A relative to the handheld controller 400B can be determined by measuring the relative magnitudes of the three distinguishable signals received in each of the coils used for receiving. In addition, the 6DOF totem subsystem 404A may include an inertial measurement unit (IMU), which is useful for providing improved accuracy and / or more timely information regarding the high-speed movement of the handheld controller 400B.

[0060] In some embodiments, the wearable system 400 may include a microphone array 407, which may include one or more microphones arranged on a headgear device 400A. In some embodiments, the microphone array 407 may include four microphones. Two microphones may be mounted on the front of the headgear 400A, and two microphones may be mounted on the rear of the headgear 400A (e.g., one on the left rear and one on the right rear). In some embodiments, the signals received by the microphone array 407 may be transmitted to a DSP 408. The DSP 408 may be configured to perform signal processing on the signals received from the microphone array 407. For example, the DSP 408 may be configured to perform noise reduction, acoustic echo cancellation, and / or beamforming on the signals received from the microphone array 407. The DSP 408 may be configured to transmit the signals to a processor 416.

[0061] In some embodiments, for example, it may be necessary to transform the coordinates from local coordinate space (e.g., a coordinate space fixed relative to the wearable head device 400A) to inertial coordinate space (e.g., a coordinate space fixed relative to the real environment) in order to compensate for the movement of the wearable head device 400A relative to coordinate system 108. For example, such a transformation may be necessary to ensure that the display of the wearable head device 400A presents virtual objects in their expected position and orientation relative to the real environment (e.g., a virtual person seated in a real chair facing forward, regardless of the position and orientation of the wearable head device), rather than in a fixed position and orientation on the display (e.g., in the same position at the lower right corner of the display), and to preserve the illusion that the virtual objects exist in the real environment (and do not appear unnaturally positioned in the real environment as the wearable head device 400A shifts and rotates). In some embodiments, compensatory transformations between coordinate spaces can be determined by processing images from a depth camera 444 using SLAM and / or visual odometry procedures to determine the transformation of the wearable head device 400A to coordinate system 108. In the embodiment shown in Figure 4, the depth camera 444 can be coupled to a SLAM / visual odometry block 406 and provide images to the block 406. The SLAM / visual odometry block 406 implementation may include a processor configured to process these images and then determine the position and orientation of the user's head, which can be used to identify transformations between the head coordinate space and another coordinate space (e.g., inertial coordinate space). Similarly, in some embodiments, an additional source of information regarding the user's head pose and location is obtained from an IMU 409. The information from the IMU 409 can be integrated with the information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information regarding faster adjustment of the user's head pose and position.

[0062] In some embodiments, the depth camera 444 can supply a 3D image to a hand gesture tracker 411, which may be implemented within the processor of the wearable head device 400A. The hand gesture tracker 411 can identify the user's hand gestures, for example, by matching the 3D image received from the depth camera 444 to a stored pattern representing the hand gesture. Other preferred techniques for identifying the user's hand gestures will also become apparent.

[0063] In some embodiments, one or more processors 416 may be configured to receive data from the wearable head device's 6DOF headgear subsystem 404B, IMU 409, SLAM / visual odometry block 406, depth camera 444, and / or hand gesture tracker 411. The processor 416 can also transmit and receive control signals from the 6DOF totem system 404A. The processor 416 may be coupled wirelessly to the 6DOF totem system 404A, for example, in embodiments where the handheld controller 400B is not tethered. The processor 416 may further communicate with additional components such as an audio / visual content memory 418, a graphical processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spatializer 422. The DSP audio spatializer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source 424 of light modulated for each image, and a right channel output coupled to a right source 426 of light modulated for each image. The GPU 420 can output stereoscopic image data to the source 424, 426 of light modulated for each image, for example, as described above with respect to Figure 2A-2D. The DSP audio spatialization device 422 can output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatialization device 422 may receive an input from the processor 419 indicating a direction vector from the user to a virtual sound source (e.g., which may be moved by the user via the handheld controller 320). Based on the direction vector, the DSP audio spatialization device 422 can determine the corresponding HRTF (e.g., by accessing the HRTF or by interpolating multiple HRTFs). The DSP audio spatialization device 422 can then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object.This can improve the credibility and realism of virtual sounds by incorporating the user's relative position and orientation to virtual sounds within a mixed reality environment; that is, by presenting virtual sounds that match the user's expectations of what they would hear if they were real sounds in a real environment.

[0064] In some embodiments, such as those shown in Figure 4, one or more of the processor 416, GPU 420, DSP audio spatialization device 422, HRTF memory 425, and audio / visual content memory 418 may be included in an auxiliary unit 400C (which may correspond to the auxiliary unit 320 described above). The auxiliary unit 400C may include a battery 427 that powers its components and / or supplies power to the wearable head device 400A or the handheld controller 400B. Including such components in an auxiliary unit that can be mounted on the user's waist can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.

[0065] Figure 4 presents elements corresponding to various components of an exemplary mixed reality system, but various other preferred arrangements of these components will also be apparent to those skilled in the art. For example, the elements shown in Figure 4 as associated with the auxiliary unit 400C may instead be associated with a wearable head device 400A or a handheld controller 400B. Furthermore, some mixed reality systems may omit the handheld controller 400B or the auxiliary unit 400C entirely. Such changes and modifications are understood to be included within the scope of the disclosed embodiments. Exemplary optical systems

[0066] The wearable head device of the exemplary mixed reality system (e.g., mixed reality system 200) may include an optical system for presenting images to the user via a display. Figures 5-7 and 8A-8C illustrate embodiments of optical systems that may be used in a wearable head device (e.g., wearable head device 2102).

[0067] Figure 5 illustrates an exemplary optical system 500 that may be used in a wearable head device (e.g., wearable head device 2102). The optical system 500 may include a light source including a plurality of LEDs 501a-c and a prism 507, an image generator 505, an angled partially transparent and partially reflective surface 509, and a projector lens assembly 503. In some embodiments, the image generator may include a spatial light modulator such as LcoS (liquid crystal on silicon).

[0068] As shown in the figure, the plurality of LEDs 501a-c may include at least one of red LED 501a, green LED 501b, and blue LED 501c. The plurality of LEDs may be arranged to project light into the prism 507. In some embodiments, the prism 507 may be an x-cube configured to combine the light from each of the LEDs 501a-c and output light 530 to the image generator 505. The prism 507 may introduce some inefficiency if some of the light incident on the prism may be lost, for example, scattered. In addition, the x-cube typically has image artifacts associated with the intersection of various prisms. In some embodiments, the light 530 may pass through a partially transparent, partially reflective surface 509 before being received by the image generator 505. As shown in the figure, the light source, for example, multiple LEDs 501a-c and a prism 507, is positioned such that the light 530 emitted by the prism 507 is on axis with respect to axis 535 (i.e., the light 530 is aligned along axis 535) and coaxial with the image generator 505.

[0069] The image generator 505 can receive light 530, reflect it, and form image light 533. The image light 533 output by the image generator 533 can be reflected toward the projector lens assembly 503 by a partially transparent, partially reflective surface 509. Some of the light 530 and image light 533 that passes through and / or is reflected by the partially transparent, partially reflective surface 509 may be lost, for example, scattered or transmitted toward the lens 503. The eyepiece of a head-wearable device may be configured to receive the image light 533 and present it to the user. In some embodiments, the field of view (FOV) of the optical system 500 may be 55 degrees. The 55-degree FOV of the system 500 may be based on the focal length of the lens assembly 503. As shown in Figure 5, the width (and correspondingly, the focal length) of the lens assembly 503 may be limited by the size of the image generator 505 and the light source 501, for example, the area occupied. In some embodiments, lens 503 may have a shorter focal length, which can increase the field of view (FOV). However, the available space for shortening the focal length, e.g., the occupied area, is limited by prisms or beam splitters. In practice, this may limit the achievable field of view to 55 degrees.

[0070] Figure 6 illustrates an exemplary optical system 600 that may be used in a wearable head device (e.g., wearable head device 2102). The optical system 600 may include a light source, including an RGB LED array 601; an image generator 605, a spatial light modulator such as an LCoS; an angled partially transparent and partially reflective surface 609; and a projector lens assembly 603. The optical system 600 may be more compact than the optical system 500. For example, the optical system 600 does not need to include a prism to combine light from the light source, thereby saving physical space.

[0071] As shown in the figure, the light source 601 may include an RGB LED array that emits light 630 to the image generator 605. In some embodiments, the light 630 may pass through a partially transparent, partially reflective surface 609 before being received by the image generator 605. A portion of the light 630 and image light 633 that passes through and / or is reflected by the partially transparent, partially reflective surface 609 may be lost, for example, scattered. In embodiments, the light source 601 and the image generator 605 are located on the same axis (635) so that light from the light source 601 is delivered to the image generator 605 along the main axis. The image light 633 output by the image generator 633 may be reflected by the partially transparent, partially reflective surface 609 toward the projector lens assembly 603. The eyepiece of the head-wearable device may be configured to receive the image light 633 and present it to the user. In some embodiments, the field of view (FOV) of the optical system 600 may be up to, for example, 55 degrees.

[0072] Figure 7 shows top and bottom views of an exemplary optical system 700 that may be used in a wearable head device (e.g., wearable head device 2102). As shown in this figure, the optical system 700 may include, for example, a light source 701, a lens assembly 703, an image generator 705, a spatial light modulator such as an LCoS, a color filter 719, and a plurality of waveguides 711. In some embodiments, the optical system 700 may further include a polarizer positioned between the light source 701 and the plurality of waveguides 711.

[0073] Multiple waveguides 711 may be arranged in parallel to form a waveguide stack 710. The waveguide stack 710 may be configured such that a first surface 723 of the waveguide stack 710 is exposed to the real-world environment 725, and a second surface 727 of the stack faces distal to the real-world environment 725, for example, the eyes 720 of a user wearing a wearable head device. The optical system 700 may also include a circular polarizer 717 and an optical element stack 707 (referred to here as “MSP”) which may include a polarizer (e.g., a linear polarizer) between a first quarter plate and a second quarter plate.

[0074] The light source 701 may include one or more LED sources 701a-c. In some embodiments, the LED sources may include LEDs placed in a reflector, for example, a CPC reflector. As shown in the figure, the light source 701 includes three separate LED sources 701a-c. The three LED sources may each correspond to a primary color, red, green, and blue. In some embodiments, each LED source may have an exit pupil or exit aperture of about 1 mm. The light source 701 may be located on the first surface 723 of the waveguide stack 710. The light source may be located near the temple area of ​​the head wearable device (e.g., as shown in Figure 2A). The light source 701 may be configured to direct light 730 through the waveguide stack 710 and toward the lens assembly 703 and the image generator 705. Compared to the exemplary optical systems 500 and 600, the light source 701 and image generator 705 shown are off-axis; for example, the axes of the light source 701 and image generator 705 are not aligned.

[0075] Light 730, generated by the light source 701, for example by LED sources 701a-c, reaches the image generator 705 by passing through a plurality of waveguides 711 and a lens assembly 703. In some embodiments, the optical system 700 may be arranged so that light from the light source 701 is transmitted through the plurality of waveguides 711 but not through the color filter 719 and / or polarizer 717. Light 730 can pass through the color filter 719 and / or polarizer 717 after being reflected from the image generator 705, as will be discussed in more detail below. The lens assembly 703 may be configured to direct the light 730 towards the image generator 705. In some embodiments, an MSP 707 may be located between the lens assembly 703 and the image generator 705. Thus, light 730 can pass through the MSP 707 before being incident on the image generator 705. The optical system 700 shown in Figure 7 may have a relatively large field of view (compared to, for example, optical systems 500 and 600) because large optical elements such as prisms, beam splitters, and waveguides, which are positioned between the lens and the image generator, can be omitted.

[0076] The image generator 705 can reflect the incident light 730 to generate image light 733. In some embodiments, the image generator may be a spatial light modulator such as an LCoS. The image light 733 may pass through the MSP 707 as it exits the image generator 705. In some embodiments, the MSP 807 may be tilted to minimize the effect of reflection in the system on the image presented to the user. The image light 733 may then be incident on the lens assembly 703, where it is directed toward the waveguide stack 710. As it reaches the second surface 727 of the waveguide stack 710, the image light 733 may pass through a circular polarizer 717 and / or a color filter 719. The circular polarizer 717 can polarize the image light 733, while the color filter can filter out wavelengths that do not correspond to the wavelength of light output by the light source 701. In some embodiments, the optical system 700 may not include a color filter 719, and for example, the light 730 may be selectively internally coupled into the waveguide stack via an internal coupling grating.

[0077] The internal coupling gratings 715a-c can then receive the filtered light. Each internal coupling grating may be configured to diffract or internally couple light of a specific wavelength into the corresponding waveguide. In other words, each of the LED sources 701a-c can output light at a specific wavelength, and the corresponding color filter 719 and internal coupling grating 715 can be tuned to the same wavelength. For example, the color filter 719a can filter out wavelengths that do not correspond to the wavelength output by the light source 701a, and the internal coupling grating 715a can be configured to diffract light corresponding to the wavelength output by the LED source 701a.

[0078] As shown in Figure 7, the LED sources 701a-c of the optical system 700 are spatially separated. The LED sources 701a-c can be spatially separated in both the horizontal and vertical directions. Similarly, the internally coupled gratings 715a-c can be spatially separated in both the horizontal and vertical directions. Therefore, light emitted from each LED source can travel along different paths and reach the individual internally coupled gratings 715a-c. For example, light emitted from LED source 701a can be spatially separated (vertically and horizontally) from light emitted from LED source 701b. Also, the image light received by internally coupled grating 715a, corresponding to the light emitted from LED source 701a, can be spatially separated (vertically and horizontally) from the light received by internally coupled grating 715b, corresponding to the light emitted from LED source 701b. This spatial separation of the light sources and internally coupled gratings can limit color crosstalk.

[0079] As illustrated in Figure 7, the optical system 700 can offer a relatively compact configuration compared to, for example, the optical systems 500 and 600. As discussed above, in the shown embodiment, the light source 701 and the image generator 705 are positioned off-axis from each other, which allows the lens assembly 703 to direct light to and from the image generator 705. Thus, the lens assembly can be adjusted to have a shorter focal length and therefore a wider field of view, and the image generator can be positioned on the rear surface of the lens assembly. This can reduce the overall depth of the optical system 700 compared to, for example, the optical systems 500 and 600. Furthermore, the field of view of the optical system 700 can be larger, for example, 70 to 100 degrees. In addition, the optical system 700 may be more efficient because less light is lost to various optical components, such as prisms and partially transparent and partially reflective surfaces.

[0080] Figures 8A–8C illustrate embodiments of the optical system 800 that may be used in a wearable head device (e.g., wearable head device 2102) according to embodiments of the present disclosure. Figure 8A shows top and bottom views of the optical system 800 that may be used in a wearable head device according to embodiments of the present disclosure. Figure 8B shows a perspective view of the waveguide stack of the optical system 800 that may be used in a wearable head device according to embodiments of the present disclosure. Figure 8C shows another view of the optical system that may be used in a wearable head device according to embodiments of the present disclosure. The optical system 800 can be used to output light and / or images, for example, to the user of a wearable head device. As will be understood by those skilled in the art, these figures each show an optical system for monocular vision. The system may be replicated or otherwise modified for binocular vision.

[0081] Figure 8A shows top and bottom views of an optical system 800 that may be used in a wearable head device according to an embodiment of the present disclosure. The optical system 800 may include, for example, a light source 801, a lens assembly 803, an image generator 805, and a plurality of waveguides 811. The plurality of waveguides 811 may be arranged in parallel to form a waveguide stack 810. The waveguide stack 810 may be configured such that a first surface 823 of the waveguide stack 810 is exposed to the real-world environment 825, and a second surface 827 of the stack faces distal to the real-world environment 825, for example, the eyes 820 of a user wearing the wearable head device. The optical system 800 may also include a circular polarizer 817 and an MSP 807. The circular polarizer may be located on the second surface 824 of the waveguide stack 810. The MSP807 may be located between the lens assembly 803 and the LCOS805. In some embodiments, the optical system 800 may include a color filter (not shown). In some embodiments, the waveguides 811 may be trimmed so that light 830 from the light source 801 does not pass through the waveguides 811, or passes through one or more of the waveguides, but not all of them, on its path to the image generator 805. This configuration can reduce light loss. In some embodiments, the light source may further include a polarizer between the exit pupil of the CPC reflector and the waveguide.

[0082] The optical system 800 may include a single light source 801. In some embodiments, the light source 801 may include a single CPC reflector with a spatial light modulator, such as a micro-LED array, positioned inside. The micro-LED array may be positioned across the bottom of the CPC reflector or inside the entrance pupil within the CPC reflector. In some embodiments, the reflector may have an exit pupil diameter of about 1 to 1.3 mm. The micro-LED array may be an RGB micro-LED array containing a plurality of red, green, and blue LEDs. The micro-LED array may be configured to sequentially turn on and off all LEDs corresponding to specific colors. For example, the micro-LEDs may be configured to turn on and off all green LEDs simultaneously, all red LEDs simultaneously, and all blue LEDs simultaneously, in a pattern that forms a sequential RGB light source. The CPC reflector can collect and shape the light from the micro-LED source, which typically has a Lambertsian or approximate Lambertsian angular distribution. Thus, the CPC reflector can form an exit pupil size optimized for the optical system and ICG. The CPC reflector can also provide light within an optimal angular range corresponding to the system's FOV. Therefore, the light source 801 can provide a uniform and efficient light source in a small, for example, single-source package. Additional explanations related to micro-LED arrays are provided below. The light source may be positioned near the temple area of ​​the head-mounted wearable display, such as shown in Figure 2A. In some embodiments, the light source may be positioned on other parts of the wearable display.

[0083] The waveguide stack 810 of the optical system 800 may include a plurality of waveguides 811, each containing an internal coupling grating 815 and an external coupling grating 813. Referring to Figure 8B, in some embodiments, three or more of the waveguides 811a-c may include internal coupling gratings 815a-c and external coupling gratings 813a-c. In some embodiments, each waveguide 811a-c may further include orthogonal pupil expanders ("OPE") 814a-c configured to propagate light 833a-c across the waveguide 811a-c and toward the corresponding external coupling grating 813a-c. As shown in the figure, the internal coupling gratings 815a-c may be spatially aligned vertically and horizontally, for example, overlapping.

[0084] In some embodiments, the internally coupled gratings 815a-c can each selectively diffract light 833a-c of a specific wavelength into the corresponding waveguides 811a-c. For example, each image light 833a-c may correspond to a different wavelength; for instance, light 833a may correspond to a red wavelength, light 833b to a green wavelength, and light 833c to a blue wavelength. Thus, the internally coupled grating 815a may be sensitive to the red wavelength of light 833a, the internally coupled grating 815b to be sensitive to the green wavelength of light 833b, and the internally coupled grating 815c to be sensitive to the blue wavelength of light 833c. For example, light 833a may correspond to a red light wavelength, and the internally coupled grating 815a may be configured to selectively diffract light 833a corresponding to the red light wavelength. When the internal coupling grating 815a receives light 833b or 833c, the light can pass through the internal coupling grating 815a without being diffracted into the waveguide 811a. The configuration described herein is illustrative, and those skilled in the art will recognize that any of the internal coupling gratings can correspond to any preferred wavelength. OPE 814 can propagate the internally coupled light 835 horizontally to the external coupling grating 813 by total internal reflection (TIR), so that the internally coupled light 835 can be directed out of the waveguide stack 810 and toward the user's eye 820. In addition to selective color internal coupling gratings, the waveguide 811 can be formed to absorb undesirable colors in its particular waveguide in order to further limit color crosstalk.

[0085] Compared to the internally coupled grids 715a-c illustrated in Figure 7, the internally coupled grids 815 may be spatially overlapping, for example, arranged to be aligned. In some cases, when the internally coupled grids overlap, the size of the projector and the overall optical system can be reduced. Furthermore, a single pupil light source may enable simpler and less expensive alignment and testing equipment.

[0086] The optical system 800 may function in a manner similar to that described above with respect to the optical system 700. Referring to Figure 8C, the light source 801 may be positioned on the first surface 823 of the waveguide stack 810. The light source 801 may be configured to direct light 830 through the waveguide stack 810 and toward the lens assembly 803 and the image generator 805. The light source 801 and the image generator 805 may be configured to be off-axis, for example, so that the axes of the light source 801 and the image generator 805 are not aligned. Light 830 passes through the lens assembly 803 so that the image generator 805 receives the light 830 generated by the light source 801. The lens assembly 803 can be configured to direct the light 830 toward the image generator 805. In some embodiments, the light generator 805 may be positioned on the rear surface of the lens assembly 803. In some embodiments, the MSP 807 may be located between the lens assembly 803 and the image generator 805. Thus, light 830 can pass through the MSP 807 before entering the image generator 805. In some embodiments, the MSP 807 may be tilted to minimize the effect of reflection in the system on the image presented to the user.

[0087] The image generator 805 can reflect the incident light 830 to generate image light 833. The image light 833 may pass through the MSP 807 as it exits the image generator 805. The image light 833 may then be incident on the lens assembly 803, where the light 833 can be directed toward the waveguide stack 810. As it reaches the second surface 827 of the waveguide stack 810, the image light 833 may pass through the circular polarizer 817 before being incident on the second surface 827 of the waveguide stack 810.

[0088] Image light 833 received by the waveguide stack 810 can be diffracted by one of the internal coupling gratings 815. For example, if image light 833 corresponds to a wavelength of the color red, image light 833 can be diffracted by an internal coupling grating 815 configured to diffract light corresponding to the same wavelength. The internal coupling light 835 can then propagate along the waveguide 811 via the TIR. The internal coupling light 835 can exit the waveguide 811 via the external coupling grating 813 and be presented to the user's eye 820. In some embodiments, an OPE may be included within the waveguide stack 810, as described with respect to Figure 8B.

[0089] As discussed above with respect to optical system 700, optical system 800 can provide a relatively compact configuration that delivers a uniform image with high processing power, e.g., efficiency. For example, since the light source 801, as shown, is off-axis from the image generator 805, the lens assembly 803 can be used to direct the light towards the image generator 805. As shown, the image generator 805 can be positioned behind the lens assembly 803, which reduces the overall depth of optical system 800 compared to, for example, optical systems 500 and 600. In addition, optical system 800 may be more efficient because less light is lost to various optical components, e.g., prisms and partially transparent and partially reflective surfaces. Furthermore, due to the reduced occupancy area of ​​the light source 801 and the spatially aligned internal coupling grating 815, optical system 800 may be smaller than optical system 700.

[0090] Figure 9 illustrates a reflector 950 according to an embodiment of the present disclosure. The reflector 950 may include an optical input aperture 951, an optical output aperture 959, and a side surface 953. As shown in the figure, the reflector 950 may have an outline corresponding to a composite parabolic condenser (CPC). For example, the side surface 953 of the reflector 950 may correspond to the outline of a CPC reflector. In some embodiments, the optical input aperture 951 and the optical output aperture 959 may be circular, elliptical, or rectangular. The optical input aperture may be configured to receive light (e.g., rays 931, 932, 933) from an optical emitter (not shown). The light can be reflected from the side surface 953 of the reflector and exit the reflector 950 through the optical output aperture 959. In this way, the reflector 950 collects edge rays. The spatial uniformity of the output light may be designed so that the distribution of light across the image generator is optimal for image quality. It is desirable that the reflector exhibits improved spatial uniformity. Thus, the CPC may be designed to control the size of the illumination and the angular distribution of light while increasing the efficiency of the system.

[0091] Figure 10 illustrates a light source 1001 according to an embodiment of the present disclosure. The light source 1001 may be a single pupil RGB light source (which may correspond to light source 801). As shown in this figure, the light source 1001 may include a reflector 1050 and a micro LED array 1040.

[0092] The reflector 1050 may have a CPC shape, such as that discussed with respect to the reflector 950 in Figure 9. In some embodiments, the side surface 1053 may have an elliptical, hyperbolic, or biconical shape. The shape of the side surface 1053 may be configured to provide an angularly controlled light output at the exit of the reflector 1050.

[0093] In some embodiments, the micro-LED array 1040 may be positioned across the light input aperture 1051. Although the micro-LED array 1040 is shown outside the reflector 1050, those skilled in the art will understand that the micro-LED array can be positioned inside the reflector 1040 at the light input aperture 1051. As described above, the light produced by the micro-LED array 1040 may include light in three primary colors: blue (B), green (G), and red (R). In other words, the micro-LED array 1040 may be an RGB micro-LED array containing multiple red, green, and blue LEDs (not shown individually). The multiple red, green, and blue LEDs may each be arranged to be uniformly distributed across the bottom aperture of the reflector 1050. In some embodiments, if more refractive power for a particular wavelength is desired, the number of micro-LEDs corresponding to the desired wavelength may be increased, and / or the number of micro-LEDs corresponding to the desired wavelength may be greater than the number of micro-LEDs corresponding to other wavelengths.

[0094] A substantially uniform distribution of LEDs across the light input aperture 1051 can reduce or mitigate the formation of hotspots in the light emitted from the light source 1001. For example, to achieve a single pupil RGB light source, three LEDs, namely a red LED, a blue LED, and a green LED, can be placed together in the reflector. However, this configuration results in substantial non-uniformity of the light emitted from the light output aperture 1059, e.g., hotspots. These hotspots in the light leaving the light source 1001 can, in turn, provide non-uniform illumination across the LCOS, which results in color localization in the image light. In a wearable head device, hotspots may appear to the user as areas on the display that are oversaturated with a particular color. For example, a display that is visible to the user may have areas that are oversaturated with a blue hue. In another embodiment, if the display is supposed to present the user with a true white image, a first area of ​​the image may appear blue, a second area of ​​the image may appear green, and a third area of ​​the image may appear red. The localization of colors visible on the display is undesirable and can detract from the user's XR experience. It may be desirable for the display to exhibit greater uniformity of light.

[0095] To improve the uniformity of light, a light source may include a diffuser to increase the diffusion of light emitted from the light source, for example, etendue. However, diffusers can be inefficient by scattering light from the light source and reducing the maximum brightness of the display. Due to the substantially uniform distribution of multiple micro-LEDs, light source 1001 can provide a uniform light output without an inefficient diffuser. Thus, light source 1001 can provide a small, for example, single-pupil package that is both uniform and efficient. For example, compared to systems 700 and 800 described above, those skilled in the art will understand that the three-pupil optical system 700 is likely to be larger than the single-pupil optical system 800. Furthermore, as discussed above, including RGB LED emitters within a single reflector results in a relatively large, inefficient reflector that provides non-uniform light. In comparison, optical system 800 can provide an efficient and uniform light source in a small single-pupil package without relying on diffusers or other techniques that may result in light loss. Exemplary micro-LED array

[0096] A light source, for example, light source 801 or 1001, may include a micro-LED array. The micro-LED array may include multiple micro-LEDs arranged within a panel. Each micro-LED may form a pixel in the micro-LED array. The micro-LED array can have several different configurations. For example, when designing a micro-LED array, the size and shape of each micro-LED and the pattern or configuration of the micro-LEDs within the array can be varied. For example, the micro-LEDs may have a circular, rectangular, hexagonal, or radial shape, or any other preferred shape. Furthermore, the micro-LEDs can be arranged in various configurations, including linear, straight, or hexagonal arrays.

[0097] As discussed above, the micro-LED array may be placed in a reflector, such as a CPC reflector, to form a light source, such as light source 801 or 1001. The shape and relative configuration of the micro-LEDs may affect the illuminance of the light emitted from the light source, such as light 830. This, in turn, may affect the illuminance of the light received by the image generator 805, and ultimately, the light presented to the user. The light emitted from the light source may correspond to light, such as light 830, that enters an optical system, such as optical system 800, through a single pupil of the light source, such as light source 801. As discussed above, this light can be reflected by the image generator, such as image generator 805, to form image light to be presented to the user. Therefore, changing the size, shape, and configuration of the micro-LEDs will affect the illuminance of the light emitted from the pupil of the light source, such as light source 801, and consequently, the light presented to the user. Individual micro-LED emitters may be identical in size or vary in size. The size of individual micro-LED emitters can range from 1 μm to 100 μm. The number of micro-LED emitters may be determined by the size of the entrance pupil of the CPC reflector. The entrance pupil size may be related to the exit pupil of the CPC reflector. In some embodiments, the exit pupil size of the CPC reflector may be based on the size and configuration of the optical system, for example, the exit pupil size may be selected to optimize efficiency and modulation transfer function (MTF). In some embodiments, the filling density of the array may be as high as possible to maximize the achievable refractive power output.

[0098] Figure 11A-18I illustrates various configurations of a micro-LED array according to embodiments of the present disclosure. Embodiments of the present disclosure are not limited to the micro-LED array configurations disclosed herein, but rather these embodiments are merely illustrative examples of the types of micro-LED array configurations according to embodiments of the present disclosure.

[0099] Figure 11A illustrates a layout of a micro-LED array 1100 according to an embodiment of the present disclosure. As shown in this figure, the micro-LED array 1100 may be rectangular in shape, and the micro-LED sources may be rectangular in shape with a height and width of approximately 30 μm. Although this figure illustrates a single-color micro-LED, those skilled in the art will understand that the micro-LED array may include additional micro-LED sources corresponding to at least two additional colors. Figure 11B illustrates the illuminance resulting from, for example, the image light reflected therefrom, across an image generator, based on the configuration of an optical system 800 including the micro-LED array 1100.

[0100] Figure 12A illustrates a layout of a micro-LED array 1200 according to an embodiment of the present disclosure. As shown in this figure, the micro-LED array 1200 may be rectangular in shape, and the micro-LEDs may be rectangular in shape with a width of about 50 μm. This figure illustrates a single-color micro-LED, but those skilled in the art will understand that the micro-LED array may include additional micro-LED sources corresponding to at least two additional colors. Figure 12B illustrates the illuminance resulting from, for example, the image light reflected therefrom, across an image generator, based on the configuration of the optical system 800 including the micro-LED array 1200.

[0101] Figure 13A illustrates a layout of a micro-LED array 1300 according to an embodiment of the present disclosure. As shown in this figure, the micro-LED array 1300 may be rectangular in shape, and the micro-LEDs may be rectangular in shape with a width and height of approximately 40 μm. This figure illustrates a single-color micro-LED source, but those skilled in the art will understand that the micro-LED array may include additional micro-LED sources corresponding to at least two additional colors. Figure 13B illustrates the illuminance resulting from, for example, the image light reflected therefrom, across an image generator, based on the configuration of an optical system 800 including the micro-LED array 1300.

[0102] Figure 14A illustrates a layout of a micro-LED array 1400 according to an embodiment of the present disclosure. As shown in this figure, the micro-LED array 1400 may be hexagonal in shape, and the micro-LEDs may be rectangular in shape with a width of about 30 μm and a height of about 26 μm, although other sizes, e.g., 25 to 50 μm, may be used. This figure illustrates a single-color micro-LED source, but those skilled in the art will understand that the micro-LED array may include additional micro-LED sources corresponding to at least two additional colors. Figure 14B illustrates the illuminance resulting from, for example, the image light reflected therefrom, across an image generator, based on the configuration of an optical system 800 including the micro-LED array 1400.

[0103] Figure 15A illustrates a layout of a micro-LED array 1500 according to an embodiment of the present disclosure. As shown in this figure, the micro-LED array 1500 may have a radial configuration, and the micro-LEDs may be rectangular in shape with a width and height of approximately 40 μm, although other sizes, e.g., 25-50 μm, may be used. This figure illustrates a configuration of a micro-LED source including three colors. Figure 15B illustrates the illuminance resulting from, for example, the image light reflected therefrom, traversing an image generator, based on the configuration of an optical system 800 including the micro-LED array 1500. Figure 15C illustrates a layout of the micro-LED array 1500 with respect to a first color of the three colors. The figure illustrates the illuminance of the light source, e.g., the light exiting the exit pupil of a reflector, for all micro-LED sources corresponding to the first color emitting simultaneously. (Unless otherwise stated, please understand that the initial emitted power and / or illuminance of each micro-LED source may be identical.) Figure 15D illustrates the illuminance of image light reflected by an image generator of an optical system with a single color shown in Figure 15C. Figure 15E illustrates the illuminance of light emitted from the pupil of a reflector with a micro-LED array 1500 for three colors of the second color, which is simultaneously emitted by, for example, a micro-LED source corresponding to the second color. Figure 15F illustrates the illuminance of image light reflected by an image generator of an optical system with a single color shown in Figure 15E. Figure 15G illustrates the illuminance of light emitted from the pupil of a reflector with a micro-LED array 1500 for three colors of the third color, which is simultaneously emitted by, for example, a micro-LED source corresponding to the third color. Figure 15H illustrates the illuminance of image light reflected by an image generator of an optical system with a single color shown in Figure 15G.

[0104] Figure 16A illustrates a layout of a microLED array 1600 according to an embodiment of the present disclosure. As shown in this figure, the microLED array 1600 may have a hexagonal configuration, and the microLEDs may be rectangular in shape with a width and height of 25 to 50 μm. The microLED array 1600 may include microLED sources corresponding to primary colors, red, green, and blue. This figure illustrates the illuminance of all microLED sources in the array 1600, for example, corresponding to all three colors, as measured at the exit pupil of a light source, e.g., a reflector. Figure 16B illustrates the illuminance of image light reflected by an image generator of an optical system with a microLED array 1600. Figure 16C illustrates a layout of the microLED array 1600 with respect to a first color of the three colors. The figure illustrates the illuminance of light from the microLED array. (Unless otherwise stated, please understand that the initial emitted power and / or illuminance of each micro-LED source may be identical.) Figure 16D illustrates the illuminance of image light reflected by an image generator of an optical system with a single color shown in Figure 16C. Figure 16E illustrates the illuminance of light emitted from the pupil of a reflector with a micro-LED array 1600 for three colors of the second color, which is simultaneously emitted by, for example, a micro-LED source corresponding to the second color. Figure 16F illustrates the illuminance of image light reflected by an image generator of an optical system with a single color shown in Figure 16E. Figure 16G illustrates the illuminance of light emitted from the pupil of a reflector with a micro-LED array 1600 for three colors of the third color, which is simultaneously emitted by, for example, a micro-LED source corresponding to the third color. Figure 16H illustrates the illuminance of image light reflected by an image generator of an optical system with a single color shown in Figure 16G.

[0105] Figure 17A illustrates the layout of a micro-LED array 1700 according to an embodiment of the present disclosure. As shown in this figure, the micro-LED array 1700 may have a hexagonal configuration, and the micro-LEDs may be circular in shape with a diameter of 25 to 50 μm. The micro-LED array 1700 may include micro-LED sources corresponding to primary colors, red, green, and blue. This figure illustrates, for example, the illuminance of all micro-LED sources in the array 1700 corresponding to all three colors. In some embodiments, all micro-LEDs in the array may include micro-LEDs corresponding to three colors, for example, red, green, and blue. Figure 17B illustrates the illuminance of the image light reflected by the image generator of the optical system with the micro-LED array 1700. Figure 17C illustrates the output of a true color detector receiving the output of the image generator when all micro-LED sources of the micro-LED array 1700 are emitting light. Figure 17D illustrates the layout of a micro-LED array 1500 with respect to a first color of the three colors. The figures illustrate the illuminance of the light source, e.g., the light exiting the exit pupil of a reflector, for all micro-LED sources corresponding to the first color being emitted simultaneously. (Unless otherwise stated, it should be understood that the initial emission power and / or illuminance of each micro-LED source may be identical.) Figure 17E illustrates the illuminance of the image light reflected by the image generator of the optical system with a single color shown in Figure 17D. Figure 17F illustrates the illuminance of the light exiting the pupil of a reflector with a micro-LED array 1700 corresponding to three colors of the second color, emitted simultaneously by, for example, a micro-LED source corresponding to the second color. Figure 17G illustrates the illuminance of the image light reflected by the image generator of the optical system with a single color shown in Figure 17F. Figure 17H ​​illustrates the illuminance of the light exiting the pupil of a reflector with a micro-LED array 1700 corresponding to three colors of the third color, emitted simultaneously by, for example, a micro-LED source corresponding to the third color. Figure 17I illustrates the illuminance of the image light reflected by the image generator of the optical system with a single color shown in Figure 17H.

[0106] Figure 18A illustrates a layout of a micro-LED array 1800 according to an embodiment of the present disclosure. As shown in this figure, the micro-LED array 1800 may have a hexagonal configuration, and the micro-LEDs may be rectangular in shape with a width of 25 to 50 μm. The micro-LED array 1800 may include micro-LEDs corresponding to primary colors, red, green, and blue. This figure illustrates the illuminance of all micro-LEDs in the array 1800, for example, corresponding to all three colors, as measured at a light source, for example, the exit pupil of a reflector. Figure 18B illustrates the illuminance of image light reflected by an image generator of an optical system with a micro-LED array 1800. Figure 18D illustrates a layout of the micro-LED array 1800 with respect to a first color of three colors. The figure illustrates the illuminance of light exiting a light source, for example, the exit pupil of a reflector, for all micro-LED sources corresponding to the first color that are simultaneously emitted. (Unless otherwise stated, please understand that the initial emission power and / or illuminance of each micro-LED source may be identical.) Figure 18D illustrates the layout of the micro-LED array 1500 for a first color of three colors. The figure illustrates the illuminance of the light source for all micro-LED sources corresponding to the first color, e.g., the light exiting the exit pupil of the reflector. (Unless otherwise stated, please understand that the initial emission power and / or illuminance of each micro-LED source may be identical.) Figure 18E illustrates the illuminance of the image light reflected by the image generator of the optical system with a single color shown in Figure 18D. Figure 18F illustrates the illuminance of the light exiting from the pupil of the reflector with a micro-LED array 1800 for a second color of three colors, emanating simultaneously from the micro-LED sources corresponding to the second color, e.g., the micro-LED sources corresponding to the second color. Figure 18G illustrates the illuminance of the image light reflected by the image generator of the optical system with a single color shown in Figure 18F. Figure 18H illustrates the illuminance of light emitted from the pupil of a reflector with a micro-LED array 1800 for three colors, where a micro-LED source corresponding to the third color is simultaneously emitting light. Figure 18I illustrates the illuminance of image light reflected by an image generator in an optical system with a single color, as shown in Figure 18H.

[0107] In some embodiments, the micro-LED array may use dithering to reduce the appearance of quantization patterns in the output of the light source's exit pupil and increase the visual uniformity of the output across the image generator. Figures 19A–19H illustrate exemplary micro-LED arrays, including slight dithering of the micro-LED elements from their central positions. Figure 19A illustrates the illuminance of all micro-LEDs contained within the array, measured at the light source's exit pupil, e.g., the exit aperture of a reflector. In some embodiments, the micro-LED elements may have a dither of ±5 μm. Figure 19A illustrates dithering, where the spacing of the micro-LED elements appears irregular, for example, compared to Figure 17D. Figure 19B illustrates the illuminance of the image light reflected by the image generator of an optical system with the micro-LED array 1900. Figure 19C illustrates the illuminance of the light exiting from the pupil of a reflector with the micro-LED array 1900 for three single colors, e.g., micro-LED sources corresponding to a single color are simultaneously emitting light. Figure 19D illustrates the illuminance of image light reflected by an image generator of an optical system with a single color, as shown in Figure 19C. Figure 19E illustrates a dithering configuration in which each column of a micro-LED array is offset by a different amount, and elements within the same column may be offset by the same amount. For example, the column offset may be 7 μm, 14 μm, or 21 μm. Figure 19F illustrates the illuminance of image light reflected by an image generator of an optical system with a single color, as shown in Figure 19E. Figure 19G illustrates a dithering configuration in which four offset values ​​are repeated. As shown in the figure, the offset may include a set of four columns with offsets of -14 μm, -5 μm, 4 μm, and 13 μm, so that this offset configuration can be repeated. Figure 19H illustrates the illuminance of image light reflected by an image generator of an optical system with a single color, as shown in Figure 19G.

[0108] Figure 20 illustrates a layout of a micro-LED array 2000 according to an embodiment of the present disclosure. As shown in the figure, the micro-LED array 2000 may have a striped configuration such that each micro-LED has multiple stripes extending traversing the micro-LED array from end to end. For example, the micro-LED array may include multiple vertical stripes, the length of which can vary based on the location of the stripes along the x-axis of the micro-LED array. In some embodiments, the stripes may be rectangular in shape with a width of about 10 to 40 μm. The spacing between the stripes may be about 10 μm or less. In some embodiments, the micro-LED array may have a diameter of about 0.7 to 1.00 mm. In some embodiments, the stripes may be arranged horizontally in columns such that the length of which can vary based on the location of the stripes along the y-axis.

[0109] The microLED array 2000 may include microLED sources corresponding to the primary colors, red, green, and blue. This figure illustrates, for example, the configuration of all microLED sources in the array 2000 corresponding to all three colors. In some embodiments, all microLEDs in the array may include microLED sources corresponding to three colors, for example, red, green, and blue. In some embodiments, the three colors may be arranged in repeating sets, for example, repeating red-green-blue, repeating blue-red-green, etc. In some embodiments, the widths of the stripes corresponding to different colors may vary. For example, the stripe corresponding to red may have a first width, the stripe corresponding to green may have a second width, and the stripe corresponding to blue may have a third width, with the first, second, and third widths being different. The relative widths of the stripes corresponding to different colors (e.g., wavelengths) may be based on the desired refractive power of a particular color. In some embodiments, the width of the stripes may vary based on their relative location within the micro-LED array, for example, narrower stripes located near the center of the micro-LED array and wider stripes located near the edges.

[0110] Referring to Figure 10, in some embodiments, the micro-LED array 1040 may be located within the bottom of the reflector 1050, traversing the light input aperture 1051. In other embodiments, the micro-LED array may be displaced from the bottom of the reflector, for example, the micro-LEDs may be located at different positions along the longitudinal axis of the reflector, the longitudinal axis corresponding to an axis perpendicular to the light input and light output apertures of the reflector. In some embodiments, moving the micro-LED array in the z-direction may affect the focus and efficiency of the light output from the light output outlet, e.g., the exit pupil of the reflector.

[0111] While the disclosed embodiments are fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be obvious to those skilled in the art. For example, one or more elements of an implementation may be combined, removed, modified, or complemented to form further implementations. Such changes and modifications are understood to fall within the scope of the disclosed embodiments as defined by the appended claims.

Claims

[Claim 1] The invention described herein.

Images