Virtual, augmented, and mixed reality systems and methods

Abstract

A head-mounted display system configured to be worn over eyes of a user includes a frame configured to be worn on a head of the user. The system also includes a display disposed on the frame over the eyes of the user. The system further includes an inwardly-facing light source disposed on the frame and configured to emit light toward the eyes of the user to improve visibility of respective portions of a face and the eyes of the user through the display. Moreover, the system includes a processor configured to control a brightness of the display, an opacity of the display, and an intensity of the light emitted by the inwardly-facing light source.

Description

VIRTUAL, AUGMENTED, AND MIXED REALITY SYSTEMS AND METHODS Cross-Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 586,880, filed on September 29, 2023. The present application is related to U.S. Utility Patent Application Serial Number 14 / 555,585 filed on November 27, 2014 under attorney docket number ML.20011.00 and entitled “VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS,” and U.S. Utility Patent Application Serial No. 14 / 738,877 filed on June 13, 2015 under attorney docket number ML.20019.00 and entitled “METHODS AND SYSTEMS FOR CREATING VIRTUAL AND AUGMENTED REALITY.” The contents of the aforementioned patent applications are hereby expressly and fully incorporated by reference in their entirety, as though set forth in full. Described in the aforementioned incorporated patent applications are various embodiments of augmented and mixed reality systems and methods. Described herein are further embodiments of virtual, augmented, and mixed reality systems and methods.Copyright Notice

[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.Field of the Invention

[0003] The present disclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods. In particular, the presentdisclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods for facilitating face-to-face interactions by illuminating the eyes of a user / wearer / viewer.Background

[0004] Modern computing and display technologies have facilitated the development of virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems. VR systems create a simulated environment for a user to experience. This can be done by presenting computer-generated imagery to the user through a head-mounted display. This imagery creates a sensory experience which immerses the user in the simulated environment. A VR scenario typically involves presentation of only computer-generated imagery rather than also including actual real-world imagery.

[0005] AR systems generally supplement a real-world environment with simulated elements. For example, AR systems may provide a user with a view of the surrounding real-world environment via a head-mounted display. However, computer-generated imagery can also be presented on the display to enhance the real-world environment. This computer-generated imagery can include elements which are contextually-related to the real-world environment. Such elements can include simulated text, images, objects, etc. MR systems also introduce simulated objects into a real-world environment, but these objects typically feature a greater degree of interactivity than in AR systems. The simulated elements can often times be interactive in real time.

[0006] Figure 1 depicts an example AR / MR scene 2 where a user sees a real-world park setting 6 featuring people, trees, buildings in the background, and a concrete platform 20. In addition to these items, computer-generated imagery is also presented tothe user. The computer-generated imagery can include, for example, a robot statue 10 standing upon the real-world platform 20, and a cartoon-like avatar character 12 flying by which seems to be a personification of a bumble bee, even though these elements 12, 10 are not actually present in the real-world environment.

[0007] Various optical systems generate images at various depths for displaying VR, AR, or MR scenarios. Some such optical systems are described in U.S. Utility Patent Application Serial No. 14 / 555,585 filed on November 27, 2014 (attorney docket number ML.20011.00), the contents of which have been previously incorporated by reference herein. Other such optical systems for displaying MR experiences are described in U.S. Utility Patent Application Serial No. 14 / 738,877 (attorney docket number ML.20019.00), the contents of which have been previously incorporated by reference herein.

[0008] Although AR and MR systems may provide a user with a view of the surrounding real-world environment via a head-mounted display, the user’s eyes are often not visible. Similarly, some VR systems are configured to operate in see-through (off) and non-see-through (on) modes, and such VR systems may render the user’s eyes not visible. An important part of communication between people involves the parties seeing each other’s eyes, which communicate emotions and focus of attention. In particular, direct eye contact in face-to-face interactions may improve communications by facilitating emotional mimicry. For individuals wearing a head-mounted VR / AR / MR display, a social face-to-face interaction with another person can feel awkward because face-to-face interactions typically involve communication cues from seeing a person’s eyes. Existing VR / AR / MR displays render the user’s eyes not visible to other people in the environment even when the user can see the other people quite well. This can lead to less successfulface-to-face interactions if one person’s eyes are invisible to the other person. One feature of VR / AR / MR displays that obscure a user’s eyes is a reflective coating / filter on the outside of the display (e.g., for privacy). In other embodiments, the coating / filter of VR / AR / MR displays reduces the ambient brightness, so that a matching virtual content brightness can be achieved with lower intensity of the display. Reducing the intensity of the display reduces power consumption and heat generation among other system benefits. Another feature is a dimmer, which is configured to block light from outside of the VR / AR / MR display. Still another is half mirror configured to display information over a transmitted light view of the real-world outside of the display. While these features improve visibility of displayed virtual images, the lower transmission of light from the real- world environment thereby reducing illumination of the user’s eyes by transmitted light.

[0009] A current solution to the problem of VR / AR / MR displays obscuring a user’s eyes is to remove the VR / AR / MR display while engaging in a social interaction in the environment. However, in some cases, it is advantageous to keep the VR / AR / MR display in position over the user’s eyes while engaging in a social interaction or conversation with another person in the environment. Another solution to the problem of VR displays obscuring a user’s eyes is reverse video pass-through, which display the wearer’s eyes on an outward facing display. However, reverse video pass-through increases system complexity, resource consumption, and heat generation. Further, reverse video pass- through may lead to an uncanny appearance, temporal delays, and display locations that are offset from a true location of the user’s eyes.

[0010] Improved systems and techniques are needed for enabling / improving visibility of a user’s eyes without removing the head-mounted display, which can then displayvirtual content to viewers / wearers / users. Improved systems and techniques are needed for addressing these issues. The systems and methods described herein are configured to address these and other challenges.

[0011] What is needed is systems and techniques to improve over legacy systems and techniques, and / or over other considered approaches. Some of the approaches described in this background section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued.Summary

[0012] In one embodiment, a head-mounted display system configured to be worn over eyes of a user includes a frame configured to be worn on a head of the user. The system also includes a display disposed on the frame over the eyes of the user. The system further includes an inwardly-facing light source disposed on the frame and configured to emit light toward the eyes of the user to improve visibility of respective portions of a face and the eyes of the user through the display. Moreover, the system includes a processor configured to control a brightness of the display, and an intensity of the light emitted by the inwardly-facing light source.

[0013] In one or more embodiments, the inwardly-facing light source includes a plurality of light emitting diodes. A majority of the plurality of light emitting diodes may be disposed on the frame above the eyes of the user. The light may be a yellow / orange light or a white light. The frame may be configured to position the light source at least a minimum distance away from the face of the user. The light source may include a light guide and / or a light diffuser.

[0014] The display may include a dimmer configured to control an opacity of the display, and the processor may be configured to control the dimmer. The processor may be configured to disable the dimmer when the inwardly-facing light source is enabled. The processor may be configured to maximize the opacity of the display when the inwardly-facing light source is enabled.

[0015] In one or more embodiments, the display is configured to display virtual content to the user. The processor may be configured to alternatively place the system in an AR / MR mode, a hybrid mode, and an eye-contact mode. In the AR / MR mode, the processor sets the opacity of the display to 100% of a maximum opacity of the display, a brightness of the display to 100% of a maximum brightness of the display, and the intensity of the light to 0% of a maximum intensity of the light. In the hybrid mode, in which the processor sets the opacity of the display to 10%-90% of the maximum opacity of the display and a brightness of the display to 10%-90% of the maximum brightness of the display. In the eye-contact mode, the processor sets the opacity of the display to a minimum opacity of the display and the brightness of the display to 0% of the maximum intensity of the display.

[0016] In one or more embodiments, a transition between the AR / MR mode and the eye-contact mode is automatically controlled by the processor. The system may also include an outwardly-facing camera. The processor may be configured to place the system in the AR / MR mode when the outwardly-facing camera or ambient light sensor detects an external light level below an ambient threshold level, which allows effective AR / MR functions. The processor may be configured to place the system in the eyecontact mode when the outwardly-facing camera detects a person. The processor maybe configured to place the system in the eye-contact mode when the outwardly-facing camera detects a face in a field of view (FOV) of the user. The processor may be configured to place the system in the eye-contact mode when the outwardly-facing camera detects a face facing the user. The processor may be configured to place the system in the eye-contact mode when the outwardly-facing camera detects a non-user face in a field of view (FOV) of the user. The processor may be configured to place the system in the eye-contact mode when the outwardly-facing camera detects a non-user face facing the user. The processor may be configured to place the system in the AR / MR mode when the outwardly-facing camera does not detect a face in a field of view (FOV) of the user for a pre-determined amount of time.

[0017] In one or more embodiments, the system includes an inwardly-facing camera configured to capture an image of an eye of the user. The processor may be configured to track a line of sight of the eye of the user by analyzing the image of the eye of the user. The processor may be configured to place the system in the eye-contact mode when the outwardly-facing camera detects a person in the line of sight of the eye of the user. The processor may be configured to place the system in the eye-contact mode when the outwardly-facing camera detects a non-user face in the line of sight of the eye of the user. The processor may be configured to place the system in the eye-contact mode when no virtual content is in the line of sight of the eye of the user. The processor may be configured to place the system in the AR / MR mode when the virtual content is in the line of sight of the eye of the user for a pre-determined amount of time. The processor may be configured to place the system in the AR / MR mode when the virtual content exceeds the threshold brightness.

[0018] In one or more embodiments, a transition between the AR / MR mode and the eye-contact mode is manually controlled by the user. The processor may be configured to dim or disable the virtual content when the inwardly-facing light source is enabled. When the system receives a critical system message, the processor may be configured to place the system in the AR / MR mode, and display the critical system message to the user.

[0019] In one or more embodiments, in the hybrid mode, the processor sets the opacity of the display to less than 50%, the brightness of the display to less than 50%, and the intensity of the light to greater than 10%. In the hybrid mode, the processor may set the opacity of the display to less than 25%, the brightness of the display to less than 25%, and the intensity of the light to greater than 75%. A transition between the AR / MR mode and the eye-contact mode may be gradual.

[0020] In one or more embodiments, the processor is configured to control the intensity of the light emitted by the inwardly-facing light source based on an environmental brightness. The processor may be configured to increase the intensity of the light emitted by the inwardly-facing light source when the environmental brightness is increased. The processor may be configured to decrease the intensity of the light emitted by the inwardly- facing light source when the environmental brightness is decreased. The processor may be configured to control a color of the light emitted by the inwardly-facing light source based on an environmental color. The processor may be configured to control a direction, a color, and the intensity of the light emitted by the inwardly-facing light source to match respective environmental light direction, environmental light color, and environmental light intensity.

[0021] In one of more embodiments, the system also includes a filter. The filter is tuned to be transparent to the light from the inwardly-facing light source and opaque to light from the display.Brief Description of the Drawings

[0022] The drawings described below are for illustration purposes only. The drawings are not intended to limit the scope of the present disclosure. The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by similar reference numerals throughout the figures. In order to better appreciate how to obtain the recited and other advantages and objects of various embodiments of the disclosure, a more detailed description of the present disclosure will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the disclosure and are not therefore to be considered limiting of its scope, the disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0023] Figure 1 illustrates a user’s view of an AR / MR scene using an example AR / MR system.

[0024] Figures 2-5 schematically depict users using VR / AR / MR systems according to some embodiments.

[0025] Figure 6 schematically depicts a AR / MR system according to some embodiments.

[0026] Figure 7 schematically depicts an AR / MR system according to some embodiments.

[0027] Figures 8 and 9 are an image and a schematic diagram of a user wearing a head-mounted AR / MR display system in AR / MR mode according to some embodiments.

[0028] Figures 10 and 11 are an image and a schematic diagram of a user wearing a head-mounted display system in eye-contact or hybrid mode according to some embodiments.

[0029] Figure 12 are four interrelated graphs depicting the transition from the AR / MR mode to the eye-contact and hybrid modes according to some embodiments.

[0030] Figure 13 is a graph showing characteristics of a filter and display and eye illumination subsystems of an AR / MR display system according to some embodiments.

[0031] Figures 14 to 25 are flowcharts depicting methods for illuminating the eyes of a user of a head-mounted display system according to some embodiments.

[0032] Figure 26 is a block diagram schematically depicting an illustrative computing system according to some embodiments.Detailed Description

[0033] Various embodiments of the disclosure are directed to systems, methods, and articles of manufacture for virtual reality (VR) / augmented reality (AR) mixed reality (MR) in a single embodiment or in multiple embodiments. Other objects, features, and advantages of the disclosure are described in the detailed description, figures, and claims.

[0034] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples so as to enable those skilled in theart to practice the disclosure. Notably, the figures and the examples below are not meant to limit the scope of the present disclosure. Where certain elements of the present disclosure may be partially or fully implemented using known components (or methods or processes), only those portions of such known components (or methods or processes) that are necessary for an understanding of the present disclosure will be described, and the detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the disclosure. Further, various embodiments encompass present and future known equivalents to the components referred to herein by way of illustration.

[0035] Embodiments in accordance with the present disclosure address the problem of implementation of VR / AR / MR systems often rely on combinations of off-the-shelf- components and custom components. In some cases the off-the-shelf components do not possess all of the features or performance characteristics that are needed to implement certain desired aspects of the to-be-deployed VR / AR / MR system. Some embodiments are directed to approaches for adding capabilities and / or repurposing resources to accommodate the desired features or performance characteristics of the to- be-deployed VR / AR / MR system. The accompanying figures and discussions herein present example environments, systems, methods, and computer program products for VR / AR / MR systems.

[0036] The head-mounted display systems described herein may be implemented independently of AR / MR systems, but some embodiments below are described in relation to AR / MR systems for illustrative purposes only. The head-mounted display systems described herein may also be used in a similar manner with VR systems.Summary of Problems and Solutions

[0037] As described above, an important part of face-to-face communication involves the parties seeing each other’s eyes for communication of non-verbal cues. For individuals wearing a head-mounted VR / AR / MR display, a social face-to-face interaction with another person can feel awkward because existing VR / AR / MR displays render the user’s eyes not visible to other people in the environment even when the user can see the other people quite well. One feature of VR / AR / MR displays that obscure a user’s eyes is a reflective coating on the outside of the display (e.g., for privacy). Another feature is a dimmer, which is configured to block light from outside of the VR / AR / MR display.

[0038] A current solution to the problem of head-mounted VR / AR / MR displays obscuring a user’s eyes is to remove the VR / AR / MR display while engaging in a social interaction in the environment. However, in some cases, it is advantageous to keep the VR / AR / MR display in position over the user’s eyes while engaging in a social interaction or conversation with another person in the environment.

[0039] The embodiments described herein include head-mounted display systems having an inwardly-facing light source to emit light toward the eyes of the user to improve visibility of the eyes and portions of the face of the user through the display. Various embodiments of head-mounted display systems also include a processor configured to control a brightness of the display, an opacity of the display, and an intensity of the light emitted by the inwardly-facing light source to optimize visibility of the eyes through the display and the portions of the face while simultaneously optimizing the display function of the head-mounted display systems. Such head-mounted display systems enable / im prove visibility of a user’s eyes through the head-mounted display systemswhile they are displaying virtual content to viewers / wearers / users, thereby addressing many of the above described issues.Illustrative Head-Mounted VR, AR, and / or MR Systems with Eye Illumination

[0040] The description that follows pertains to illustrative head-mounted VR, AR, and / or MR systems with which embodiments of various virtual image display systems may be practiced. However, it is to be understood that the embodiments also lends themselves to applications in other types of head-mounted display systems (including other types of VR, AR, and / or MR systems), and therefore the embodiments are not to be limited to only the illustrative systems disclosed herein.

[0041] VR / AR / MR systems disclosed herein can include a head-mounted display which presents computer-generated (i.e., virtual) imagery (video / image data) to a viewer / wearer / user 50. The head-mounted display systems are wearable, which may advantageously provide a more immersive VR / AR / MR experience. Various components of head-mounted VR, AR, and / or MR virtual image systems 100 are depicted in Figures 2 to 5. The virtual image display system 100 includes a frame structure 102 worn by an end user 50, a display subsystem 110 carried by the frame structure 102, such that the display subsystem 110 is positioned in front of the eyes of the end user 50, and a speaker 106 carried by the frame structure 102, such that the speaker 106 is positioned adjacent the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent the other ear canal of the end user 50 to provide for stereo / shapeable sound control). The display subsystem 110 is designed to present the eyes of the end user 50 with light patterns that can be comfortably perceived as augmentations to physical reality, with high-levels of image quality and three-dimensional perception, as well as beingcapable of presenting two-dimensional content. The display subsystem 110 presents a sequence of frames at high frequency that provides the perception of a single coherent scene.

[0042] In the illustrated embodiments, the display subsystem 110 employs “optical see-through” display through which the user can directly view light from real objects via transparent (or semi-transparent) elements. The transparent element, often referred to as a “combiner,” superimposes light from the display over the user’s view of the real world. To this end, the display subsystem 110 includes a partially transparent display. In some embodiments, the transparent display may be electronically controlled. In some embodiments, the transparent display may optionally include dimming to control opacity of the transparent display or one or more portions thereof. In some embodiments, the transparent display may include global dimming to control opacity of the entirety of the transparent display. The transparent display is positioned in the end user’s 50 field of view between the eyes of the end user 50 and an ambient environment, such that direct light from the ambient environment is transmitted through the transparent display to the eyes of the end user 50.

[0043] In the illustrated embodiments, an image projection assembly provides light to the partially transparent display, thereby combining with the direct light from the ambient environment, and being transmitted from the transparent display to the eyes of the user 50. The projection subsystem may be an optical fiber scan-based projection device, and the transparent display may be a waveguide-based display into which the scanned light from the projection subsystem is injected to produce, e.g., images at a single optical viewing distance closer than infinity (e.g., arm’s length), images at multiple, discreteoptical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent volumetric 3D objects. These layers in the light field may be stacked closely enough together to appear continuous to the human visual system (i.e. , one layer is within the cone of confusion of an adjacent layer). Additionally or alternatively, picture elements (i.e., sub-images) may be blended across two or more layers to increase perceived continuity of transition between layers in the light field, even if those layers are more sparsely stacked (i.e., one layer is outside the cone of confusion of an adjacent layer). The display subsystem 110 may be monocular or binocular.

[0044] The virtual image display system 100 may also include one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the head 54 of the end user 50 and / or the eye position and inter-ocular distance of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and / or gyros). Many of these sensors operate on the assumption that the frame 102 on which they are affixed is in turn substantially fixed to the user’s head, eyes, and ears.

[0045] The virtual image display system 100 may further include outwardly-facing cameras 660 to capture images of a field-of-view of a user. The virtual image display system 100 may also include respective inwardly-facing eye illumination systems 680 to emit light (e.g., visible light) toward each eye of the user. In some embodiments, each eye illumination system 680 is a plurality of individual light sources 682, such as individual light emitting diodes (LEDs). In some embodiments, the eye illumination system 680 can be controlled to emit various amounts of light, including but not limited to, 0%, 10%, 25%,50%, 75%, 90%, and 100% of the maximum light deliverable by the eye illumination system 680.

[0046] Moreover, the virtual image display system 100 may include one or more dimmers 692 (e.g., LCD dimmers), which together with the respective LOEs 690, form display subsystems 110. The above-mentioned eye illumination system 680 is a different lighting system compared to the display subsystem 110. In particular, the eye illumination system 680 is configured to emit light to illuminate a user’s eyes but not to display images to the user. The light emitted by the eye illumination system 680 is not pixelated, and does not display virtual content. In some embodiments, the eye illumination subsystem 680 is a different light source from that illuminating the LOEs 690. In some embodiments, the eye illumination subsystem 680 and the LOEs 690 may share a single light source with two different optical paths such that an intensity of the illumination subsystem 680 and a brightness of the display subsystem 110 are independently controllable. As such, the eye illumination system 680 can be on while the display subsystem 110 is off, and the display subsystem 110 can be on while the eye illumination system 680 is off.

[0047] In some embodiments, the dimmers are configured to control the opacity of (i.e., resistance to external light passing through) the display subsystem 110, by adjusting an opacity of the dimmers 692. In some embodiments, the dimmers 692 can be controlled to provide various amounts of opacity through the display subsystem 110, including but not limited to, 0%, 10%, 25%, 50%, 75%, 90%, and 100%.

[0048] The virtual image display system 100 may also include a user orientation detection module. The user orientation module detects the instantaneous position of the head 54 of the end user 50 (e.g., via sensors coupled to the frame 102) and may predictthe position of the head 54 of the end user 50 based on position data received from the sensors. Detecting the instantaneous position of the head 54 of the end user 50 facilitates determination of the specific actual object that the end user 50 is looking at, thereby providing an indication of the specific virtual object to be generated in relation to that actual object and further providing an indication of the position in which the virtual object is to be displayed. The user orientation module may also track the eyes of the end user 50 based on the tracking data received from the sensors.

[0049] The virtual image display system 100 may also include a control subsystem that may take any of a large variety of forms. The control subsystem includes a number of controllers, for instance one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), display bridge chips, display controllers, programmable gate arrays (PGAs), for instance field PGAs (FPGAs), and / or programmable logic controllers (PLUs).

[0050] The control subsystem of virtual image display system 100 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a three-dimensional database for storing three-dimensional scene data. The CPU may control overall operation, while the GPU may render frames (i.e., translating a three-dimensional scene into a two-dimensional image) from the three-dimensional data stored in the three-dimensional database and store these frames in the frame buffers. One or more additional integrated circuits may control the reading into and / or reading out of frames from the frame buffers and operation of the image projection assembly of the display subsystem 110.

[0051] The various processing components of the virtual image display system 100 may be physically contained in a distributed subsystem. For example, as illustrated in Figures 2 to 5, the virtual image display system 100 may include a local processing and data module 130 operatively coupled, such as by a wired lead or wireless connectivity 136, to a local display bridge 142, the display subsystem 110, and sensors. The local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (Figure 2), fixedly attached to a helmet or hat 56 (Figure 3), removably attached to the torso 58 of the end user 50 (Figure 4), or removably attached to the hip 60 of the end user 50 in a belt-coupling style configuration (Figure 5). The virtual image display system 100 may also include a remote processing module 132 and remote data repository 134 operatively coupled, such as by a wired lead or wireless connectivity 138, 140, to the local processing and data module 130 and the local display bridge 142, such that these remote modules 132, 134 are operatively coupled to each other and available as resources to the local processing and data module 130 and the local display bridge 142.

[0052] The local processing and data module 130 and the local display bridge 142 may each include a power-efficient processor or controller, as well as digital memory, such as flash memory, both of which may be utilized to assist in the processing, caching, and storage of data captured from the sensors and / or acquired and / or processed using the remote processing module 132 and / or remote data repository 134, possibly for passage to the display subsystem 110 after such processing or retrieval. The remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remotedata repository 134 may include a relatively large-scale digital data storage facility, which may be available through the internet or other networking configuration in a “cloud” resource configuration. In some embodiments, all data is stored and all computation is performed in the local processing and data module 130 and the local display bridge 1 2, allowing fully autonomous use from any remote modules.

[0053] The couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports for providing wires or optical communications, or one or more wireless interfaces or ports, such as via RF, microwave, and IR for providing wireless communications. In some implementations, all communications may be wired, while in other implementations all communications may be wireless. In still further implementations, the choice of wired and wireless communications may be different from that illustrated in Figures 2 to 5. Thus, the particular choice of wired or wireless communications should not be considered limiting.

[0054] In some embodiments, the user orientation module is contained in the local processing and data module 130 and / or the local display bridge 142, while CPU and GPU are contained in the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be contained in the local processing and data module 130 and / or the local display bridge 142. The 3D database can be associated with the remote data repository 134 or disposed locally.

[0055] Figure 6 schematically depicts an AR or MR system 600 (hereinafter referred to as “system 600”) according to some embodiments. The system 600 uses a light guiding optical element (hereinafter referred to as “LOE 690”). The system 600 generally includes one or more processors 610, one or more light sources 620, one or morecontroller / display bridges (DB) 630, one or more spatial light modulators (SLM) 640, and one or more LOE 690 that function as a multiple plane focus system. The system 600 may also include an eye-tracking subsystem 650.

[0056] The processor 610 is configured to generate virtual content to be displayed to the user. The processor 610 may convert an image or video associated with the virtual content to a format that can be projected to the user in 3D. For example, in generating 3D content, the virtual content may need to be formatted such that portions of a particular image are displayed at a particular depth plane while others are displayed at other depth planes. In one embodiment, all of the image may be generated at a particular depth plane. In another embodiment, the processor 610 may be programmed to provide slightly different images to the right and left eyes such that when viewed together, the virtual content appears coherent and comfortable to the user’s eyes.

[0057] The processor 610 may further include a memory 612, a GPU 614, a CPU 616, and other circuitry for image processing. The processor 610 may be programmed with the desired virtual content to be presented to the user of the system 600. It should be appreciated that in some embodiments, the processor 610 may be housed in the system 600. In other embodiments, the processor 610 and other circuitry may be housed in a belt pack that is coupled to the system 600. In some embodiments, the processor 610, or one or more components thereof, may be a part of a local processing and data module (e.g., local processing and data module 130). As mentioned above, the local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (Figure 2), fixedly attached to a helmet or hat 56(Figure 3), removably attached to the torso 58 of the end user 50 (Figure 4), or removably attached to the hip 60 of the end user 50 in a belt-coupling style configuration (Figure 5).

[0058] The processor 610 is operatively coupled to the light source 620 which projects light associated with the desired virtual content and one or more spatial light modulators 640. The light source 620 is compact and has high resolution. The light source 620 is operatively coupled to a controller / DB 630. The light source 620 may include color specific LEDs and lasers disposed in various geometric configurations. Alternatively, the light source 620 may include LEDs or lasers of like color, each one linked to a specific region of the field of view of the display. In another embodiment, the light source 620 may include a broad-area emitter such as an incandescent or fluorescent lamp with a mask overlay for segmentation of emission areas and positions. Although the light source 620 is directly connected to the system 600 in Figure 2B, the light source 620 may be connected to the system 600 via optical fibers (not shown). The system 600 may also include condenser (not shown) configured to collimate the light from the light source 620.

[0059] The SLM 640 may be reflective (e.g., an LCDS, an FLCOS, a DLP DMD, or a MEMS mirror system), transmissive (e.g., an LCD) or emissive (e.g. an FSD or an OLED) in various exemplary embodiments. The type of SLM 640 (e.g., speed, size, etc.) can be selected to improve the creation of the 3D perception. While DLP DMDs operating at higher refresh rates may be easily incorporated into stationary systems 600, wearable systems 600 may use DLPs of smaller size and power. The power of the DLP changes how 3D depth planes / focal planes are created. The processor 610 is operatively coupled to the SLM 640, which encodes the light from the light source 620 with the desired virtualcontent. Light from the light source 620 may be encoded with the image information when it reflects off of, emits from, or passes through the SLM 640.

[0060] Light from the SLM 640 is directed to the LOE 690 such that light beams encoded with image data for one depth plane and / or color by the SLM 640 are effectively propagated along the LOE 690 for delivery to an eye of a user. The light source 620 and LOE 690 can selectively project images (synchronously encoded by the SLM 640 under the control of controller / DB 630) that appear to originate from various depth planes or positions in space. Accordingly, the system 600 can generate a 3D image of virtual objects at various depth planes that appear to exist simultaneously in the 3D image.

[0061] The controller / DB 630 is in communication with and operatively coupled to the processor 610, the light source 620, the SLM 640, and the dimmer 692 to coordinate the synchronous display of images by instructing the SLM 640 to encode the light beams from the light source 620 with appropriate image information from the processor 610. While the system includes an processor 610, the controller / DB 630, in some embodiments, may also perform at least some image processing including, for example, the processes of the memory 612, the GPU 614, and / or the CPU 616. In some embodiments, the controller / DB 630 may include one or more components shown in the processor 610, such as, for example, the memory 612, the GPU 614, and / or the CPU 616.

[0062] The processor 610 may be configured to control one or more characteristics of the light emitted by the inwardly-facing illumination units 680, including but not limited to, intensity (e.g., 0% to 100%), color, direction, etc. The processor 610 may also be optionally configured to control an opacity of the dimmer 682 (e.g., 0% to 100%). Theprocessor may be configured to place the system 100 in an AR / MR mode, a hybrid mode, and an eye-contact mode. The AR / MR mode maximizes the quality of virtual content displayed to the user. In the AR / MR mode, the processor sets the opacity of the display to 100%, the brightness of the display to 100%, and the intensity of the light to 0%. The hybrid mode allows the eyes of the viewer / wearer / user to be at least partially visible to the other users through the display while virtual content is displayed to the user. In the hybrid mode, the processor sets the opacity of the display to 10%-90% and the brightness of the display to 10%-90%. The eye-contact mode enables / improves visibility of the eyes of the viewer / wearer / user without removing the system 100 to facilitate face-to-face interactions with people in the environment. In the eye-contact mode, the processor sets the opacity of the display to 0% and the brightness of the display to 0%. Minimizing brightness of the display in eye-contact mode also minimizes virtual content displayed to the user that may partially occlude the faces of conversation partners during social interactions.

[0063] The system 600 may also include an optional eye-tracking subsystem 650 that is configured to track the user’s eyes to determine respective lines of sight of the users eyes and / or the user’s point of vergence. In one embodiment, the system 600 is configured to display virtual content at a particular depth through the LOE 690, based on input from the eye-tracking subsystem 650 such that the image is generated at a desired depth plane that coincides with the user’s point of vergence. For example, if the user’s eyes are parallel to each other, the system 600 may display virtual content through the LOE 690 that is configured to deliver collimated light to the user’s eyes, such that the image appears to originate from optical infinity. In another example, if the eye-trackingsubsystem 650 determines that the user’s point of vergence is at 1 meter away, the LOE 690 that is configured to focus approximately within that range may be illuminated instead.

[0064] Figure 7 schematically depicts an AR or MR system 700 (hereinafter referred to as “system 700”) according to some embodiments. The system 700 includes a frame 702 configured to be worn on the head of a user. The system 700 also includes a pair of displays, which in turn each include an LOE 790 and a dimmer 792. The system further includes two pluralities of inwardly-facing illumination systems / light sources 780. Each plurality of inwardly-facing illumination system / light source 780 includes a plurality of individual light sources 782 (e.g., individual light emitting diodes or LEDs). Using LEDs as individual light sources 782 reduces power consumption, heat generation, and light source size.

[0065] Figures 8 and 9 are an image and a schematic diagram of a user 50 wearing a head-mounted AR / MR display system 800, 900 in AR / MR mode according to some embodiments. In the AR / MR mode, visibility of the eyes and face of a user 50 through the display subsystem is minimized to facilitate display of virtual content to the user. The head-mounted AR / MR display systems 800, 900 include all of the parts schematically depicted in Figures 6 and 7, some of which are similarly numbered in Figures 8 and 9. The head-mounted AR / MR display systems 800, 900 include light guiding optical element 890, 990 and dimmer 892, 992, which together form a display subsystem. In some embodiments, in the AR / MR mode: the opacity of the dimmer 892, 992, and therefore the display subsystem, is set to 100% of the maximum opacity of the dimmer 892, 992; a brightness of the display subsystem is set to 100% of the maximum brightness of thedisplay subsystem; and the intensity of an inwardly-facing eye illumination system (not shown) is set to 0%.

[0066] Figures 10 and 11 are an image and a schematic diagram of a user 50 wearing a head-mounted AR / MR display system 1000, 1100 in eye-contact mode or hybrid according to some embodiments. In the eye-contact mode, visibility of the eyes and face of a user 50 through the display subsystem is maximized to facilitate face-to-face interaction of a user wearing the AR / MR display system 1000, 1100 and an outside observer. In the hybrid or eye-contact mode, visibility of the eyes and face of a user 50 through the display subsystem is set to an intermediate degree (e.g., 10% to 90%) to facilitate display of virtual content to the user while simultaneously facilitating face-to-face interactions without removing the AR / MR display system 1000, 1100. The head-mounted AR / MR display systems 1000, 1100 include all of the parts schematically depicted in Figures 6 and 7, some of which are similarly numbered in Figures 10 and 11. The headmounted AR / MR display systems 1000, 1100 include light guiding optical element 1090, 1190 and dimmer 1092, 1192, which together form a display subsystem.

[0067] In some embodiments, in the eye-contact mode: the opacity of the dimmer 1092, 1192, and therefore the display subsystem, is set to 0%; a brightness of the display subsystem is set to 0%; and the intensity of an inwardly-facing eye illumination system (not shown) is set to 100% of the maximum intensity of the inwardly-facing eye illumination system (not shown). Setting the inwardly-facing eye illumination system (not shown) to 100% in Figures 10 and 11 enables / improves visibility of the eyes 52 and a portion of the face 53 of the user 50 through the respective display subsystems. In some embodiments, in the hybrid mode: the opacity of the dimmer 1092, 1192, and thereforethe display subsystem, is set to 10% to 90% of the maximum opacity of the dimmer 1092, 1192; and a brightness of the display subsystem is set to 10% to 90% of the maximum brightness of the display subsystem. Setting the dimmer opacity, the display subsystem brightness, and the eye illumination system intensity to intermediate values in the hybrid mode facilitates display of virtual content to the user while simultaneously facilitating face- to-face interactions without removing the AR / MR display system 1000, 1100.

[0068] Figure 10 shows light (e.g., from the eye illumination system or the display subsystem) leaking out of the AR / MR display system 1000, 1100 and illuminating portions of the user’s face around (e.g., above and below) AR / MR display system 1000, 1100. This leaking light results in an undesirable appearance of the user’s face that is exacerbated in dark environments. In some embodiments, the AR / MR display system 1000, 1100 includes an elastic seal or a fabric drape that minimizes or eliminates leaking light from the eye illumination system or the display subsystem by closing the gap between the AR / MR display system 1000, 1100 and the user’s face.

[0069] The inwardly-facing illumination systems / light sources 780 may lead to glare and discomfort for the user. Further, the inwardly-facing illumination systems / light sources 780 consume system power and generate heat. For these and other reasons, the system 700 includes a processor (not shown) configured to enable / activate the inwardly-facing illumination systems / light sources 780 only when they are needed (i.e., when a wearer of the system 700 engages in face-to-face communications). Various methods for triggering / enabling / activating the inwardly-facing illumination system / light sources 780 are described herein.

[0070] In the embodiment depicted in Figure 7, using a plurality of individual light sources 782 to illuminate a user’s eyes reduces glare and discomfort for the user by distributing the light for illumination. In some embodiments, a diffuser or a light guide is used to distribute the light for illumination to reduce glare and discomfort. Distributing the light for illumination (e.g., using a plurality of individual light sources 782, a diffuser, and / or light guide) also reduces shadows on a user’s face and provides a more flattering appearance of the user’s face.

[0071] In the embodiment depicted in Figure 7, the frame 702 is configured to position the individual light sources 782 at least 2 mm away from the user’s face. Keeping the individual light sources 782 a minimum distance away from the user’s face reduces user discomfort from heat transferred from the individual light sources 782 to the user’s face.

[0072] In the embodiment depicted in Figure 7, a majority of the plurality of individual light sources 782 is disposed above / to the side of a user’s eyes to facilitate illumination of the user’s eyes without an eerie / uncanny “illuminated from below” appearance.

[0073] In some embodiments, it may be preferable to have a hybrid mode in which the individual light sources 782 are partially enabled, to enable both the viewer / wearer / user to see the virtual content displayed by the system 700 and the viewer / wearer / user’s eyes to be seen through the display 790 by the user’s communication partners (see e.g., Figures 10 and 11 ).

[0074] In some embodiments, the individual light sources 782 can be activated / enabled automatically (e.g., by software running on a processor) or manually (e.g., with a hardware switch or knob or a software GUI). In some embodiments, the individual light sources 782 can be activated / enabled automatically, which can beoverridden by the user manually or by software running on the processor and / or by the operating system.

[0075] In some embodiments, the individual light sources 783 are configured to emit white, white-orange, and / or white-yellow light color light. White-orange and white-yellow light are a good compromise between light that is pleasing to an outside observer and comfortable for a system user. For instance, blue light can scatter more and be more disturbing to system users, and other hues such as red can be more comfortable for system users, but more disturbing for outside observers.

[0076] In some embodiments, the transition from the AR / MR mode to the eye-contact and hybrid modes occurs gradually as shown schematically in the interrelated graphs 1200 in Figure 12. Before ti , an AR / MR display system is functioning in AR / MR mode in which display opacity is set to 100%, display brightness is set to 100%, and eye illumination system intensity is set to 0%. At ti, eye-contact mode is enabled either automatically by software running on a processor or manually by the user via a user interface. After ti , the system gradually transitions from the AR / MR mode to the eyecontact mode in which display opacity set to 0%, display brightness is set to 0%, and eye illumination system intensity is set to 100%. At t2, hybrid mode is enabled either automatically by software running on a processor or manually by the user via a user interface. After t , the system gradually transitions from the eye-contact mode to the hybrid mode in which display opacity set to 50%, display brightness is set to 50%, and eye illumination system intensity is set to 50%. At ts, AR / MR mode is enabled either automatically by software running on a processor or manually by the user via a user interface. After ts, the system gradually transitions from the hybrid mode to the AR / MRmode. Gradually transitioning between different modes avoids sudden changes in intensity of the individual light sources 783, which may be disturbing to the user and the outside observer.

[0077] In some embodiments, the processor is configured to control an intensity, a color, and / or a direction of the light emitted by the inwardly-facing light source 780 based on corresponding environmental light characteristics. For instance, in a bright environment, a reflective coating will reflect more light, and more light is needed on the user’s eyes. Conversely, the light on the user’s eyes should not be too bright in a dark environment to create a more natural appearance.

[0078] In some embodiments, the intensity of lights at different positions around the user’s eyes could also be configured to match the main light directions of the environment to recreate the natural light situation, leading to a more natural look of the face through the display system. For instance, if the environment is bright from the right side, the individual light sources 782 on the right side of the user’s eyes may have a higher intensity compared to those on the left side of the user’s eyes.

[0079] In some embodiments, the display subsystem includes a filter configured to control a wavelength at which the filter can be transparent. To reduce the required intensity of the light while still providing privacy, the frequency response of the filter can be tuned to mostly block the wavelengths of the light from the display subsystem, but to be transparent for other wavelengths which can be used to illuminate the user’s eyes and face.

[0080] Figure 13 is a graph 1300 showing characteristics of such a filter and display and eye illumination subsystems according to some embodiments. The graph 1300shows the emission spectrum of the display subsystem 1310 and the emission spectrum of the eye illumination system 1320. The graph 1300 also shows the tuned filter transparency 1330, where the wavelength of the light is on the horizontal axis and the intensity and transparency on the vertical axis. As shown in Figure 13, the filter transparency 1330 has been tuned to match the wavelengths of the eye illumination system 1322 to facilitate visibility of the user’s eyes and face through the display system. The filter transparency 1330 has also been tuned to block the wavelengths of the display subsystem 1310 to maintain privacy of the virtual content displayed. Another advantage of a display system having a filter is that the light from the eye illumination system 780 requires less power and creates less visual and thermal discomfort.Illustrative Eye Illumination Methods For Head-Mounted Display Systems

[0081] The following embodiments are methods for illuminating the eyes of the user of head-mounted display systems to enable / improve visibility of the user’s eyes and portions of the user’s face. In some embodiments, the opacity and brightness of a display subsystem is also controlled to enable / improve visibility of the user’s eyes and portions of the user’s face. The following methods can be performed with the VR, AR, and / or MR virtual image systems 100, 600, 700 disclosed herein.

[0082] As described above, the processor may be configured to place the system 100 in an AR / MR mode, a hybrid mode, and an eye-contact mode. In the AR / MR mode, the processor sets the opacity of the display to 100%, the brightness of the display to 100%, and the intensity of the light to 0%. In the hybrid mode, the processor sets the opacity of the display to 10%-90% and the brightness of the display to 10%-90%. In the eye-contactmode, the processor sets the opacity of the display to 0%, the brightness of the display to 0%.

[0083] In some embodiments, in the hybrid mode, the processor sets the opacity of the display to less than 50%, the brightness of the display to less than 50%, and the intensity of the light to greater than 75%. In some embodiments, in the hybrid mode, the processor sets the opacity of the display to less than 25%, the brightness of the display to less than 25%, and the intensity of the light to greater than 75%. In some embodiments, a transition between the AR / MR mode and the eye-contact mode is automatically controlled by the processor. In some embodiments, a transition between the AR / MR mode and the eye-contact mode is manually controlled by the user, because the automatically controlled transition between the AR-MR mode and the eye-contact mode may not always be desired by the user. In some embodiments, a transition between the AR / MR mode and the eye-contact mode may be gradual or abrupt.

[0084] In some embodiments, the processor is configured to control the intensity, the color, and / or the direction of light emitted by the inwardly-facing light source based on respective environmental light intensity, environmental light color, and / or environmental light direction. In some embodiments, the processor is configured to control the intensity, the color, and / or the direction of light emitted by the inwardly-facing light source to match respective environmental light intensity, environmental light color, and / or environmental light direction.

[0085] Figure 14 depicts a method 1400 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 1410, the processor detects that the inwardly-facing eye illumination system is enabled. At step 1412, inresponse to detecting that the inwardly-facing eye illumination system is enabled, the processor disables the dimmer and the display subsystem.

[0086] Figure 15 depicts a method 1500 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 1510, the processor detects that the inwardly-facing eye illumination system is enabled. At step 1512, in response to detecting that the inwardly-facing eye illumination system is enabled, the processor minimizes the opacity of the display subsystem and disables the display subsystem.

[0087] Figure 16 depicts a method 1600 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 1610, the processor and the outwardly-facing camera detect an external light level below an ambient threshold level. At step 1612, in response to detecting an external light level below an ambient threshold level, which allows effective AR / MR functions, the processor places the system in the AR / MR mode.

[0088] Figure 17 depicts a method 1700 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 1710, the processor and the outwardly-facing camera detect a person in the field-of-view (FOV) of the user. At step 1712, in response to detecting a person in the user FOV, the processor places the system in the eye-contact mode.

[0089] Figure 18 depicts a method 1800 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 1810, the processor and the outwardly-facing camera detect a face in the field-of-view (FOV) of the user. At step 1812, in response to detecting a face in the user FOV, the processor places thesystem in the eye-contact mode. In some embodiments, the processor places the system in the eye-contact mode only when the processor and the outwardly-facing camera detect a face facing the user. In some embodiments, the processor places the system in the eye-contact mode only when the processor and the outwardly-facing camera detect a non-user face in the user FOV. In some embodiments, the processor places the system in the eye-contact mode only when the processor and the outwardly-facing camera detect a non-user face facing the user. Eliminating a user’s face as a trigger for eye-contact mode prevents accidental entry into eye-contact mode when the outwardly facing camera captures a reflected image. In some embodiments, the processor places the system in the eye-contact mode only when the processor and the outwardly-facing camera detect a non-user face directly facing the user to avoid accidental entry into eye-contact mode when a non-user walks by the user.

[0090] Figure 19 depicts a method 1900 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 1910, the processor and the outwardly-facing camera detect a person in the user’s line of sight. At step 1912, in response to detecting a person in the user’s line of sight, the processor places the system in the eye-contact mode.

[0091] Figure 20 depicts a method 2000 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 2010, the processor and the outwardly-facing camera detect a non-user face in the user’s line of sight. At step 2012, in response to detecting a non-user face in the user’s line of sight, the processor places the system in the eye-contact mode.

[0092] Figure 21 depicts a method 2100 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 2110, the processor determines that no virtual content is in the user’s line of sight. At step 2112, in response to determining that no virtual content is in the user’s line of sight, the processor places the system in the eye-contact mode.

[0093] Figure 22 depicts a method 2200 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 2210, the processor determines that virtual content is in the user’s line of sight. At step 2212, in response to determining that virtual content is in the user’s line of sight, the processor places the system in the AR / MR mode.

[0094] Figure 23 depicts a method 2300 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 2310, the processor determines that virtual content exceeds a threshold brightness (e.g., greater than 6,000 lumens). At step 2312, in response to determining that virtual content exceeds the threshold brightness, the processor places the system in the AR / MR mode.

[0095] Figure 24 depicts a method 2400 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 2410, the processor receives a critical system message (e.g., a system fault message). At step 2412, in response to receiving the critical system message, the processor places the system in the AR / MR mode and displays the critical system message.

[0096] Figure 25 depicts a method 2500 for illuminating the eyes of the user of headmounted display systems according to some embodiments. At step 2510, the processor detects that the inwardly-facing eye illumination system is enabled. At step 2512, inresponse to detecting that the inwardly-facing eye illumination system is enabled, the processor dims or disables display of virtual content.System Architecture Overview

[0001] Figure 26 is a block diagram of an illustrative computing system 2600 suitable for implementing an embodiment of the present disclosure. Computer system 2600 includes a bus 2606 or other communication mechanism for communicating information, which interconnects subsystems and devices, such as processor 2607, system memory 2608 (e.g., RAM), static storage device 2609 (e.g., ROM), disk drive 2610 (e.g., magnetic or optical), communication interface 2614 (e.g., modem or Ethernet card), display 2611 (e.g., CRT or LCD), input device 2612 (e.g., keyboard), and cursor control.

[0002] According to one embodiment of the disclosure, computer system 2600 performs specific operations by processor 2607 executing one or more sequences of one or more instructions contained in system memory 2608. Such instructions may be read into system memory 2608 from another computer readable / usable medium, such as static storage device 2609 or disk drive 2610. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the disclosure. Thus, embodiments of the disclosure are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term “logic” shall mean any combination of software or hardware that is used to implement all or part of the disclosure.

[0003] The term “computer readable medium” or “computer usable medium” as used herein refers to any medium that participates in providing instructions to processor 2607 for execution. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as disk drive 2610. Volatile media includes dynamic memory, such as system memory 2608.

[0004] Common forms of computer readable media includes, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND flash, NOR flash), any other memory chip or cartridge, or any other medium from which a computer can read.

[0005] In an embodiment of the disclosure, execution of the sequences of instructions to practice the disclosure is performed by a single computer system 2600. According to other embodiments of the disclosure, two or more computer systems 2600 coupled by communication link 2615 (e.g., LAN, PTSN, or wireless network) may perform the sequence of instructions required to practice the disclosure in coordination with one another.

[0006] Computer system 2600 may transmit and receive messages, data, and instructions, including program, i.e., application code, through communication link 2615 and communication interface 2614. Received program code may be executed by processor 2607 as it is received, and / or stored in disk drive 2610, or other non-volatile storage for later execution. Database 2632 in storage medium 2631 may be used to store data accessible by system 2600 via data interface 2633.

[0007] While the eye illumination embodiments described herein include various triggers for interactive and AR / MR modes, other embodiments may include various triggers for various degrees of display opacity and brightness, and eye illumination.

[0097] Certain aspects, advantages and features of the disclosure have been described herein. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment of the disclosure. Thus, the disclosure may be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other advantages as may be taught or suggested herein.

[0098] Embodiments have been described in connection with the accompanying drawings. However, it should be understood that the figures are not drawn to scale. Distances, angles, etc. are merely illustrative and do not necessarily bear an exact relationship to actual dimensions and layout of the devices illustrated. In addition, the foregoing embodiments have been described at a level of detail to allow one of ordinary skill in the art to make and use the devices, systems, methods, and the like described herein. A wide variety of variations is possible. Components, elements, and / or steps may be altered, added, removed, or rearranged.

[0099] The devices and methods described herein can advantageously be at least partially implemented using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. Software modules can include computer executable code, stored in a computer’s memory, for performing the functions described herein. In some embodiments, computer-executable code is executed by one or more general purpose computers. However, a skilled artisan will appreciate, in light of this disclosure, that any module that can be implemented using software to be executed on a general purpose computer can also be implemented using a different combination of hardware, software, or firmware. For example, such a module can be implementedcompletely in hardware using a combination of integrated circuits. Alternatively or additionally, such a module can be implemented completely or partially using specialized computers designed to perform the particular functions described herein rather than by general purpose computers. In addition, where methods are described that are, or could be, at least in part carried out by computer software, it should be understood that such methods can be provided on non-transitory computer-readable media that, when read by a computer or other processing device, cause it to carry out the method.

[0100] While certain embodiments have been explicitly described, other embodiments will become apparent to those of ordinary skill in the art based on this disclosure.

[0101] The various processors and other electronic components described herein are suitable for use with any optical system for projecting light. The various processors and other electronic components described herein are also suitable for use with any audio system for receiving voice commands.

[0102] Various exemplary embodiments of the disclosure are described herein. Reference is made to these examples in a non-limiting sense. They are provided to illustrate more broadly applicable aspects of the disclosure. Various changes may be made to the disclosure described and equivalents may be substituted without departing from the true spirit and scope of the disclosure. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process act(s) or step(s) to the objective(s), spirit or scope of the present disclosure. Further, as will be appreciated by those with skill in the art, each of the individual variations described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other several embodimentswithout departing from the scope or spirit of the present disclosure. All such modifications are intended to be within the scope of claims associated with this disclosure.

[0103] The disclosure includes methods that may be performed using the subject devices. The methods may include the act of providing such a suitable device. Such provision may be performed by the end user. In other words, the “providing” act merely requires the end user obtain, access, approach, position, set-up, activate, power-up or otherwise act to provide the requisite device in the subject method. Methods recited herein may be carried out in any order of the recited events which is logically possible, as well as in the recited order of events.

[0104] Exemplary aspects of the disclosure, together with details regarding material selection and manufacture have been set forth above. As for other details of the present disclosure, these may be appreciated in connection with the above-referenced patents and publications as well as generally known or appreciated by those with skill in the art. The same may hold true with respect to method-based aspects of the disclosure in terms of additional acts as commonly or logically employed.

[0105] In addition, though the disclosure has been described in reference to several examples optionally incorporating various features, the disclosure is not to be limited to that which is described or indicated as contemplated with respect to each variation of the disclosure. Various changes may be made to the disclosure described and equivalents (whether recited herein or not included for the sake of some brevity) may be substituted without departing from the true spirit and scope of the disclosure. In addition, where a range of values is provided, it is understood that every intervening value, between theupper and lower limit of that range and any other stated or intervening value in that stated range, is encompassed within the disclosure.

[0106] Also, it is contemplated that any optional feature of the variations described may be set forth and claimed independently, or in combination with any one or more of the features described herein. Reference to a singular item, includes the possibility that there are plural of the same items present. More specifically, as used herein and in claims associated hereto, the singular forms “a,” “an,” “said,” and “the” include plural referents unless the specifically stated otherwise. In other words, use of the articles allow for “at least one” of the subject item in the description above as well as claims associated with this disclosure. It is further noted that such claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation.

[0107] Without the use of such exclusive terminology, the term “comprising” in claims associated with this disclosure shall allow for the inclusion of any additional element- irrespective of whether a given number of elements are enumerated in such claims, or the addition of a feature could be regarded as transforming the nature of an element set forth in such claims. Except as specifically defined herein, all technical and scientific terms used herein are to be given as broad a commonly understood meaning as possible while maintaining claim validity.

[0108] The breadth of the present disclosure is not to be limited to the examples provided and / or the subject specification, but rather only by the scope of claim language associated with this disclosure.

[0109] In the foregoing specification, the disclosure has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure. For example, the above-described process flows are described with reference to a particular ordering of process actions. However, the ordering of many of the described process actions may be changed without affecting the scope or operation of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.

Claims

ClaimsWhat is claimed is:1 . A head-mounted display system configured to be worn over eyes of a user, comprising: a frame configured to be worn on a head of the user; a display disposed on the frame over the eyes of the user; an inwardly-facing light source disposed on the frame and configured to emit light toward the eyes of the user to improve visibility of respective portions of a face and the eyes of the user through the display; and a processor configured to control: a brightness of the display, and an intensity of the light emitted by the inwardly-facing light source.

2. The system of claim 1 , wherein the inwardly-facing light source comprises a plurality of light emitting diodes.

3. The system of claim 2, wherein a majority of the plurality of light emitting diodes are disposed on the frame above the eyes of the user.

4. The system of claim 1 , wherein the light is a yellow / orange light.

5. The system of claim 1 , wherein the light is a white light.

6. The system of claim 1 , wherein the frame is configured to position the light source at least a minimum distance away from the face of the user.

7. The system of claim 1 , wherein the light source comprises a light guide.

8. The system of claim 1 , wherein the light source comprises a light diffuser.

9. The system of claim 1 , wherein the display comprises a dimmer configured to control an opacity of the display, and wherein the processor is configured to control the dimmer.

10. The system of claim 9, wherein the processor is configured to disable the dimmer when the inwardly-facing light source is enabled.11 . The system of claim 9, wherein the processor is configured to minimize the opacity of the display when the inwardly-facing light source is enabled.

12. The system of claim 9, wherein the display is configured to display virtual content to the user, and wherein the processor is configured to alternatively place the system in: an AR / MR mode, in which the processor setsthe opacity of the display to 100% of a maximum opacity of the display, a brightness of the display to 100% of a maximum brightness of the display, and the intensity of the light to 0% of a maximum intensity of the light; a hybrid mode, in which the processor sets the opacity of the display to 10%-90% of the maximum opacity of the display, and the brightness of the display to 10%-90% of the maximum brightness of the display; and an eye-contact mode, in which the processor sets the opacity of the display to a minimum opacity of the display, and a brightness of the display to 0% of the maximum intensity of the display.

13. The system of claim 12, wherein a transition between the AR / MR mode and the eye-contact mode is automatically controlled by the processor.

14. The system of claim 13, further comprising an outwardly-facing camera.

15. The system of claim 14, wherein the processor is configured to place the system in the AR / MR mode when the outwardly-facing camera detects an external light level below an ambient threshold level.

16. The system of claim 14, wherein the processor is configured to place the system in the eye-contact mode when the outwardly-facing camera detects a person.

17. The system of claim 14, wherein the processor is configured to place the system in the eye-contact mode when the outwardly-facing camera detects a face in a field of view (FOV) of the user.

18. The system of claim 14, wherein the processor is configured to place the system in the eye-contact mode when the outwardly-facing camera detects a face facing the user.

19. The system of claim 14, wherein the processor is configured to place the system in the eye-contact mode when the outwardly-facing camera detects a non-user face in a field of view (FOV) of the user.

20. The system of claim 14, wherein the processor is configured to place the system in the eye-contact mode when the outwardly-facing camera detects a non-user face facing the user.

21. The system of claim 14, further comprising an inwardly-facing camera configured to capture an image of an eye of the user,wherein the processor is configured to track a line of sight of the eye of the user by analyzing the image of the eye of the user.

22. The system of claim 21 , wherein the processor is configured to place the system in the eye-contact mode when the outwardly-facing camera detects a person in the line of sight of the eye of the user.

23. The system of claim 21 , wherein the processor is configured to place the system in the eye-contact mode when the outwardly-facing camera detects a non-user face in the line of sight of the eye of the user24. The system of claim 21 , wherein the processor is configured to place the system in the eye-contact mode when no virtual content is in the line of sight of the eye of the user.

25. The system of claim 21 , wherein the processor is configured to place the system in the AR / MR mode when the virtual content is in the line of sight of the eye of the user for a pre-determined amount of time.

26. The system of claim 13, wherein the processor is configured to place the system in the AR / MR mode when the virtual content exceeds the threshold brightness.

27. The system of claim 12, wherein a transition between the AR / MR mode and the eye-contact mode is manually controlled by the user.

28. The system of claim 12, wherein the processor is configured to dim or disable the virtual content when the inwardly-facing light source is enabled.

29. The system of claim 12, wherein when the system receives a critical system message, the processor is configured to: place the system in the AR / MR mode, and display the critical system message to the user.

30. The system of claim 12, wherein, in the hybrid mode, the processor sets the opacity of the display to less than 50% the brightness of the display to less than 50%, and the intensity of the light to greater than 10%.31 . The system of claim 12, wherein, in the hybrid mode, the processor sets the opacity of the display to less than 25% the brightness of the display to less than 25%, and the intensity of the light to greater than 75%.

32. The system of claim 12, wherein a transition between the AR / MR mode and the eye-contact mode is gradual.

33. The system of claim 1 , wherein the processor is configured to control the intensity of the light em itted by the inwardly-facing light source based on an environmental brightness.

34. The system of claim 33, wherein the processor is configured to increase the intensity of the light emitted by the inwardly-facing light source when the environmental brightness is increased.

35. The system of claim 33, wherein the processor is configured to decrease the intensity of the light emitted by the inwardly-facing light source when the environmental brightness is decreased.

36. The system of claim 1 , wherein the processor is configured to control a color of the light emitted by the inwardly-facing light source based on an environmental color.

37. The system of claim 1 , wherein the processor is configured to control a direction, a color, and the intensity of the light emitted by the inwardly-facing light source to match respective environmental light direction, environmental light color, and environmental light intensity.

38. The system of claim 1 , further comprising a filter,wherein the filter is tuned to be transparent to the light from the inwardly-facing light source and opaque to light from the display.

Images