Extended reality (XR) device with a camera for capturing facial images

Abstract

Systems and techniques are described herein for capturing facial images. For instance, an apparatus for capturing facial images is provided. The apparatus may include a display configured for placement proximate to an eye of a user; a camera configured to capture images of the eye; and a mirror positioned between the camera and the display, the mirror configured to redirect light from the eye to the camera.

Description

EXTENDED REALITY (XR) DEVICE WITH A CAMERA FOR CAPTURING FACIAL IMAGESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Greek Provisional Application No. 20240100426, filed June 7. 2024, which is incorporated herein by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure generally relates to imaging. For example, aspects of the present disclosure include systems and techniques for capturing an image of an eye of a user and / or portions of the user’s face.BACKGROUND

[0003] Extended reality (XR) technologies can be used to present virtual content to users, and / or can combine real environments from the physical world and virtual environments to provide users with XR experiences. The term XR can encompass virtual reality (VR), augmented reality' (AR), mixed reality (MR), and the like. XR systems can allow users to experience XR environments by overlaying virtual content onto images of a real-world environment, which can be viewed by a user through an XR device. For example, an XR device can display an environment to a user. The environment is at least partially different from the real-world environment in which the user is in. The user can generally change their view of the environment interactively, for example by tilting or moving the XR device (e.g.. the HMD or other device).

[0004] XR devices can include head-mounted devices (HMDs). Some HMDs may include one or more cameras to capture images of eyes and / or a portion of a face of a user.SUMMARY

[0005] The following presents a simplified summary relating to one or more aspects disclosed herein. Thus, the following summary should not be considered an extensive overview relating to all contemplated aspects, nor should the following summary be considered to identify key or critical elements relating to all contemplated aspects or todelineate the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts relating to one or more aspects relating to the mechanisms disclosed herein in a simplified form to precede the detailed description presented below.

[0006] Systems and techniques are described for capturing facial images. According to at least one example, an apparatus for capturing facial images is provided. The apparatus may include: a display configured for placement proximate to an eye of a user; a camera configured to capture images of the eye; and a mirror positioned between the camera and the display, the mirror configured to redirect light from the eye to the camera.

[0007] In some aspects, one or more of the apparatuses described herein is. can be part of, or can include an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle (or a computing device, system, or component of a vehicle), a mobile device (e.g., a mobile telephone or so-called “smart phone”, a tablet computer, or other type of mobile device), a smart or connected device (e.g., an Intemet-of-Things (loT) device), a wearable device, a personal computer, a laptop computer, a video server, a television (e.g., a network-connected television), a robotics device or system, or other device. In some aspects, each apparatus can include an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, each apparatus can include one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, each apparatus can include one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, each apparatus can include one or more sensors. In some cases, the one or more sensors can be used for determining a location of the apparatuses, a state of the apparatuses (e.g., a tracking state, an operating state, a temperature, a humidity7level, and / or other state), and / or for other purposes.

[0008] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0009] The foregoing, together with other features and aspects, will become more apparent upon referring to the following specification, claims, and accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Illustrative examples of the present application are described in detail below with reference to the following figures:

[0011] FIG. 1 is a diagram illustrating an example extended-reality (XR) system, according to aspects of the disclosure;

[0012] FIG. 2 is a block diagram illustrating an architecture of an example extended reality (XR) system, in accordance with some aspects of the disclosure;

[0013] FIG. 3A is a diagram of an example apparatus for capturing images of eyes of a user;

[0014] FIG. 3B is a diagram of an example apparatus for capturing images of eyes of a user;

[0015] FIG. 4 includes three different example facial images that may be used for three different tasks;

[0016] FIG. 5A is a diagram illustrating an example system including a camera for capturing images of an eye of a user;

[0017] FIG. 5B is a diagram illustrating an example system including a camera for capturing images of an eye of a user;

[0018] FIG. 6 is a conceptual diagram illustrating an example system including a camera for capturing images of eye, according to various aspects of the present disclosure;

[0019] FIG. 7 is a conceptual diagram of an example system including a camera for capturing images of eye, according to various aspects of the present disclosure;

[0020] FIG. 8 is a conceptual diagram illustrating an example mirror, according to various aspects of the present disclosure;

[0021] FIG. 9 is a conceptual diagram illustrating an example system 900 including a camera for capturing images of eye, according to various aspects of the present disclosure;

[0022] FIG. 10 is a conceptual diagram illustrating an example system including a camera for capturing images of eye, according to various aspects of the present disclosure;

[0023] FIG. 11 includes three images of a simulation of virtual position of a camera and light sources, according to various aspects of the present disclosure;

[0024] FIG. 12 includes several diagrams illustrating a mirror, according to various aspects of the present disclosure;

[0025] FIG. 13 is a conceptual diagram illustrating a system including a planar volume hologram (e.g., mirror), according to various aspects of the present disclosure;

[0026] FIG. 14 includes graphs illustrating angular bandwidth of a mirror, according to various aspects of the present disclosure;

[0027] FIG. 15 includes graphs illustrating spectral bandwidth of a minor, according to various aspects of the present disclosure;

[0028] FIG. 16 is a conceptual diagram illustrating a system including a volume holographic reflector with curved grating planes, according to various aspects of the present disclosure;

[0029] FIG. 17A includes two conceptual diagrams illustrating two respective mirrors, according to various aspects of the present disclosure;

[0030] FIG. 17B includes two conceptual diagrams illustrating two respective mirrors, according to various aspects of the present disclosure;

[0031] FIG. 17C includes a conceptual diagram illustrating a curved mirror, according to various aspects of the present disclosure;

[0032] FIG. 18 is a conceptual diagram illustrating step in a planar- volume holographic-grating fabrication scheme according to various aspects of the present disclosure;

[0033] FIG. 19 is a conceptual diagram representing roll-to-roll low-cost hologram replication;

[0034] FIG. 20 is a conceptual diagram illustrating step in a curved-volume holographic-grating fabrication scheme according to various aspects of the present disclosure;

[0035] FIG. 21 is a conceptual diagram representing roll-to-roll low-cost hologram replication;

[0036] FIG. 22 is a conceptual diagram illustrating a first example illumination scheme for a planar holographic reflector;

[0037] FIG. 23 is a conceptual diagram illustrating a second example illumination scheme for a planar holographic reflector;

[0038] FIG. 24 is a conceptual diagram illustrating a first example illumination scheme for a curved holographic reflector;

[0039] FIG. 25 is a conceptual diagram illustrating a second example illumination scheme for a curved holographic reflector;

[0040] FIG. 26 is a block diagram illustrating an example computing-device architecture of an example computing device which can implement the various techniques described herein.DETAILED DESCRIPTION

[0041] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and description are not intended to be restrictive.

[0042] The ensuing description provides example aspects only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuingdescription of the exemplary aspects will provide those skilled in the art with an enabling description for implementing an exemplary aspect. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.

[0043] The terms “exemplary’' and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term “aspects of the disclosure” does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0044] As noted previously, an extended reality (XR) system or device can provide a user with an XR experience by presenting virtual content to the user (e.g., for a completely immersive experience) and / or can combine a view of a real -world or physical environment with a display of a virtual environment (made up of virtual content). The real-world environment can include real-world objects (also referred to as physical objects), such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. As used herein, the terms XR system and XR device are used interchangeably. Examples of XR systems or devices include head-mounted displays (HMDs) (which may also be referred to as a head-mounted devices), XR glasses (e.g., AR glasses, MR glasses, etc.) (also referred to as smart or network-connected glasses), among others. In some cases, XR glasses are an example of an HMD. In some cases, an XR system can track parts of the user (e.g., a hand and / or fingertips of a user) to allow the user to interact with items of virtual content.

[0045] XR systems can include virtual reality (VR) systems facilitating interactions with VR environments, augmented reality (AR) systems facilitating interactions with AR environments, mixed reality (MR) systems facilitating interactions with MR environments, and / or other XR systems.

[0046] For instance, VR provides a complete immersive experience in a three- dimensional (3D) computer-generated VR environment or video depicting a virtual version of a real-world environment. VR content can include VR video in some cases, which can be captured and rendered at very high quality, potentially providing a truly immersive virtual reality experience. Virtual reality applications can include gaming,training, education, sports video, online shopping, among others. VR content can be rendered and displayed using a VR system or device, such as a VR HMD or other VR headset, which fully covers a user's eyes during a VR experience.

[0047] AR is a technology that provides virtual or computer-generated content (referred to as AR content) over the user’s view of a physical, real -world scene or environment. AR content can include virtual content, such as video, images, graphic content, location data (e.g., global positioning system (GPS) data or other location data), sounds, any combination thereof, and / or other augmented content. An AR system or device is designed to enhance (or augment), rather than to replace, a person’s current perception of reality. For example, a user can see a real stationary or moving physical object through an AR device display, but the user’s visual perception of the physical object may be augmented or enhanced by a virtual image of that object (e.g., a real-world car replaced by a virtual image of a DeLorean), by AR content added to the physical object (e.g., virtual wings added to a live animal), by AR content displayed relative to the physical object (e.g., informational virtual content displayed near a sign on a building, a virtual coffee cup virtually anchored to (e.g., placed on top of) a real-world table in one or more images, etc ), and / or by displaying other types of AR content. Various types of AR systems can be used for gaming, entertainment, and / or other applications.

[0048] MR technologies can combine aspects of VR and AR to provide an immersive experience for a user. For example, in an MR environment, real-world and computergenerated objects can interact (e.g., a real person can interact with a virtual person as if the virtual person were a real person).

[0049] An XR environment can be interacted with in a seemingly real or physical way. As a user experiencing an XR environment (e.g., an immersive VR environment) moves in the real world, rendered virtual content (e.g.. images rendered in a virtual environment in a VR experience) also changes, giving the user the perception that the user is moving within the XR environment. For example, a user can turn left or right, look up or down, and / or move forwards or backwards, thus changing the user’s point of view of the XR environment. The XR content presented to the user can change accordingly , so that the user's experience in the XR environment is as seamless as it would be in the real world.

[0050] In some cases, an XR system can match the relative pose and movement of objects and devices in the physical world. For example, an XR system can use tracking information to calculate the relative pose of devices, objects, and / or features of the real- world environment in order to match the relative position and movement of the devices, objects, and / or the real-world environment. In some examples, the XR system can use the pose and movement of one or more devices, objects, and / or the real-world environment to render content relative to the real-world environment in a convincing manner. The relative pose information can be used to match virtual content with the user’s perceived motion and the spatio-temporal state of the devices, objects, and real-world environment. In some cases, an XR system can track parts of the user (e.g., a hand and / or fingertips of a user) to allow the user to interact with items of virtual content.

[0051] XR systems or devices can facilitate interaction with different types of XR environments (e.g., a user can use an XR system or device to interact with an XR environment). One example of an XR environment is a metaverse virtual environment. A user may virtually interact with other users (e.g., in a social setting, in a virtual meeting, etc.), virtually shop for items (e.g.. goods, services, property, etc.), to play computer games, and / or to experience other services in a metaverse virtual environment. In one illustrative example, an XR system may provide a 3D collaborative virtual environment for a group of users. The users may interact with one another via virtual representations of the users in the virtual environment. The users may visually, audibly, haptically, or otherwise experience the virtual environment while interacting with virtual representations of the other users.

[0052] A virtual representation of a user may be used to represent the user in a virtual environment. A virtual representation of a user is also referred to herein as an avatar. An avatar representing a user may mimic an appearance, movement, mannerisms, and / or other features of the user. In some examples, the user may desire that the avatar representing the person in the virtual environment appear as a digital twin of the user. In any virtual environment, it is important for an XR system to efficiently generate high- quality avatars (e.g., realistically representing the appearance, movement, etc. of the person) in a low-latency manner. It can also be important for the XR system to render audio in an effective manner to enhance the XR experience.

[0053] In some cases, an XR system can include an optical “see-through” or “pass- through” display (e.g., see-through or pass-through AR HMD or AR glasses), allowing the XR system to display XR content (e.g., AR content) directly onto a real-world view without displaying video content. For example, a user may view physical objects through a display (e g., glasses or lenses), and the AR system can display AR content onto the display to provide the user with an enhanced visual perception of one or more real-world objects. In one example, a display of an optical see-through AR system can include a lens or glass in front of each eye (or a single lens or glass over both eyes). The see-through display can allow the user to see a real-world or physical object directly, and can display (e.g., projected or otherwise displayed) an enhanced image of that object or additional AR content to augment the user’s visual perception of the real world.

[0054] As mentioned previously, an XR device or system may include an HMD, such as AR HMD or AR glasses, that may be worn by a user of the XR system. An HMD may include one or more cameras (e.g., near-infrared (NIR) cameras and / or other type(s) of cameras) to capture images of eyes and / or a portion of a face of a user. For example, some HMDs may include cameras positioned and angled to capture to capture images of the eyes and / or eyebrows of a user.

[0055] Images of eyes of users (and / or portions of faces of the users) can be used for various tasks, for example, eye tracking, facial avatar, and iris authentication. Images that are captured from a point along (or close to) an optical axis of the eye may improve the performance of such tasks. However, placing one or more camera in an optical axis of an eye may cause system integration issues and camera design constraints. For example, placement of camera(s) in the optical axis of an eye can force the camera(s) to be positioned in front of or behind displays (or lenses of the displays) of the HMDs. Such positioning of the camera(s) relative to the displays (or lenses of the displays) limits the physical size of such cameras and thus resolution and sensitivity7, due to smaller pixel pitch design, and optical quality7, due to constraint lens design to limit z-height. to example modules of less than 1.7xl.7x2mm3.

[0056] Systems, apparatuses, methods (also referred to as processes), and computer- readable media (collectively referred to herein as “systems and techniques”) are described herein for capturing images of eyes and / or a portion of a face of a user of an HMD. For example, the systems and techniques described herein provide an HMD including acamera (e.g., an NIR camera or other type of camera) positioned in a particular location of the HMD and a mirror positioned between the camera and the eye (e.g., in the optical path of the eye). The HMD also includes a display configured for placement proximate to an eye or eyes of a user (e.g.. the HMD is placed proximate to the eye or eyes of the user when worn by the user). The camera may be positioned to capture images of the eye(s) and / or a portion of a face of the user. For example, the HMD can include the camera near a temple of a user (e.g., on an arm of a pair of XR glasses) and a mirror in the optical path of the eye that is configured to direct light from the eye to the camera.

[0057] In some aspects, the camera (and optionally light sources that are typically paired with cameras) can be positioned on the HMD away from the display. For example, the camera (and / or light sources) may be positioned to the side of the HMD, for instance on an arm of glasses. Positioning the camera away from the display may relax design constraints imposed by other approaches.

[0058] As noted previously, to capture images of the eye, the HMD may include a mirror positioned in the optical path of the eye. In some aspects, the mirror may be a volume hologram designed to reflect non-visible light (e.g., NIR light) at a specific angle and to be transparent (e.g., entirely transparent) to visible light, thus not interfering with display or ambient light contrary to usual diffraction gratings. The mirror projects the camera and optionally emitters into a virtual plane that is on the eye's optical axis, without having the camera physically placed there. As another advantage, it increases the distance between the camera and the eye, thus lessening the need for fisheye lenses for facial avatar use-cases. For example, the mirror may redirect light reflected off the eye and face to the camera. Optionally, if a light source(s) is positioned by the camera, the mirror may redirect light from the light source to the face. The mirror may be designed and placed in such a way as to make the camera, and optionally light source(s), appear to be positioned along the optical axis of the eye, at a larger distance from the eye than would otherwise be physically possible (e.g., equivalent to positioning the camera in front of the HMD).

[0059] Various aspects of the application will be described with respect to the figures below.

[0060] FIG. 1 is a diagram illustrating an example extended-reality (XR) system 100, according to aspects of the disclosure. As shown, XR system 100 includes an XR device102. XR device 102 may implement, as examples, image-capture, object-detection, gazetracking, view-tracking, localization, facial-feature tracking, facial-avatar generation, iris authentication, computational and / or display aspects of extended reality, including virtual reality (VR), augmented reality (AR), and / or mixed reality (MR). For example, XR device 102 may include one or more scene-facing cameras that may capture images of a scene in which user 108 uses XR device 102. XR device 102 may detect objects in the scene based on the images of the scene. Further, XR device 102 may include one or more userfacing cameras that may capture images of eyes and / or facial features of user 108. XR device 102 may determine a gaze of user 108 based on the images of user 108. XR device 102 may determine an object of interest in the scene based on the gaze of user 108. XR device 102 may obtain and / or render information (e.g., text, images, and / or video based on the object of interest). XR device 102 may display the information to a user 108 (e.g., within a field of view 110 of user 108).

[0061] XR device 102 may display the information to be viewed by a user 108 in field of view 110 of user 108. For example, in a ”see-through'’ configuration, XR device 102 may include a transparent surface (e.g., optical glass) such that information may be displayed on (e.g., by being projected onto) the transparent surface to overlay the information onto the scene as viewed through the transparent surface. In a “pass-through” configuration or a “video see-through” configuration, XR device 102 may include a scene-facing camera that may capture images of the scene of user 108. XR device 102 may display images or video of the scene, as captured by the scene-facing camera, and information overlaid on the images or video of the scene.

[0062] In various examples, XR device 102 may be. or may include, a head-mounted device (HMD), a virtual reality headset, and / or smart glasses. XR device 102 may include one or more cameras, including scene-facing cameras and / or user-facing cameras, a GPU, one or more sensors (e.g., such as one or more inertial measurement units (IMUs), image sensors, and / or microphones), and / or one or more output devices (e.g., such as speakers, display, and / or smart glass).

[0063] In some aspects, XR device 102 may be, or may include, two or more devices. For example. XR device 102 may include a display device and a processing device. The processing device may receive data from the display device, such as image data (e.g.. from user-facing cameras and / or scene-facing cameras) and / or motion data (from aninertial measurement unit (IMU)). The processing device may process the data and / or other data. Further the processing unit may generate data to be displayed at the displaydevice. The processing device and the display device may be communicatively connected, for example, wirelessly.

[0064] FIG. 2 is a diagram illustrating an architecture of an example extended reality (XR) system 200, in accordance with some aspects of the disclosure. XR system 200 may execute XR applications and implement XR operations.

[0065] In this illustrative example, XR system 200 includes one or more image sensors 202, an accelerometer 204, a gyroscope 206, storage 208, an input device 210, a display 212, Compute components 214. an XR engine 226. an image processing engine 228, a rendering engine 230, and a communications engine 232. It should be noted that the components 202-232 shown in FIG. 2 are non-limiting examples provided for illustrative and explanation purposes, and other examples may include more, fewer, or different components than those shown in FIG. 2. For example, in some cases. XR system 200 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SOD AR) sensors, sound navigation and ranging (SONAR) sensors, audio sensors, etc.), one or more display devices, one more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components that are not shown in FIG. 2. While various components of XR system 200, such as image sensor 202, may be referenced in the singular form herein, it should be understood that XR system 200 may include multiple of any component discussed herein (e.g., multiple image sensors 202).

[0066] Display 212 may be, or may include, a glass, a screen, a lens, a projector, and / or other display mechanism that allows a user to see the real-world environment and also allows XR content to be overlaid, overlapped, blended with, or otherwise displayed thereon.

[0067] XR system 200 may include, or may be in communication with, (wired or wirelessly) an input device 210. Input device 210 may include any suitable input device, such as a touchscreen, a pen or other pointer device, a keyboard, a mouse a button or key, a microphone for receiving voice commands, a gesture input device for receiving gesturecommands, a video game controller, a steering wheel, a joystick, a set of buttons, a trackball, a remote control, any other input device discussed herein, or any combination thereof. In some cases, image sensor 202 may capture images that may be processed for interpreting gesture commands.

[0068] XR system 200 may also communicate with one or more other electronic devices (wired or wirelessly). For example, communications engine 232 may be configured to manage connections and communicate with one or more electronic devices. In some cases, communications engine 232 may correspond to communication interface 2626 of FIG. 26.

[0069] In some implementations, image sensors 202. accelerometer 204, gyroscope 206, storage 208, display 212, compute components 214, XR engine 226, image processing engine 228, and rendering engine 230 may be part of the same computing device. For example, in some cases, image sensors 202, accelerometer 204, gyroscope 206, storage 208, display 212, compute components 214, XR engine 226. image processing engine 228, and rendering engine 230 may be integrated into an HMD, extended reality glasses, smartphone, laptop, tablet computer, gaming system, and / or any other computing device. However, in some implementations, image sensors 202, accelerometer 204. gyroscope 206, storage 208. display 212, compute components 214, XR engine 226, image processing engine 228, and rendering engine 230 may be part of two or more separate computing devices. For instance, in some cases, some of the components 202-232 may be part of, or implemented by, one computing device and the remaining components may be part of. or implemented by, one or more other computing devices. For example, such as in a split perception XR system, XR system 200 may include a first device (e.g., an HMD), including display 212, image sensor 202, accelerometer 204, gyroscope 206, and / or one or more compute components 214. XR system 200 may also include a second device including additional compute components 214 (e.g.. implementing XR engine 226. image processing engine 228, rendering engine 230, and / or communications engine 232). In such an example, the second device may generate virtual content based on information or data (e g., images, sensor data such as measurements from accelerometer 204 and gyroscope 206) and may provide the virtual content to the first device for display at the first device. The second device may be, or may include, a smartphone, laptop, tablet computer, personal computer, gaming system, aserver computer or server device (e.g., an edge or cloud-based server, a personal computer acting as a server device, or a mobile device acting as a server device), any other computing device and / or a combination thereof.

[0070] Storage 208 may be any storage device(s) for storing data. Moreover, storage 208 may store data from any of the components of XR system 200. For example, storage 208 may store data from image sensor 202 (e g., image or video data), data from accelerometer 204 (e.g., measurements), data from gyroscope 206 (e.g., measurements), data from compute components 214 (e.g., processing parameters, preferences, virtual content, rendering content, scene maps, tracking and localization data, object detection data, privacy data, XR application data, face recognition data, occlusion data, etc.), data from XR engine 226. data from image processing engine 228, and / or data from rendering engine 230 (e.g., output frames). In some examples, storage 208 may include a buffer for storing frames for processing by compute components 214.

[0071] Compute components 214 may be, or may include, a central processing unit (CPU) 216, a graphics processing unit (GPU) 218, a digital signal processor (DSP) 220, an image signal processor (ISP) 222, a neural processing unit (NPU) 224, which may implement one or more trained neural networks, and / or other processors. Compute components 214 may perform various operations such as image enhancement, computer vision, graphics rendering, extended reality operations (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, predicting, etc ), image and / or video processing, sensor processing, recognition (e.g., text recognition, facial recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), trained machine-learning operations, filtering, and / or any of the various operations described herein. In some examples, compute components 214 may implement (e.g., control, operate, etc.) XR engine 226, image processing engine 228, and rendering engine 230. In other examples, compute components 214 may also implement one or more other processing engines.

[0072] Image sensor 202 may include any image and / or video sensors or capturing devices. In some examples, image sensor 202 may be part of a multiple-camera assembly, such as a dual-camera assembly. Image sensor 202 may capture image and / or video content (e.g.. raw image and / or video data), which may then be processed by computecomponents 214, XR engine 226, image processing engine 228, and / or rendering engine 230 as described herein.

[0073] In some examples, image sensor 202 may capture image data and may generate images (also referred to as frames) based on the image data and / or may provide the image data or frames to XR engine 226, image processing engine 228, and / or rendering engine 230 for processing. An image or frame may include a video frame of a video sequence or a still image. An image or frame may include a pixel array representing a scene. For example, an image may be a red-green-blue (RGB) image having red, green, and blue color components per pixel; a luma, chroma-red, chroma-blue (YCbCr) image having a luma component and two chroma (color) components (chroma-red and chromablue) per pixel; or any other suitable type of color or monochrome image.

[0074] In some cases, image sensor 202 (and / or other camera of XR system 200) may be configured to also capture depth information. For example, in some implementations, image sensor 202 (and / or other camera) may include an RGB-depth (RGB-D) camera. In some cases, XR system 200 may include one or more depth sensors (not shown) that are separate from image sensor 202 (and / or other camera) and that may capture depth information. For instance, such a depth sensor may obtain depth information independently from image sensor 202. In some examples, a depth sensor may be physically installed in the same general location or position as image sensor 202 but may operate at a different frequency or frame rate from image sensor 202. In some examples, a depth sensor may take the form of a light source that may project a structured or textured light pattern, which may include one or more narrow bands of light, onto one or more objects in a scene. Depth information may then be obtained by exploiting geometrical distortions of the projected pattern caused by the surface shape of the object. In one example, depth information may be obtained from stereo sensors such as a combination of an infra-red structured light projector and an infra-red camera registered to a camera (e.g., an RGB camera).

[0075] XR system 200 may also include other sensors in its one or more sensors. The one or more sensors may include one or more accelerometers (e.g., accelerometer 204), one or more gyroscopes (e.g.. gyroscope 206). and / or other sensors. The one or more sensors may provide velocity, orientation, and / or other position-related information to compute components 214. For example, accelerometer 204 may detect acceleration byXR system 200 and may generate acceleration measurements based on the detected acceleration. In some cases, accelerometer 204 may provide one or more translational vectors (e.g.. up / down, left / right, forward / back) that may be used for determining a position or pose of XR system 200. Gyroscope 206 may detect and measure the orientation and angular velocity of XR system 200. For example, gyroscope 206 may be used to measure the pitch, roll, and yaw of XR system 200. In some cases, gyroscope 206 may provide one or more rotational vectors (e.g., pitch, yaw, roll). In some examples, image sensor 202 and / or XR engine 226 may use measurements obtained by accelerometer 204 (e.g., one or more translational vectors) and / or gyroscope 206 (e.g., one or more rotational vectors) to calculate the pose of XR system 200. As previously noted, in other examples, XR system 200 may also include other sensors, such as an inertial measurement unit (IMU), a magnetometer, a gaze and / or eye tracking sensor, a machine vision sensor, a smart scene sensor, a speech recognition sensor, an impact sensor, a shock sensor, a position sensor, a tilt sensor, etc.

[0076] As noted above, in some cases, the one or more sensors may include at least one IMU. An IMU is an electronic device that measures the specific force, angular rate, and / or the orientation of XR system 200, using a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers. In some examples, the one or more sensors may output measured information associated with the capture of an image captured by image sensor 202 (and / or other camera of XR system 200) and / or depth information obtained using one or more depth sensors of XR system 200.

[0077] The output of one or more sensors (e.g., accelerometer 204, gyroscope 206. one or more IMUs, and / or other sensors) can be used by XR engine 226 to determine a pose of XR system 200 (also referred to as the head pose) and / or the pose of image sensor 202 (or other camera of XR system 200). In some cases, the pose of XR system 200 and the pose of image sensor 202 (or other camera) can be the same. The pose of image sensor 202 refers to the position and orientation of image sensor 202 relative to a frame of reference (e g., with respect to a field of view 110 of FIG. 1). In some implementations, the camera pose can be determined for 6-Degrees Of Freedom (6DoF), which refers to three translational components (e.g.. which can be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a frame of reference, such as the image plane) andthree angular components (e.g. roll, pitch, and yaw relative to the same frame of reference). In some implementations, the camera pose can be determined for 3-Degrees of Freedom (3DoF), which refers to the three angular components (e.g. roll, pitch, and yaw).

[0078] In some cases, a device tracker (not shown) can use the measurements from the one or more sensors and image data from image sensor 202 to track a pose (e.g., a 6D0F pose) of XR system 200. For example, the device tracker can fuse visual data (e.g.. using a visual tracking solution) from the image data with inertial data from the measurements to determine a position and motion of XR system 200 relative to the physical world (e.g., the scene) and a map of the physical world. As described below-, in some examples, when tracking the pose of XR system 200. the device tracker can generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates for a 3D map of the scene. The 3D map updates can include, for example and without limitation, new or updated features and / or feature or landmark points associated with the scene and / or the 3D map of the scene, localization updates identifying or updating a position of XR system 200 within the scene and the 3D map of the scene, etc. The 3D map can provide a digital representation of a scene in the real / physical world. In some examples, the 3D map can anchor position-based objects and / or content to real-world coordinates and / or objects. XR system 200 can use a mapped scene (e.g., a scene in the physical world represented by, and / or associated with, a 3D map) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.

[0079] In some aspects, the pose of image sensor 202 and / or XR system 200 as a whole can be determined and / or tracked by compute components 214 using a visual tracking solution based on images captured by image sensor 202 (and / or other camera of XR system 200). For instance, in some examples, compute components 214 can perform tracking using computer vision-based tracking, model-based tracking, and / or simultaneous localization and mapping (SLAM) techniques. For instance, compute components 214 can perform SLAM or can be in communication (wired or wireless) with a SLAM system (not shown). SLAM refers to a class of techniques where a map of an environment (e.g., a map of an environment being modeled by XR system 200) is created while simultaneously tracking the pose of a camera (e.g.. image sensor 202) and / or XR system 200 relative to that map. The map can be referred to as a SLAM map and can bethree-dimensional (3D). The SLAM techniques can be performed using color or grayscale image data captured by image sensor 202 (and / or other camera of XR system 200) and can be used to generate estimates of 6D0F pose measurements of image sensor 202 and / or XR system 200. Such a SLAM technique configured to perform 6DoF tracking can be referred to as 6DoF SLAM. In some cases, the output of the one or more sensors (e g., accelerometer 204, gyroscope 206, one or more IMUs, and / or other sensors) can be used to estimate, correct, and / or otherwise adjust the estimated pose.

[0080] In some cases, the 6DoF SLAM (e.g., 6DoF tracking) can associate features observed from certain input images from the image sensor 202 (and / or other camera) to the SLAM map. For example, 6DoF SLAM can use feature point associations from an input image to determine the pose (position and orientation) of the image sensor 202 and / or XR system 200 for the input image. 6DoF mapping can also be performed to update the SLAM map. In some cases, the SLAM map maintained using the 6DoF SLAM can contain 3D feature points triangulated from two or more images. For example, keyframes can be selected from input images or a video stream to represent an observed scene. For every key frame, a respective 6DoF camera pose associated with the image can be determined. The pose of the image sensor 202 and / or the XR system 200 can be determined by projecting features from the 3D SLAM map into an image or video frame and updating the camera pose from verified 2D-3D correspondences.

[0081] In one illustrative example, the compute components 214 can extract feature points from certain input images (e.g., every input image, a subset of the input images, etc.) or from each key frame. A feature point (also referred to as a registration point) as used herein is a distinctive or identifiable part of an image, such as a part of a hand, an edge of a table, among others. Features extracted from a captured image can represent distinct feature points along three-dimensional space (e.g., coordinates on X, Y, and Z- axes), and every- feature point can have an associated feature location. The feature points in key frames either match (are the same or correspond to) or fail to match the feature points of previously-captured input images or key frames. Feature detection can be used to detect the feature points. Feature detection can include an image processing operation used to examine one or more pixels of an image to determine whether a feature exists at a particular pixel. Feature detection can be used to process an entire captured image or certain portions of an image. For each image or key frame, once features have beendetected, a local image patch around the feature can be extracted. Features may be extracted using any suitable technique, such as Scale Invariant Feature Transform (SIFT) (which localizes features and generates their descriptions), Learned Invariant Feature Transform (LIFT). Speed Up Robust Features (SURF), Gradient Location-Orientation histogram (GLOH), Oriented Fast and Rotated Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoint (FREAK), KAZE, Accelerated KAZE (AKAZE), Normalized Cross Correlation (NCC), descriptor matching, another suitable technique, or a combination thereof.

[0082] As one illustrative example, the compute components 214 can extract feature points corresponding to a mobile device, or the like. In some cases, feature points corresponding to the mobile device can be tracked to determine a pose of the mobile device. As described in more detail below, the pose of the mobile device can be used to determine a location for projection of AR media content that can enhance media content displayed on a display of the mobile device.

[0083] In some cases, the XR system 200 can also track the hand and / or fingers of the user to allow the user to interact with and / or control virtual content in a virtual environment. For example, the XR system 200 can track a pose and / or movement of the hand and / or fingertips of the user to identify or translate user interactions with the virtual environment. The user interactions can include, for example and without limitation, moving an item of virtual content, resizing the item of virtual content, selecting an input interface element in a virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and / or other virtual interface), providing an input through a virtual user interface, etc.

[0084] FIG. 3A is a diagram of an example apparatus 300 for capturing images of eyes of a user. Apparatus 300 may be an HMD, for example, a XR device. Apparatus 300 includes two displays 302. When apparatus 300 is worn by a user, displays 302 may be proximate to eyes of the user. Additionally, apparatus 300 includes cameras 308, which are positioned such that when apparatus 300 is worn by a user, cameras 308 are positioned and angled to capture images of eyes of the user. Apparatus 300 also includes light sources 304, which are positioned such that when apparatus 300 is worn by a user, light sources 304 are positioned to illuminate the eyes of the user.

[0085] In the present disclosure, references to light and illumination include electromagnetic radiation of any wavelength, including as examples, ultraviolet UV, visible, near infrared (NIR), and infrared (IR). Examples of light sources include lightemitting diodes (LEDs), edge-emitting lasers (EELs), and vertical-cavity surface-emitting lasers (VCSELs). In some cases in the present disclosure, LEDs are used as examples of light sources. Unless explicitly stated otherwise, VCSELs may be substituted for LEDs. In some cases, VCSELs may be preferable to LEDs because VCSELs may be able to produce light with narrower bandwidth range than LEDs.

[0086] In the present disclosure, references to “eyes” should be understood to apply to one eye or two eyes. For example, in some aspects, a device may capture an images of one eye of a user. Additionally, references to capturing “images of eyes,” “eye images,” “facial images” “images of eyes and / or face,” and like terms, should be understood to apply to capturing images of eyes and / or other portions of a user's face, such as eyelids, eyebrows, brow, nose, cheeks, lips, mouth, etc.

[0087] FIG. 3B is a diagram of an example apparatus 310 for capturing images of eyes of a user. Apparatus 310 includes lenses 312 (which may be referred to as “pancake lenses”). A user may view a display through lenses 312. For example, lenses 312 mayfocus light from the display to eyes of the user. Additionally, apparatus 310 includes cameras 314 which may capture images of eyes of the user. Apparatus 310 may also include light sources (not labelled in FIG. 3B) that may illuminate eyes of the user 108.

[0088] FIG. 4 includes three different example facial images that may be used for three different tasks. As mentioned previously, various tasks may be performed using images of eyes. Such tasks includes as examples, eye tracking, facial-avatar generation and / or mimicking, and iris authentication.

[0089] Eye tracking may involve tracking a user's gaze. Eye tracking may be use for gaze-based selection and / or foveation, among other tasks. Eye tracking may involve illuminating an eye with a pattern and comparing a pupil of the eye to the pattern. Additionally or alternatively, eye tracking may involve resolving shape of ring and pupil contour and using centers (e.g., a center of a pupil and a center of a reflected ring of illumination) for triangulation. For eye tracking, images captured from a “head-on view” may produce the best results. For example, an image captured along an optical axis of theeye may allow eye tracking to produce the best results. Additionally or alternatively, illuminating the eye along the optical axis may allow for the best results. Eye-tracking applications may be benefitted by capturing many images of the eye over time. For example, to have the current gaze information, it may be beneficial for an eye-tracker to capture many (e.g., 200) frames per second (fps). Image 402 is an example image of an eye that may be suitable for eye tracking.

[0090] Facial-avatar generation and / or mimicking may involve generating a visual representation of the user, and / or causing a visual representation of the user to mimic the user's expression. Facial-avatar generation and / or mimicking may work best with images that capture portions of the user's face, such as eyebrows and eyelids (upper and lower). Images capture from the side or wide-angle images of eyes may be well suited to facialavatar generation and / or mimicking. Images 404 are examples of images that may be suitable for facial avatar generation and / or mimicking.

[0091] Iris authentication may involve authenticating a user based on an image of the iris of the user. Iris authentication may involve resolving fine iris structures. For iris authentication, images captured from a “head-on view” may produce the best results. Iris authentication may benefit from having high-resolution images of the eye. Image 406 is an example images of an eye that may be suitable for iris authentication.

[0092] Images used for eye tracking, facial-avatar generation and / or mimicking, and iris authentication may each have different key performance indicators. For example, for eye tracking, frame rate may be more important than resolution. For facial-avatar generation and / or mimicking, capturing a wide-angle images the face may be more important than resolution or frame rate. For iris authentication, resolution may be more important that frame rate.

[0093] FIG. 5A is a diagram illustrating an example system 500 including a camera 502 for capturing images of an eye 506 of a user. System 500 may have a relatively short eye relief 508. In the present disclosure, the term “eye relief’ may refer to a distance between an eye and a camera that may capture an image of the eye. Having a short eye relief may necessitate camera 502 having a relative wide field of view (FoV) to capture an image of eye 506. For example, camera 502 may need a wide-angle lens to capture an image of eye 506 and / or to capture an image of portions of the face around eye 506.1Additionally, system 500 may suffer from tight size constraints due to the placement of camera 502. Further, camera 502 may be along the optical path of a display (or lens of a display). For example, camera 502 may block a user's view of a display.

[0094] FIG. 5B is a diagram illustrating an example system 510 including a camera 512 for capturing images of an eye 516 of a user. Camera 512 is placed along the optical axis of eye 506. Eye relief 518 of camera 512 relative to eye 516 is larger than eye relief 508 of camera 502 relative to eye 506. Accordingly, camera 512 may not need as wide an FoV as camera 502 to capture images of eye 516 and portions of the face around eye 516. For example, camera 512 may not need a wide-angle lens. Higher resolution and higher sensitivity results in a larger footprint. Similar to system 500, system 510 may suffer from tight size constraints due to the placement of camera 512. Further, camera 512 may be along the optical path of a display (or lens of a display). For example, camera 512 may block a user's view of a display.

[0095] FIG. 6 is a conceptual diagram illustrating an example system 600 including a camera 602 for capturing images of eye 606, according to various aspects of the present disclosure. System 600 includes a mirror 610 which may reflect light from eye 606 to camera 602.

[0096] Camera 602 of system 600 may be positioned out of the direct optical path of eye 606. For example, camera 602 may be positioned near a temple or ear of a user where camera 602 cannot be directly seen by eye 606. Thus, camera 602 may not block a view of eye 606 of a display.

[0097] Mirror 610 may reflect light from eye 606 to camera 602 such that camera 602 may capture images of eye 606. Thus, camera 602 may be considered as virtual camera 614 (e.g.. if a virtual optical path were extended from eye 606, camera 602 would be at the position in which virtual camera 614 is illustrated).

[0098] Camera 602 may have a relatively long eye relief 608. With a relatively long eye relief 608, camera 602 need not include a wide-angle lens to capture images of eye 606 and portions of the face around eye 606.

[0099] Further, because camera 602 is not in an optical path between eye 606 and a display, camera 602 may be larger than other cameras that are designed to be nearer eye606. Because camera 602 may be large, camera 602 may have fewer constraints (e.g., in terms of size and power) than other cameras. In some aspects, camera 602 may include a micro-lens array. The micro-lens array may be, or may include, wafer-level optics, metalenses etc. The micro-lens array may aid depth perception, super-resolution etc.

[0100] Mirror 610 may be a physical mirror that may implement virtual mirror 612. For example, mirror 610 may be a holographic mirror including a diffraction grating (e.g., a Bragg grating). Mirror 610 may reflect light from eye relief 608 to camera 602 as if mirror 610 were at an angle, for example, the angle illustrated by virtual mirror 612.

[0101] In addition to reflecting light from eye 606 to camera 602, mirror 610 may reflect light from light source 604 to eye 606. For example, light source 604 (which may be, or may include, VCSELs) may emit light. Minor 610 may reflect the light toward eye 606. Eye 606 may reflect the light back toward mirror 610. Mirror 610 may reflect the light to camera 602. Thus, light source 604 may be considered as virtual light source 616.

[0102] FIG. 7 is a conceptual diagram of an example system 700 including a camera 702 for capturing images of eye 706, according to various aspects of the present disclosure. System 700 includes a mirror 710 which may reflect light from eye 706 to camera 702.

[0103] Mirror 710 may be similar to mirror 610 of FIG. 6. Additionally, mirror 710 may be virtually curved. For example, mirror 710 may include a holographic mirror (e.g., including a diffraction grating) that may simulate a curved mirror. For example, mirror 710 may reflect light from a wider FoV of a face of a user than a flat mirror would. In this way, mirror 710 may simulate virtual mirror 712. Virtual mirror 712 may be angled like virtual mirror 612 of FIG. 6.

[0104] Mirror 710 may be highly conformable. Mirror 710 may include a diffraction grating be fabricated in a thin photopolymer film. Mirror 710 may be attached on the front surface of a lens or a display. In some aspects, mirror 710 may be fabricated to be virtually angled and virtually curved and to be attached to a flat surface (e g., of a lens, a display, or another flat surface). In other aspects, mirror 710 may be fabricated to be virtually angled and virtually curv ed and to be attached to a curved surface (e.g., to a curved lens, such as lenses 312 of FIG. 3B).

[0105] Further, mirror 710 may be wavelength selective. The diffraction grating of mirror 710 may be designed to reflect a desired wavelength (e.g., 940 nanometers (nm) or 850nm) and to no impact light of other wavelengths. Thus, mirror 710 may be transparent in visible wavelengths and not impact to the display image. In other words, eye 706 may see clearly through mirror 710 to a display. Mirror 710 may be different from other diffraction gratings that may have diffraction orders for visible light, lowering overall efficiency and creating rainbow artifact effect to the user.

[0106] Further still, mirror 710 may have a low cost. For example, the grating film may be replicated through roll-to-roll mass productions.

[0107] Mirror 710 may be configured to be positioned, for example, 1-3 centimeters from eye 706. Mirror 710 may have a substantially flat surface. Further, minor 710 may have a curved hologram embedded in mirror 710. Mirror 710 may reflect light coming in front of it from the face in a+-5 degrees towards the camera. In some aspects, mirror 710 may be sized to cover an eye-piece / glasses / display. In other aspects, mirror 710 may cover only a portion of an eye-piece / glasses / display. Mirror 710 may be large enough to capture image from at least 50mmx50mm of around the eye.

[0108] Camera 702 may be positioned, for example, on the side of the head, near an arm of glasses or an HMD. Camera 702 may be about the same distance from mirror 710 as eye 706 is from mirror 710. Camera 702 may be laterally offset from an optical path of eye 706. In some aspects, camera 702 may include a micro-lens array. The micro-lens array may be, or may include, wafer-level optics, metalenses etc. The micro-lens array may aid depth perception, super-resolution etc.

[0109] - Light sources 704 may include, for example, 8-18 light sources around the display, see-though glasses, and / or around camera 702. Light sources 704 may be. or may include, narrowband lasers, such as VCSELS or edge-emitting lasers.

[0110] FIG. 8 is a conceptual diagram illustrating an example mirror 810, according to various aspects of the present disclosure. Mirror 810 may be an example of mirror 610 of FIG. 6 and / or minor 710 of FIG. 7.[OHl] In addition to being wavelength selective, mirror 810 may be selective based on an angle of incidence (or angle selective). For example, mirror 810 may reflect lightfrom a selected angle and transmit light from other angles. For example, mirror 810 may allow light from a display to pass through mirror 810 unaffected. Further, mirror 810 may reflect light from an eye to a camera that is at the selected angle from mirror 810.

[0112] FIG. 9 is a conceptual diagram illustrating an example system 900 including a camera 902 for capturing images of eye 906, according to various aspects of the present disclosure. For example, system 900 includes a mirror 910 that may reflect light 918 from eye 906 to camera 902.

[0113] Mirror 910 may be the same as, may be substantially similar to, and / or may perform the same, or substantially the same, operations as mirror 610 of FIG. 6, mirror 710 of FIG. 7, or mirror 810 of FIG. 8. For example, mirror 910 may be a volume hologram including a diffraction grating configured to angle light from eye 906 (and portions efface around eye 906) to camera 902. Mirror 910 may be virtually angled and / or virtually curved.

[0114] Mirror 910 may be visually transparent. For example, mirror 910 may be wavelength selective such that visible light, such as light 914, which may be from a display or from the real world, may pass through mirror 910 unaffected. Further, mirror 910 may reflect light 916 from light sources 904 toward eye 906 based on light 916 having a predetermined wavelength. Further still, mirror 910 may reflect light 918 from eye 906 to camera 902 based on light 918 being light 916 reflected by eye 906 (and other portions of the face).

[0115] Camera 902 may be positioned on a portion of an HMD (e.g., frame 912) outside a view of eye 906. For example, camera 902 may be positioned on an arm of glasses or another portion of an XR headset.

[0116] Light sources 904 may be, or may include, a ring of 8-10 light sources. The 8- 10 corresponding glints on the eye may be used to compute eye gaze direction. Other algorithm can use the center of the iris with respect to the glint of a single source etc. Light sources 904 may be, or may include, VCSELs. Light sources 904 may not necessarily be aligned on a cross-sectional plane to conform to the space and the head. Light sources 904 may appear as a circle from the front of optical axis. Light sources 904 may be, or may include, a virtual "ring" of light sources in 3D light sources 904 may be, or may include, an ellipse.

[0117] System 900 includes light sources 904 at a first example position (e.g., proximate to camera 902). Positioning light sources 904 proximate to camera 902 may provide more space for light sources 904 to be arranged around the camera than positioning light sources 904 in another location. Additionally, positioning light sources 904 proximate to camera 902 may allow for easier alignment of light sources 904. Additionally, positioning light sources 904 proximate to camera 902 may prevent camera 902 from being blinded by direct light from light sources 904.

[0118] FIG. 10 is a conceptual diagram illustrating an example system 1000 including a camera 1002 for capturing images of eye 1006, according to various aspects of the present disclosure. For example, system 1000 includes a mirror 1010 that may reflect light 1018 from eye 1006 to camera 1002.

[0119] Mirror 910 may be the same as, may be substantially similar to, and / or may perform the same, or substantially the same, operations as mirror 610 of FIG. 6, mirror 710 of FIG. 7, mirror 810 of FIG. 8, and / or mirror 910 of FIG. 9. Camera 1002 may be positioned on a portion of an HMD (e.g., frame 1012) outside a view of eye 1006.

[0120] System 1000 includes light sources 1004 at a second example position (e.g., proximate to a display or lens). Positing light sources 1004 proximate to a display or lens may allow for higher efficiency than positioning light sources 904 proximate to camera 902 because light 1018 undergoes one reflection at mirror 1010 whereas light 916 is reflected at mirror 910 and light 918 is also reflected by mirror 910. Light may lose energy' at each reflection at mirror 910 or mirror 1010. Thus, reducing the number of reflections may conserve energy.

[0121] FIG. 11 includes three images (image 1102, image 1104, and image 1106) of a simulation of virtual position of a camera and light sources, according to various aspects of the present disclosure. Image 1102, image 1104, and image 1106, were generated using a Zemax simulation. Image 1102, image 1104, and image 1106 illustrate a ring of light sources around a camera. The rings of reflected light in illustrated in image 1106 may be used by an algorithm (e.g., a glint-based eye-tracking algorithm) for eye tracking. Each light source produces an image that appears as one of the reflected bright glints on the eye. If the light source are spatially placed in a ring the reflections of the light sources may look like a ring on the eye (the eye may' act as a mirror reflecting light from the lightsources). Roughly speaking the algorithm may determine the center of this ring, the center of the iris of the eye, and determine the gaze direction from based on the center of the ring and the center of the iris.

[0122] FIG. 12 includes several diagrams illustrating a mirror 1210, according to various aspects of the present disclosure. For example, FIG. 12 includes a diagram 1202 illustrating a diffraction grating of mirror 1210. Mirror 1210 may be an example of any of mirror 610 of FIG. 6, mirror 710 of FIG. 7, mirror 810 of FIG. 8, mirror 910 of FIG. 9, or mirror 1010 of FIG. 10. Mirror 1210 includes grating planes which may be, or may include, interfaces between regions of different indices of refraction. The angle and spacing of the grating planes may determine the reflective angle, the angular selectivity, and / or the wavelength selectivity of the mirror. Mirror 1210 may reflect input light, which may arrive at a first angle of incidence and be reflected at a different angle. Diagram 1204 is a kbat diagram illustrating the relationship between angles of incidence and reflection. Diagram 1202 and diagram 1204 relate to light being reflected from light sources (e.g., light sources 904) to an eye (e.g., eye 906).

[0123] Diagram 1206 illustrates mirror 1210 reflecting input light, which may arrive at a first angle of incidence, as reflected light at a different angle. Diagram 1208 is a kbat diagram illustrating the relationship between angles of incidence and reflection. Diagram 1206 and diagram 1208 relate to light being reflected from an eye (e.g., eye 906) to a camera (e.g., camera 902).

[0124] Mirror 1210 may be fabricated using photopolymer. Photopolymer may result in a thinner hologram with larger spectral and angular bandwidth thickness between 1 Oum to lOOum. Additionally or alternatively, mirror 1210 may be, or may include, a photorefractive material (e.g,. a photorefractive glass). Photorefractive glass may allow for thicker holograms (1mm) very efficient but low angular bandwidth of only few milliradians (mrad).

[0125] Mirror 1210 may be fabricated by superimposing several holograms to allow larger angular bandwidth.

[0126] In some aspects, mirror 1210 may be positioned (e.g., affixed or mounted) on a slim piece of glass between an eye of a user and a display. The slim piece of glass may be flat and mirror 1210 may be fabricated to reflect the light according to the diagrams ofFIG. 12 while positioned on the flat glass. In other aspects, mirror 1210 may be positioned on a lens, which may not be flat (for example, the lens may be curved). Mirror 1210 maybe fabricated to reflect the light according to the diagrams of FIG. 12 while positioned on the non-flat surface.

[0127] In some aspects, a display may be positioned behind a lens (e.g., a pancake lens) in front of the eye. In some aspects, a microdisplay using a waveguide in front of the eye may project an image to the eye. In either cases, mirror 1210 may be between the display or waveguide and the eye.

[0128] In some aspects, mirror 1210 may be virtually curved. For example, mirror 1210 may reflect light from a wide area toward a single point (e.g., a lens of a camera). In such cases, mirror 1210 may have a different vector K across mirror 1210. For example, the grating planes may be curved, additionally or alternatively the recorded hologram may be curved.

[0129] FIG. 13 is a conceptual diagram illustrating a system 1300 including a planar volume hologram (e.g., mirror 1310), according to various aspects of the present disclosure. Mirror 1310 may be an example of any of mirror 610 of FIG. 6, mirror 710 of FIG. 7, mirror 810 of FIG. 8, mirror 910 of FIG. 9, mirror 1010, of FIG. 10, or mirror 1210 of FIG. 12.

[0130] Planar reflection holograms exhibit angular bandwidth limitations due to Bragg angular selectivity. For example, virtual camera 1314 may have a limited view of portions of the face. A large refractive index modulation (An) of the holographic grating is desirable for achieving high diffraction efficiency and broader angular bandwidth. A thinner holographic medium thickness results in a larger angular bandwidth, but it also leads to lower diffraction efficiency. To capture the area of interest (diameter q>), due to the angular bandwidth limitation (with angle 20o), the distance of the virtual camera to the face (L) shall satisfy the following requirement:

[0131] FIG. 14 includes graphs illustrating angular bandwidth of a mirror, according to various aspects of the present disclosure. For example, graph 1402 illustrates diffraction efficiency as a function of angle deviation for a mirror of refractive indexmodulation (An) of 0.05 for a thickness (d) of 10 microns (urn) and 5 microns. Graph 1404 illustrates diffraction efficiency as a function of angle deviation for a mirror of refractive index modulation of 0. 10 for a thickness of 10 microns and 5 microns.

[0132] Planar reflection holograms exhibit spectral bandwidth limitations due to Bragg angular selectivity. A large refractive index modulation (An) of the holographic grating is desirable for achieving high diffraction efficiency and broader spectral bandwidth. A thinner holographic medium thickness results in a broader spectral bandwidth, but it also leads to reduced diffraction efficiency. To overcome the spectral bandwidth limitation, it is advisable to use a narrow-spectrum light source, such as a VCSEL (Vertical-Cavity Surface-Emitting Laser). A narrow spectral width ensures that the hologram will not interfere with visible light, such as the display and the real-world scene.

[0133] FIG. 15 includes graphs illustrating spectral bandwidth of a mirror, according to various aspects of the present disclosure. For example, graph 1502 illustrates diffraction efficiency as a function of wavelength deviation for a mirror of refractive index modulation of 0.05 for a thickness of 10 microns and 5 microns. Graph 1504 illustrates diffraction efficiency as a function of wavelength deviation for a mirror of refractive index modulation of 0. 10 for a thickness of 10 microns and 5 microns.

[0134] FIG. 16 is a conceptual diagram illustrating a system 1600 including a volume holographic reflector with curved grating planes (e.g., mirror 1610), according to various aspects of the present disclosure. Mirror 1610 may be an example of any of mirror 610 of FIG. 6. mirror 710 of FIG. 7, mirror 810 of FIG. 8, mirror 910 of FIG. 9, mirror 1010, of FIG. 10, or mirror 1210 of FIG. 12.

[0135] The diffraction angle increases as the light move away from the center of the curved grating. The curved grating has a similar angular bandwidth to the planar grating at each position, but it diffracts light at a larger angle away from the center. This redirection ensures that light enters the camera aperture, which would otherwise be lost with a planar hologram. The curved grating reflector introduces image distortion, which can be digitally corrected. The curved grating reflector effective increases the FOV of the image system.

[0136] FIG. 17A includes two conceptual diagrams illustrating two respective mirrors, according to various aspects of the present disclosure. Diagram 1702 illustrates light from point sources being reflected by a flat mirror. The point source may be an eye or portion of a face of a user. The point source may be reflecting light from a light source that is illuminating the face, such as a VCSEL. Diagram 1702 includes a lens aperture that may focus light to a sensor (e.g., an image sensor). Light from a point source (0, 0) may be reflected by the flat mirror to lens aperture. The lens aperture may focus the light onto the sensor. Thus, the sensor may capture an image of point source (0, 0). In contrast, light from a point source (0, -10) may be reflected by the flat mirror, but may not arrive at the aperture or sensor. Thus, point source (0, -10) may not be captured in the image capture by the sensor.

[0137] Diagram 1704 illustrates light from point sources being reflected by a curved mirror. In contrast to diagram 1702, light from a point source (0, 0) may be reflected by the curved mirror to lens aperture and light from a point source (0, -10) may be reflected by the curved mirror to the lens aperture. Thus, the image sensor may capture an image including point source (0, 0) and the point source (0, -10).

[0138] Some holographic mirrors can only reflect light incoming a small angle on the mirror (e.g., +-5°). Using a flat mirror (e.g. the flat mirror of diagram 1702), a small portion of a face (e.g., portions around point source (0. 0)) will reflect light to the camera. Light at the edge of the FOV (e.g., from portions around point source (0, -10)) will be lost.

[0139] As illustrated by diagram 1704. a curved (or virtually curved mirror) may cause light from wider portion of a face to be reflected to a camera. A virtually curved mirror may include a succession of small flat mirrors that may direct the light of each part of the face towards the camera to form an image. As illustrated by the example curved mirror of diagram 1704, a determined curvature may redirect the light of each part of the FOV to the camerato form an image. For example, light from point source (0, 0) and light from point source (0, -10) are both reflected to the camera.

[0140] FIG. 17B includes two conceptual diagrams illustrating two respective mirrors, according to various aspects of the present disclosure. Diagram 1712 illustrates light from a point source (e.g., a point on a face of a user) being reflected by a planarmirror 1714 (e.g., a planar reflection hologram) toward a camera 1716. Diagram 1722 illustrates light from a point source (e.g., a point on a face of a user) being reflected by a curved mirror 1724 (e.g., a virtually-curved reflection hologram) toward a camera 1726. The curved mirror 1724 focuses light from a wider area toward the camera 1726 than mirror 1714 reflects toward camera 1716.

[0141] The reflective volume hologram (e.g., curved mirror 1724) functions as a mirror, redirecting light from the eye to camera 1726. Curved minor 1724 may be designed to operate at a specific wavelength, such as near-infrared (NIR). For other wavelengths, like visible light from the display, curved mirror 1724 behaves like a transparent piece of glass, allowing the light to pass through. Curved mirror 1724 creates a virtual image of camera 1726 in front of the eyes of the user, making it appear as though camera 1726 is looking directly at the eye from the front.

[0142] A volume hologram (e.g., curved mirror 1724) may be formed by multiple layers of high and low refractive index modulations within a holographic medium. For a planar reflection volume hologram (e.g., planar mirror 1714), the index modulation layers are flat and planar, allowing it to function as a planar mirror. For a curved reflection volume hologram (e.g. curved mirror 1724), the index modulation layers are curved, even though the hologram material itself is flat. This enables such a volume hologram to function as a curved mirror. In other words, curved mirror 1724 may be virtually curved. For instance, curved mirror 1724 may be flat, but it may redirect light as if it were curved. In some aspects, curved mirror 1724 may be physically curved. For example, curved mirror 1724 may be fitted to a curved surface. The virtual curvature of curved mirror 1724 may account for such a physical curvature of such an instance of curved mirror 1724. For example, the virtual curvature of curved mirror 1724 may account for any physical curvature of curved mirror 1724 such that curved mirror 1724 redirects light from a face of a user toward a camera. Whether curved mirror 1724 is physically flat or physically- curved. curved mirror 1724 may cause reflected to converge., curved mirror 1724 may enhance light collection efficiency by directing more light into the camera aperture, as illustrated by diagram 1712 and diagram 1722.

[0143] For example, curved mirror 1724 may redirect light from a first area (e.g.. a portion of a face of the user) toward a second area (e.g.. toward an aperture of camera 1726). Curved mirror 1724 may focus light from a larger first area toward a smallersecond area. For example, if the first area and the second area are equidistant from curved mirror 1724, based on camera 1726 focusing the light, the second area may be smaller than the first area. For example, the first area may be offset from the substrate at a distance and the second area may be offset from the substrate at the distance.

[0144] For example, FIG. 17C includes a conceptual diagram 1730 illustrating a curved mirror 1724, according to various aspects of the present disclosure. Curved mirror 1724 may be physically flat or physically curved. Curved mirror 1724 may be virtually curved (e.g., by including curved index -modulation layers). Curved mirror 1724 may be curved to redirect light from area 1734 to area 1744. Area 1734 may be larger than area 1744. For example, curved mirror 1724 may focus light from area 1734 to area 1744.

[0145] Curved mirror 1724 may be included in a volume hologram. The volume hologram may have a surface facing in direction 1732 (e.g., toward the face of the user). Camera 1726 may face in direction 1742 (e.g., away from the face of the user).

[0146] Curved mirror 1724 may redirect light from direction 1742 to direction 1732. For example, if camera 1726 includes a light source (e.g., an LED ring), curved mirror 1724 may redirect light from the ring from camera 1726 to the face of the user. Additionally, curved mirror 1724 may redirect light from a direction opposite direction 1732 in a direction opposite direction 1742.

[0147] FIG. 18 is a conceptual diagram illustrating step in a planar- volume holographic-grating fabrication scheme according to various aspects of the present disclosure. For example, a holographic media may record (e.g.. in the polymer of the hologram) the master hologram with two coherent waves: G and p. o is based on the angle of the illuminator (e.g., VCSEL ring), p may be normal to the hologram plate. FIG. 18 is a conceptual diagram representing recording a master hologram with two collimated coherent plane waves (collimated laser beams).

[0148] To replicate a hologram, hologram film may be illuminated and placed on the top of the master hologram with a coherent place wave p normal to the master hologram. FIG. 19 is a conceptual diagram representing roll-to-roll low-cost hologram replication.

[0149] FIG. 20 is a conceptual diagram illustrating step in a curved-volume holographic-grating fabrication scheme according to various aspects of the presentdisclosure. For example, a holographic media may record (e.g., in the polymer of the hologram) record the master hologram with two coherent waves: G and p. o may be a divergent wave, p may be normal to the hologram plate.

[0150] To replicate a hologram, hologram film may be illuminated and placed on the top of the master hologram wi th a coherent place wave p normal to the master hologram. FIG. 21 is a conceptual diagram representing roll-to-roll low-cost hologram replication.

[0151] FIG. 22 is a conceptual diagram illustrating a first example illumination scheme for a planar holographic reflector. According to the first example illumination scheme, a VCSEL ring is located around the camera. The divergent angle of each VCSEL illuminator may be no larger than the hologram angular bandwidth. The center ray of the VCSEL illuminators may be identical to the o ray angle in the planar master hologram recording scheme.

[0152] FIG. 23 is a conceptual diagram illustrating a second example illumination scheme for a planar holographic reflector. According to the second example illumination scheme, a VCSEL ring is located around the display eyepiece. There is no constraint on the divergent angle of each VCSEL illuminator.

[0153] FIG. 24 is a conceptual diagram illustrating a first example illumination scheme for a curved holographic reflector. According to the first example illumination scheme, a VCSEL ring is located around the camera. The divergent angle of each VCSEL illuminator may be no larger than the hologram angular bandwidth. The center ray of all the VCSEL illuminator may be identical to the o ray angle in the curved master hologram recording scheme.

[0154] FIG. 25 is a conceptual diagram illustrating a second example illumination scheme for a curved holographic reflector. According to the second example illumination scheme, a VCSEL ring is located around the display eyepiece. There is no constraint on the divergent angle of each VCSEL illuminator.

[0155] FIG. 26 illustrates an example computing-device architecture 2600 of an example computing device which can implement the various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR)device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or computing device of a vehicle), or other device. For example, the computing-device architecture 2600 may include, implement, or be included in any or all of XR system 100 of FIG. 1. XR system 200 of FIG. 2. apparatus 300 of FIG. 3A. apparatus 310 of FIG. 3B, system 600 of FIG. 6, system 700 of FIG. 7, system 800 of FIG. 8, system 900 of FIG. 9, system 1000 of FIG. 10, and / or other devices, modules, or systems described herein. Additionally or alternatively, computing-device architecture 2600 may be configured to perform process described herein.

[0156] The components of computing-device architecture 2600 are shown in electrical communication with each other using connection 2612, such as a bus. The example computing-device architecture 2600 includes a processing unit (CPU or processor) 2602 and computing device connection 2612 that couples various computing device components including computing device memon 2610, such as read only memory (ROM) 2608 and random-access memory7(RAM) 2606, to processor 2602.

[0157] Computing-device architecture 2600 can include a cache of high-speed memory connected directly with, in close proximity7to, or integrated as part of processor 2602. Computing-device architecture 2600 can copy data from memory72610 and / or the storage device 2614 to cache 2604 for quick access by processor 2602. In this way, the cache can provide a performance boost that avoids processor 2602 delays while waiting for data. These and other modules can control or be configured to control processor 2602 to perform various actions. Other computing device memory72610 may be available for use as well. Memory 2610 can include multiple different types of memory7with different performance characteristics. Processor 2602 can include any general-purpose processor and a hardware or software service, such as service 1 2616, service 2 2618, and service 3 2620 stored in storage device 2614, configured to control processor 2602 as well as a special-purpose processor where software instructions are incorporated into the processor design. Processor 2602 may be a self-contained system, containing multiple cores or processors, a bus. memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0158] To enable user interaction with the computing-device architecture 2600. input device 2622 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motioninput, speech and so forth. Output device 2624 can also be one or more of a number of output mechanisms known to those of skill in the art, such as a display, projector, television, speaker device, etc. In some instances, multimodal computing devices can enable a user to provide multiple types of input to communicate with computing-device architecture 2600. Communication interface 2626 can generally govern and manage the user input and computing device output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0159] Storage device 2614 is a non-volatile memory and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory’ devices, digital versatile disks, cartridges, random-access memories (RAMs) 2606, read only memory (ROM) 2608, and hybrids thereof. Storage device 2614 can include services 2616, 2618, and 2620 for controlling processor 2602. Other hardware or software modules are contemplated. Storage device 2614 can be connected to the computing device connection 2612. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary’ hardware components, such as processor 2602, connection 2612, output device 2624. and so forth, to carry out the function.

[0160] The term “substantially,” in reference to a given parameter, property, or condition, may refer to a degree that one of ordinary’ skill in the art would understand that the given parameter, property, or condition is met with a small degree of variance, such as, for example, within acceptable manufacturing tolerances. By way of example, depending on the particular parameter, property, or condition that is substantially met, the parameter, property, or condition may be at least 90% met, at least 95% met, or even at least 99% met.

[0161] Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) including or coupled to one or more active depth sensing systems. While described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to specific devices.

[0162] The term “device” is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system and so on). As used herein, a device may be any electronic device with one or more parts that may implement at least some portions of this disclosure. While the below description and examples use the term “device” to describe various aspects of this disclosure, the term “device” is not limited to a specific configuration, type, or number of objects. Additionally, the term “system” is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. While the below description and examples use the term “system” to describe various aspects of this disclosure, the term “system” is not limited to a specific configuration, type, or number of objects.

[0163] Specific details are provided in the description above to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by one of ordinary skill in the art that the aspects may be practiced without these specific details. For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks including devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the aspects.

[0164] Individual aspects may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.

[0165] Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general-purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

[0166] The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and / or data. A computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, magnetic or optical disks, USB devices provided with non-volatile memory, networked storage devices, any suitable combination thereof, among others. A computer-readable medium may have stored thereon code and / or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data. etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.

[0167] In some aspects the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0168] Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware descriptionlanguages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

[0169] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.

[0170] In the foregoing description, aspects of the application are described with reference to specific aspects thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, while illustrative aspects of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are. accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate aspects, the methods may be performed in a different order than that described.

[0171] One of ordinary skill will appreciate that the less than C'<”) and greater than (“>”) symbols or terminology used herein can be replaced with less than or equal to (‘"<”) and greater than or equal to (“>”) symbols, respectively, without departing from the scope of this description.

[0172] Where components are described as being “configured to” perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.

[0173] The phrase “coupled to” refers to any component that is physically connected to another component either directly or indirectly, and / or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and / or other suitable communication interface) either directly or indirectly.

[0174] Claim language or other language reciting “at least one of’ a set and / or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any duplicate information or data (e.g., A and A, B and B, C and C, A and A and B, and so on), or any other ordering, duplication, or combination of A, B, and C. The language “at least one of’ a set and / or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” may mean A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases “at least one” and “one or more” are used interchangeably herein.

[0175] Claim language or other language reciting “at least one processor configured to,” “at least one processor being configured to,” “one or more processors configured to,” “one or more processors being configured to,” or the like indicates that one processor or multiple processors (in any combination) can perform the associated operation(s). For example, claim language reciting “at least one processor configured to: X, Y, and Z” means a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each tasked with a certain subset of operations X, Y, and Z such that together the multiple processors perform X, Y. and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claimlanguage reciting “at least one processor configured to: X, Y, and Z” can mean that any single processor may only perform at least a subset of operations X, Y, and Z.

[0176] Where reference is made to one or more elements performing functions (e.g., steps of a method), one element may perform all functions, or more than one element may collectively perform the functions. When more than one element collectively performs the functions, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in whole by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, where reference is made to one or more elements configured to cause another element (e.g.. an apparatus) to perform functions, one element may be configured to cause the other element to perform all functions, or more than one element may collectively be configured to cause the other element to perform the functions.

[0177] Where reference is made to an entity (e.g., any entity or device described herein) performing functions or being configured to perform functions (e.g., steps of a method), the entity may be configured to cause one or more elements (individually or collectively) to perform the functions. The one or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more (or all) of the functions, and / or any combination thereof. Where reference to the entity performing functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to collectively perform the functions. When the entity is configured to cause more than one component to collectively perform the functions, each function need not be performed by each of those components (e.g., different functions may be performed by different components) and / or each function need not be performed in whole by only one component (e.g., different components may perform different sub-functions of a function).

[0178] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally interms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0179] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general-purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer- readable data storage medium including program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random-access memory (RAM) such as synchronous dynamic random-access memory (SDRAM), read-only memory' (ROM), non-volatile randomaccess memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer- readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0180] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or statemachine. A processor may also be implemented as a combination of computing devices, such as, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.

[0181] Illustrative aspects of the disclosure include:

[0182] Aspect 1. An apparatus for capturing facial images, the apparatus comprising: a display configured for placement proximate to an eye of a user; a camera configured to capture images of the eye; and a mirror positioned between the camera and the display, the mirror configured to redirect light from the eye to the camera.

[0183] Aspect 2. The apparatus of aspect 1. wherein the mirror is configured to expand a field of view of the camera.

[0184] Aspect 3. The apparatus of any one of aspects 1 or 2, wherein the mirror is configured to focus light from a face of the user toward the camera.

[0185] Aspect 4. The apparatus of any one of aspects 1 to 3, wherein the mirror is virtually curved.

[0186] Aspect 5. The apparatus of any one of aspects 1 to 4, wherein the mirror comprises a holographic mirror.

[0187] Aspect 6. The apparatus of any one of aspects 1 to 5, wherein the mirror comprises a photopolymer.

[0188] Aspect 7. The apparatus of any one of aspects 1 to 6, wherein the mirror comprises a photorefractive glass.

[0189] Aspect 8. The apparatus of any one of aspects 1 to 7, wherein the mirror comprises a diffraction grating.

[0190] Aspect 9. The apparatus of any one of aspects 1 to 8, wherein the mirror comprises a Bragg grating.

[0191] Aspect 10. The apparatus of any one of aspects 1 to 9, wherein the mirror has an angular bandwidth, and wherein the mirror is positioned relative to the eye and the camera based on the angular bandwidth.

[0192] Aspect 11. The apparatus of any one of aspects 1 to 10, wherein the mirror has a spectral bandwidth.

[0193] Aspect 12. The apparatus of any one of aspects 1 to 11, wherein the mirror is flat.

[0194] Aspect 13. The apparatus of any one of aspects 1 to 12, wherein the mirror is fitted to a flat surface between the display and the eye.

[0195] Aspect 14. The apparatus of any one of aspects 1 to 13, wherein the mirror is fitted to a curved surface between the display and the eye and wherein the mirror is configured to be fitted to the curved surface.

[0196] Aspect 15. The apparatus of any one of aspects 1 to 14, further comprising a light source configured to illuminate the eye.

[0197] Aspect 16. The apparatus of aspect 15, wherein the light source comprises at least one vertical-cavity surface-emitting laser (VCSEL).

[0198] Aspect 17. The apparatus of any one of aspects 15 or 16, wherein the light source comprises at least one edge emitting laser (EEL).

[0199] Aspect 18. The apparatus of any one of aspects 15 to 17, wherein the mirror has a spectral bandwidth and the at least one light source is configured to emit light within the spectral bandwidth.

[0200] Aspect 19. The apparatus of any one of aspects 15 to 18, wherein the light source is configured to emit light outside the visible spectrum.

[0201] Aspect 20. The apparatus of any one of aspects 15 to 19, wherein the light source is configured to emit light in the near infrared (NIR) spectrum.

[0202] Aspect 21 . The apparatus of any one of aspects 15 to 20, wherein the mirror is configured to reflect light from the light source to the eye.

[0203] Aspect 22. An apparatus for capturing facial images, the apparatus comprising: a display configured for placement proximate to an eye of a user; a camera configured to capture images of the eye; and a volume hologram positioned between the camera and the display, the volume hologram configured to redirect light from the eye to the camera, wherein the volume hologram is virtually curved to expand a field of view of the camera.

[0204] Aspect 23. The apparatus of aspect 22, wherein the volume hologram is configured to redirect light from a face of the user toward the camera.

[0205] Aspect 24. The apparatus of any one of aspects 22 or 23, wherein the volume hologram comprises a photopolymer.

[0206] Aspect 25. The apparatus of any one of aspects 22 to 24, wherein the volume hologram comprises a photorefractive material.

[0207] Aspect 26. The apparatus of any one of aspects 22 to 25, wherein the volume hologram comprises a diffraction grating.

[0208] Aspect 27. The apparatus of any one of aspects 22 to 26, wherein the volume hologram comprises a Bragg grating.

[0209] Aspect 28. The apparatus of any one of aspects 22 to 27, wherein the volume hologram has an angular bandwidth, and wherein the volume hologram is positioned relative to the eye and the camera based on the angular bandwidth.

[0210] Aspect 29. The apparatus of any one of aspects 22 to 28, wherein the volume hologram has a spectral bandwidth.

[0211] Aspect 30. The apparatus of any one of aspects 22 to 29, wherein the volume hologram is flat.

[0212] Aspect 31. The apparatus of any one of aspects 22 to 30, wherein the volume hologram is fitted to a flat surface between the display and the eye.

[0213] Aspect 32. The apparatus of any one of aspects 22 to 31, wherein the volume hologram is fitted to a curved surface between the display and the eye and wherein the volume hologram is configured to be fitted to the curved surface.

[0214] Aspect 33. The apparatus of any one of aspects 22 to 32, further comprising a light source configured to illuminate the eye.

[0215] Aspect 34. The apparatus of aspect 33, wherein the light source comprises at least one vertical -cavity surface-emitting laser (VCSEL).

[0216] Aspect 35. The apparatus of aspect 33, wherein the light source comprises at least one edge emitting laser (EEL).

[0217] Aspect 36. The apparatus of any one of aspects 33 to 35, wherein the volume hologram has a spectral bandwidth and the light source is configured to emit light within the spectral bandwidth.

[0218] Aspect 37. The apparatus of any one of aspects 33 to 36, wherein the light source is configured to emit light outside the visible spectrum.

[0219] Aspect 38. The apparatus of aspect 36, wherein the volume hologram is configured to reflect light from the light source to the eye.

[0220] Aspect 39. A method for capturing facial images, the method comprising: displaying an image at a display placed proximate to an eye of a user; capturing an image of the eye using a camera; and redirecting light from the eye to the camera with a volume hologram positioned between the camera and the display, wherein the volume hologram is virtually curved to expand a field of view of the camera.

[0221] Aspect 40. The method of aspect 39, wherein the volume hologram comprises a diffraction grating.

[0222] Aspect 41. The method of any one of aspects 39 or 40, wherein the volume hologram comprises a Bragg grating.

[0223] Aspect 42. The method of aspect 39. according to any of aspects 23 to 38.

[0224] Aspect 43. An apparatus for capturing facial images, wherein the apparatus is configured to be worn on a head of a user, wherein the apparatus comprises: a substrate with a surface facing in a first direction, wherein when the apparatus is worn on the head of the user, the first direction is toward the face of the user; a camera facing in a second direction when the apparatus is worn on the head of the user, the second direction is awayfrom the face of the user; and a volume hologram positioned between the camera and the substrate, wherein the volume hologram is configured to redirect light passing through a first area toward a second area, wherein the first area is larger than the second area.

[0225] Aspect 44. The apparatus of aspect 43, wherein: the volume hologram is configured to redirect light from a third direction to a fourth direction, the third direction is opposite the first direction, and the fourth direction is opposite the second direction.

[0226] Aspect 45. The apparatus of any one of aspects 43 or 44, wherein: the first area is offset from the substrate at a distance, and the second area is offset from the substrate at the distance.

[0227] Aspect 46. The apparatus of any one of aspects 43 to 45, wherein when the apparatus is worn on the head of the user, the first area includes a portion of the face of the user, and the second area includes an aperture of the camera.

[0228] Aspect 47. The apparatus of any one of aspects 43 to 46, wherein the volume hologram is virtually curved.

[0229] Aspect 48. The apparatus of any one of aspects 43 to 47, wherein the volume hologram is virtually curved to focus light from the first area towards the camera.

[0230] Aspect 49. The apparatus of any one of aspects 43 to 48, wherein the substrate comprises a lens configured to focus light from a display (e.g., of the apparatus) in the first direction.

[0231] Aspect 50. The apparatus of any one of aspects 43 to 49, wherein the substrate comprises a display configured to emit light in the first direction.

[0232] Aspect 51. The apparatus of any one of aspects 43 to 50, wherein the volume hologram comprises a photopolymer.

[0233] Aspect 52. The apparatus of any one of aspects 43 to 51, wherein the volume hologram comprises a photorefractive material.

[0234] Aspect 53. The apparatus of any one of aspects 43 to 52, wherein the volume hologram comprises a diffraction grating.

[0235] Aspect 54. The apparatus of any one of aspects 43 to 53, wherein the volume hologram comprises a Bragg grating.

[0236] Aspect 55. The apparatus of any one of aspects 43 to 54, wherein the volume hologram has an angular bandwidth, and wherein the volume hologram is positioned relative to the eye and the camera based on the angular bandwidth.

[0237] Aspect 56. The apparatus of any one of aspects 43 to 55, wherein the volume hologram has a spectral bandwidth.

[0238] Aspect 57. The apparatus of any one of aspects 43 to 56, wherein the volume hologram is flat.

[0239] Aspect 58. The apparatus of any one of aspects 43 to 57, wherein the volume hologram is fitted to a flat surface between a display (e.g., of the apparatus) and an eye of the user.

[0240] Aspect 59. The apparatus of any one of aspects 43 to 58, wherein the volume hologram is fitted to a curved surface between a display (e.g., of the apparatus) and an eye of the user and wherein the volume hologram is configured to be fitted to the curved surface.

[0241] Aspect 60. The apparatus of any one of aspects 43 to 59. further comprising a light source configured to illuminate the eye.

[0242] Aspect 61. The apparatus of aspect 60, wherein the volume hologram has a spectral bandwidth and the light source is configured to emit light within the spectral bandwidth.

[0243] Aspect 62. The apparatus of aspect 61, wherein the volume hologram is configured to reflect light from the light source to the eye.

[0244] Aspect 63. A method for capturing facial images, the method comprising: redirecting light passing through a first area toward a second area at a volume hologram, wherein the first area is larger than the second area, wherein the volume hologram is positioned between a camera of an apparatus and a substrate of the apparatus, wherein the substrate comprises a surface facing in a first direction, wherein when the apparatus isworn on the head of the user, the first direction is toward the face of the user, wherein camera faces in a second direction, and wherein when the apparatus is worn on the head of the user, the second direction is away from the face of the user, and capturing images using the camera.

[0245] Aspect 64. The method of aspect 63, wherein: the volume hologram is configured to redirect light from a third direction to a fourth direction, the third direction is opposite the first direction, and the fourth direction is opposite the second direction.

[0246] Aspect 65. The method of any one of aspects 63 or 64, wherein: the first area is offset from the substrate at a distance, and the second area is offset from the substrate at the distance.

[0247] Aspect 66. The method of any one of aspects 63 to 65, wherein when the apparatus is worn on the head of the user, the first area includes a portion of the face of the user, and the second area includes an aperture of the camera.

[0248] Aspect 67. The method of any one of aspects 63 to 66, wherein the volume hologram is virtually curved.

[0249] Aspect 68. The method of any one of aspects 63 to 67, wherein the volume hologram is virtually cun ed to focus light from the first area tow ards the camera.

[0250] Aspect 69. The method of any one of aspects 63 to 68, wherein the substrate comprises a lens configured to focus light from a display in the first direction.

[0251] Aspect 70. The method of any one of aspects 63 to 69, wherein the substrate comprises a display configured to emit light in the first direction.

[0252] Aspect 71. The method of any one of aspects 63 to 70, wherein the volume hologram comprises a photopolymer.

[0253] Aspect 72. The method of any one of aspects 63 to 71, wherein the volume hologram comprises a photorefractive material.

[0254] Aspect 73. The method of any one of aspects 63 to 72. wherein the volume hologram comprises a diffraction grating.

[0255] Aspect 74. The method of any one of aspects 63 to 73, wherein the volume hologram comprises a Bragg grating.

[0256] Aspect 75. The method of any one of aspects 63 to 74, wherein the volume hologram has an angular bandwidth, and wherein the volume hologram is positioned relative to the eye and the camera based on the angular bandwidth.

[0257] Aspect 76. The method of any one of aspects 63 to 75, wherein the volume hologram has a spectral bandwidth.

[0258] Aspect 77. The method of any one of aspects 63 to 76. wherein the volume hologram is flat.

[0259] Aspect 78. The method of any one of aspects 63 to 77, wherein the volume hologram is fitted to a flat surface between the display and the eye.

[0260] Aspect 79. The method of any one of aspects 63 to 78, wherein the volume hologram is fitted to a curved surface between the display and the eye and wherein the volume hologram is configured to be fitted to the curved surface.

[0261] Aspect 80. The method of any one of aspects 63 to 79, wherein the apparatus further comprising a light source configured to illuminate the eye.

[0262] Aspect 81. The method of aspect 80. wherein the volume hologram has a spectral bandwidth and the light source is configured to emit light within the spectral bandwidth.

[0263] Aspect 82. The method of aspect 81. wherein the volume hologram is configured to reflect light from the light source to the eye.

[0264] Aspect 83. A method comprising: displaying image data at a display of an apparatus; and receiving image data from a camera of the apparatus, wherein the apparatus is configured to be worn on a head of a user, wherein the apparatus comprises: a substrate with a surface facing in a first direction, wherein when the apparatus is worn on the head of the user, the first direction is toward the face of the user; the camera facing in a second direction when the apparatus is worn on the head of the user, the second direction is away from the face of the user; and a volume hologram positioned betw eenthe camera and the substrate, wherein the volume hologram is configured to redirect light passing through a first area toward a second area, wherein the first area is larger than the second area.

[0265] Aspect 84. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: display image data at a display of an apparatus; and receive image data from a camera of the apparatus, wherein the apparatus is configured to be worn on a head of a user, wherein the apparatus comprises: a substrate with a surface facing in a first direction, wherein when the apparatus is worn on the head of the user, the first direction is toward the face of the user; the camera facing in a second direction when the apparatus is worn on the head of the user, the second direction is away from the face of the user; and a volume hologram positioned between the camera and the substrate, wherein the volume hologram is configured to redirect light passing through a first area toward a second area, wherein the first area is larger than the second area.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. An apparatus for capturing facial images, wherein the apparatus is configured to be worn on a head of a user, wherein the apparatus comprises: a substrate with a surface facing in a first direction, wherein when the apparatus is worn on the head of the user, the first direction is toward a face of the user; a camera facing in a second direction when the apparatus is worn on the head of the user, the second direction is away from the face of the user; and a volume hologram positioned between the camera and the substrate, wherein the volume hologram is configured to redirect light passing through a first area toward a second area, wherein the first area is larger than the second area.

2. The apparatus of claim 1, wherein: the volume hologram is configured to redirect light from a third direction to a fourth direction, the third direction is opposite the first direction, and the fourth direction is opposite the second direction.

3. The apparatus of claim 1. wherein: the first area is offset from the substrate at a distance, and the second area is offset from the substrate at the distance.

4. The apparatus of claim 1. wherein when the apparatus is worn on the head of the user, the first area includes a portion of the face of the user, and the second area includes an aperture of the camera.

5. The apparatus of claim 1, wherein the volume hologram is virtually curved.

6. The apparatus of claim 1 , wherein the volume hologram is virtually curved to focus light from the first area towards the camera.

7. The apparatus of claim 1, wherein the substrate comprises a lens configured to focus light from a display in the first direction.

8. The apparatus of claim 1, wherein the substrate comprises a display configured to emit light in the first direction.

9. The apparatus of claim 1. wherein the volume hologram comprises a photopolymer.

10. The apparatus of claim 1, wherein the volume hologram comprises a photorefractive material.

11. The apparatus of claim 1. wherein the volume hologram comprises a diffraction grating.

12. The apparatus of claim 1. wherein the volume hologram comprises a Bragg grating.

13. The apparatus of claim 1, wherein the volume hologram has an angular bandwidth, and wherein the volume hologram is positioned relative to an eye of the user and the camera based on the angular bandwidth.

14. The apparatus of claim 1, wherein the volume hologram has a spectral bandwidth.

15. The apparatus of claim 1. wherein the volume hologram is flat.

16. The apparatus of claim 1, wherein the volume hologram is fitted to a flat surface between a display and an eye of the user.

17. The apparatus of claim 1, wherein the volume hologram is fitted to a curved surface between a display and an eye of the user and wherein the volume hologram is configured to be fitted to the curved surface.

18. The apparatus of claim 1, further comprising a light source configured to illuminate an eye of the user.

19. The apparatus of claim 18. wherein the volume hologram has a spectral bandwidth and the light source is configured to emit light within the spectral bandwidth.

20. The apparatus of claim 19, wherein the volume hologram is configured to reflect light from the light source to the eye.

Images