Hybrid gaze tracking circuit

Abstract

The invention relates to a hybrid gaze tracking circuit. An eyewear, such as a head-mounted device, may include an adjustable prescription lens and / or may include a display. The eyewear may include a gaze tracking circuit that tracks a gaze direction of a user. The gaze tracking circuit may include a camera, a light source, and a range finder. The light source may be a light emitting diode that generates an eye flash on the user's eye. The camera may capture an eye image including the image of the flash. The pupil position and the flash position may be determined based on the captured image. The range finder may measure an eye distance to an eye of the user. The gaze direction may be determined based on the pupil position, the flash position, and the eye distance, even when the camera captures only one eye flash. The range finder may be an ultrasonic range finder, an optical range finder, or any other suitable range finder.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 975,954, filed on December 10, 2024, and U.S. Provisional Patent Application No. 63 / 626,390, filed on January 29, 2024, the entire contents of which are hereby incorporated by reference. Technical Field

[0002] The present invention generally relates to electronic devices, and more particularly to wearable electronic devices such as head-mounted devices. Background Art

[0003] Head-mounted devices and other eye-wearables may use gaze tracking circuitry to track a user's gaze.

[0004] Designing a gaze tracking circuitry with satisfactory performance can be challenging. If not careful, the gaze tracking circuitry may produce inaccurate measurement results, or may exhibit other performance limitations, such as excessive power consumption. Summary of the Invention

[0005] Eye-wearables such as head-mounted devices may include adjustable prescription lenses and / or may include a display. The lens and the display may be mounted to a support structure, such as a support frame or other head-mounted support structure.

[0006] The eye-wearable may include gaze tracking circuitry for tracking a user's gaze direction. The gaze tracking circuitry may include a camera, a light source, and a rangefinder. The light source may be a light-emitting diode that produces an eye flash on the user's eye. The camera may capture an eye image including the flash. The pupil position and the flash position may be determined based on the captured image.

[0007] The rangefinder may measure the eye distance to the user's eye. The gaze direction may be determined based on the pupil position, the flash position, and the eye distance, even when the camera captures only a single eye flash. The rangefinder may be an ultrasonic rangefinder, an optical rangefinder, or any other suitable rangefinder.

[0008] If desired, the camera and the light source may be maintained in a low-power state until the rangefinder detects a change in the eye distance (which may indicate a change in the gaze direction). In response to detecting a change in the eye distance, the light source and the camera may be turned on to measure the pupil position and the flash position, such that a new gaze direction may be determined based on the new eye distance. Brief Description of the Drawings

[0009] Figure 1 is a top view of an exemplary head-mounted device that may include gaze tracking circuitry, according to one embodiment.

[0010] Figure 2Rear view of an exemplary head-mounted device that may include a gaze tracking circuit, according to one embodiment.

[0011] Figure 3 Schematic diagram of an exemplary head-mounted device that may include a gaze tracking circuit, according to one embodiment.

[0012] Figure 4 Top view of an exemplary gaze tracking circuit for tracking a user's gaze direction, according to one embodiment.

[0013] Figure 5 Side view of a portion of an exemplary head-mounted device having an optical module that includes a display, a lens, and a gaze tracking circuit, according to one embodiment.

[0014] Figure 6 Top view of an exemplary rangefinder that includes an ultrasonic transducer array, according to one embodiment.

[0015] Figure 7 Side view of an exemplary rangefinder that includes a self-mixing interferometer, according to one embodiment.

[0016] Figure 8 Diagram showing an exemplary quantitative model for determining a gaze direction using a rangefinder, a camera, and a light source, according to one embodiment.

[0017] Figure 9 Flowchart of exemplary steps for tracking a gaze direction involving the use of a rangefinder, a camera, and a light source, according to one embodiment. Detailed Description

[0018] Eye-wear such as a pair of glasses or other head-mounted devices may include one or more eye-monitoring components such as a gaze tracking circuit. These components may include, for example, one or more cameras, one or more light sources, and one or more rangefinders (e.g., distance sensors). The light source may illuminate the user's eyes while the camera captures an image of the eyes. In an exemplary configuration, the light source may include a light-emitting diode that produces a flash on the user's eyes and / or illuminates the user's pupil and iris. The pupil position and the flash position may be determined based on the eye image captured by the camera. The rangefinder may include an optical rangefinder, an ultrasonic rangefinder, or any other suitable rangefinder, and may be used to determine the distance to the eyes (sometimes referred to as the eye distance). The user's gaze direction may be determined based on the position of the flash, the position of the pupil, and the distance to the eyes.

[0019] Using a hybrid gaze tracking circuit that includes both a camera and a light source for flash detection and a rangefinder for distance sensing can allow for determining the gaze direction using fewer flashes (e.g., only a single flash from a single light-emitting diode if desired). This can be beneficial in arrangements where fewer light sources are desired and / or in scenarios where multiple flashes cannot be obtained from a particular user's eyes. In some arrangements, the camera, light source, and rangefinder can remain powered on during operation of device 10 and can be used continuously or periodically for tracking the gaze direction. In other arrangements, power savings can be achieved by keeping the camera and / or light source off (or otherwise in a low-power state) until the rangefinder detects a change in the eye distance, which can indicate a change in the gaze direction. When a change in the eye distance is detected by the rangefinder, the camera and light source can be turned on to assist in determining the user's new gaze direction.

[0020] Figure 1 A top view of an exemplary head-mounted device or other eye-wear is shown. As Figure 1 shown, a head-mounted device such as electronic device 10 can have a head-mounted support structure such as housing 12. Housing 12 can include a portion (e.g., support structure 12T) for allowing the device 10 to be worn on a user's head. Support structure 12T can be formed of fabric, polymer, metal, and / or other materials. Support structure 12T can form a strap or other head-mounted support structure to help support device 10 on the user's head. The main support structure (e.g., main housing portion 12M) of housing 12 can support electronic components such as display 14. Main housing portion 12M can include a housing structure formed of metal, polymer, glass, ceramic, and / or other materials. For example, housing portion 12M can have a housing wall on the front face F and housing walls on adjacent top, bottom, left, and right side faces, which are formed of a rigid polymer or other rigid support structure, and these rigid walls can optionally be covered with electronic components, fabric, leather, or other soft materials, etc. The walls of housing portion 12M can enclose internal components 38 in the internal region 34 of device 10 and can separate the internal region 34 from the environment (external region 36) around device 10. Internal components 38 can include integrated circuits, actuators, batteries, sensors, and / or other circuits and structures for device 10. Housing 12 can be configured to be worn on a user's head and can form glasses, a hat, a helmet, goggles, and / or other head-mounted devices. In this document, the configuration in which housing 12 forms goggles is sometimes described as an example.

[0021] The front face F of the housing 12 can face outward away from the user's head and face. The opposite back face R of the housing 12 can face the user. The portion of the housing 12 located on the back face R (e.g., a portion of the main housing 12M) can form a covering member, such as the covering member 12C (sometimes referred to as a shade). The presence of the covering member 12C on the back face R can help hide the internal housing structure, internal components 38, and other structures in the internal area 34 from being seen by the user.

[0022] The device 10 can have a left optical module and a right optical module 40. Each optical module can include a corresponding display 14, a lens 30, and a support structure 32. The support structure 32, which can sometimes be referred to as a lens barrel or an optical module support structure, can include a hollow cylindrical structure with an open end or other support structures for accommodating the display 14 and the lens 30. The support structure 32 can include, for example, a left lens barrel that supports the left display 14 and the left lens 30 and a right lens barrel that supports the right display 14 and the right lens 30.

[0023] The display 14 can include a pixel array or other display devices to generate images. The display 14 can include, for example, organic light-emitting diode pixels formed on a substrate with thin-film circuits and / or formed on a semiconductor substrate, pixels formed by crystalline semiconductor die, liquid crystal display pixels, scanned display devices, and / or other display devices for generating images.

[0024] The lens 30 can include one or more lens elements for providing image light from the display 14 to the corresponding eye zone 13. The lens 30 can be implemented using refractive glass lens elements, using mirror lens structures (reflection-refraction lenses), using Fresnel lenses, using holographic lenses, and / or other lens systems.

[0025] When the user's eyes are located in the eye zone 13, the displays (display panels) 14 operate together to form the display of the device 10 (e.g., the user's eyes can view the images provided by the corresponding left and right optical modules 40 in the eye zone 13, such that a stereoscopic image is created for the user). When the user views the display, the left image from the left optical module is fused with the right image from the right optical module.

[0026] If desired, device 10 may include additional lenses, such as lens 50. Lens 50 may be a fixed lens or may be an adjustable lens, such as a liquid crystal lens, a fluid-filled lens, or other suitable adjustable lens. Lens 50 may be configured to accommodate different focal length ranges and / or correct vision defects, such as myopia, hyperopia, presbyopia, astigmatism, higher order aberrations, and / or other vision defects. For example, lens 50 may be an adjustable prescription lens having a first set of optical characteristics for a first user having a first prescription and a second set of optical characteristics for a second user having a second prescription. Lens 50 may be removably or permanently attached to housing 12. In an arrangement where lens 50 is removable, lens 50 may have mating engagement features, magnets, clamps, or other attachment structures that allow lens 50 to be attached to housing 12 (e.g., individually or in pairs).

[0027] If desired, device 10 may be used solely for vision correction (e.g., device 10 may be a pair of spectacles, glasses, etc.), and some of the other components, such as display 14, lens 30, and optical module 40, may be omitted Figure 1 In other arrangements, device 10 (sometimes referred to as an eye-wear 10, glasses 10, a head-mounted device 10, etc.) may include a display for presenting virtual reality, mixed reality, and / or augmented reality content. With this type of arrangement, lens 50 may be a prescription lens and / or may be used to move content between focal planes from the perspective of the user. If desired, lens 50 may be omitted. Arrangements where device 10 is a head-mounted device having one or more displays are sometimes described herein as illustrative examples.

[0028] It may be desirable to monitor a user's eyes when the user's eyes are located within the comfortable viewing zone 13. For example, it may be desirable to use a camera to capture an image of the user's iris (or other part of the user's eye) for user authentication. It may also be desirable to monitor the direction of the user's gaze. Gaze tracking information may be used as a form of user input and / or may be used to determine where within an image the image content resolution should be locally enhanced in a foveated imaging system. To ensure that device 10 can capture a satisfactory eye image when the user's eyes are located within the comfortable viewing zone 13, each optical module 40 may be provided with a gaze tracking circuit 62. Gaze tracking circuit 62 may include one or more cameras (such as camera 42), one or more light sources (such as light source 44 (e.g., a light-emitting diode, a laser, a lamp, etc.)), and one or more rangefinders (such as rangefinder 48).

[0029] The camera 42 and the light-emitting diode 44 can operate at any suitable wavelength (visible light, infrared light, and / or ultraviolet light). In the exemplary configurations sometimes described herein by way of example, the diode 44 emits infrared light that is invisible (or nearly invisible) to the user. This allows the eye monitoring operation to be continuously performed without interfering with the user's ability to view the image on the display 14.

[0030] The rangefinder 48 (sometimes referred to as the depth sensor 48) can be any suitable rangefinder, such as an optical rangefinder (e.g., a light source and a light sensor that acquire time-of-flight measurements, phase-based measurements, self-mixing sensors, light detection and ranging (lidar) sensors, structured light sensors, and / or depth sensors based on stereoscopic imaging devices that capture three-dimensional images), an ultrasonic rangefinder (e.g., one or more capacitive micromachined ultrasonic transducers, piezoelectric micromachined transducers, and / or other suitable ultrasonic transducers for emitting and / or detecting acoustic signals), and / or any other suitable rangefinder.

[0031] Not all users have the same interpupillary distance IPD. To provide the device 10 with the ability to adjust the interpupillary distance between the modules 40 along the lateral dimension X and thus adjust the distance IPD between the eye comfort zones 13 to accommodate different user interpupillary distances, the device 10 can be provided with an actuator 43. The actuator 43 can be a manually controlled and / or computer-controlled actuator (e.g., a computer-controlled motor) for moving the support structures 32 relative to each other. Information about the position of the user's eyes can be acquired using, for example, the camera 42. The position of the eye comfort zones 13 can then be adjusted accordingly.

[0032] As Figure 2 shown, the cover 12C can cover the back F while leaving the lens 30 of the optical module 40 uncovered (e.g., the cover 12C can have an opening that aligns with and receives the module 40). When the modules 40 move relative to each other along the dimension X to accommodate different pupil distances of different users, the modules 40 move relative to the walls of a fixed housing structure such as the main portion 12M and relative to each other.

[0033] Figure 3 shows a schematic diagram of an exemplary electronic device such as a head-mounted device or other wearable device. Figure 3 The device 10 can operate as a stand-alone device and / or the resources of the device 10 can be used for communication with external electronic equipment. For example, the communication circuitry in the device 10 can be used to send user input information, sensor information, and / or other information to an external electronic device (e.g., wirelessly or via a wired connection). Each of these external devices can include Figure 3 components of the type shown in the device 10.

[0034] AsFigure 3 As shown, a head-mounted device such as device 10 may include control circuitry 20. The control circuitry 20 may include storage and processing circuitry for supporting the operation of device 10. The storage and processing circuitry may include storage devices such as non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid state drive), volatile memory (e.g., static or dynamic random access memory), etc. The processing circuitry in the control circuitry 20 may be used to acquire inputs from sensors and other input devices and may be used to control output devices. The processing circuitry may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors and other wireless communication circuitry, power management units, audio chips, application specific integrated circuits, etc. During operation, the control circuitry 20 may use the display 14 and other output devices to provide visual output and other output to the user.

[0035] To support communication between device 10 and external equipment, the control circuitry 20 may communicate using communication circuitry 22. The circuitry 22 may include an antenna, radio frequency transceiver circuitry, and other wireless communication circuitry and / or wired communication circuitry. The circuitry 22 (which may sometimes be referred to as control circuitry and / or control and communication circuitry) may support two-way wireless communication between device 10 and external equipment (e.g., a companion device such as a computer, cellular phone, or other electronic device, an accessory such as a pointing device, computer stylus, or other input device, a speaker, or other output device, etc.) via a wireless link. For example, the circuitry 22 may include radio frequency transceiver circuitry such as a wireless local area network transceiver circuitry configured to support communication via a wireless local area network link, a near field communication transceiver circuitry configured to support communication via a near field communication link, a cellular phone transceiver circuitry configured to support communication via a cellular phone link, or a transceiver circuitry configured to support communication via any other suitable wired or wireless communication link. For example, wireless communication may be supported via a link, a link, a wireless link operating at a frequency between 10 GHz and 400 GHz, a 60 GHz link or other millimeter wave link, a cellular phone link, or other wireless communication link. Device 10 (if desired) may include power circuitry for transmitting and / or receiving wired and / or wireless power and may include a battery or other energy storage device. For example, device 10 may include a coil and a rectifier to receive wireless power provided to the circuitry in device 10.

[0036] Device 10 may include input-output devices such as device 24. The input-output device 24 may be used to collect user input, to collect information about the user's surrounding environment, and / or to provide output to the user. The device 24 may include one or more displays such as display 14. The display 14 may include one or more display devices such as an organic light-emitting diode display panel (a panel having organic light-emitting diode pixels formed on a polymer substrate or a silicon substrate including pixel control circuits), a liquid crystal display panel, a microelectromechanical systems display (e.g., a two-dimensional mirror array or a scanning mirror display device), a display panel having a pixel array formed by crystal semiconductor light-emitting diode dies (sometimes referred to as micro-LEDs), and / or other display devices.

[0037] The sensor 16 in the input-output device 24 may include a force sensor (e.g., a strain gauge, a capacitive force sensor, a resistive force sensor, etc.), an audio sensor (such as a microphone), a touch and / or proximity sensor (such as a capacitive sensor, such as a touch sensor forming a button, a touchpad, or other input device), and other sensors. If desired, the sensor 16 may include an optical sensor (such as an optical sensor that emits and detects light), an ultrasonic sensor, an optical touch sensor, an optical proximity sensor, and / or other touch sensors and / or proximity sensors, a monochromatic and color ambient light sensor, an image sensor, a fingerprint sensor, an iris scan sensor, a retina scan sensor, and other biometric sensors, a temperature sensor, a sensor for measuring three-dimensional contactless gestures ("air gestures"), a pressure sensor, sensors for detecting position, orientation, and / or movement (e.g., an accelerometer, a magnetic sensor such as a compass sensor, a gyroscope, and / or an inertial measurement unit including some or all of these sensors), health sensors such as a blood oxygen sensor, a heart rate sensor, a blood flow sensor, and / or other health sensors, a radio frequency sensor, a depth sensor (e.g., a structured light sensor and / or a depth sensor based on a stereoscopic imaging device that captures three-dimensional images), an optical sensor such as a self-mixing sensor and a light detection and ranging (laser) sensor that acquires time-of-flight measurements, a humidity sensor, a moisture sensor, a gaze tracking sensor, an electromyography sensor that senses muscle activation, a face sensor, and / or other sensors. In some arrangements, the device 10 may use the sensor 16 and / or other input-output devices to collect user input. For example, a button may be used to collect button press input, a touch sensor overlapping the display may be used to collect user touch screen input, a touchpad may be used to collect touch input, a microphone may be used to collect audio input, an accelerometer may be used to monitor when a finger touches the input surface and thus may be used to collect finger press input, etc.

[0038] If desired, the electronic device 10 may include additional components (see, for example, other devices 18 in the input-output device 24). The additional components may include a haptic output device, an actuator for moving a movable housing structure, an audio output device such as a speaker, a light-emitting diode for a status indicator, a light source such as a light-emitting diode for illuminating a portion of the housing and / or the display structure, other optical output devices, and / or other circuitry for collecting input and / or providing output. The device 10 may also include a battery or other energy storage device, a connector port for supporting wired communication with auxiliary equipment and for receiving wired power, and other circuitry.

[0039] Figure 4 is a top view of an exemplary gaze tracking circuit 62. As Figure 4 shown, the gaze tracking circuit 62 may include one or more cameras (such as camera 42), one or more light sources (such as light source 44), and one or more rangefinders (such as rangefinder 48). The rangefinder 48 may be configured to measure the distance to the eye 58 (e.g., the distance to the specular reflection point on the cornea, sometimes referred to as the eye distance). As the user's gaze moves around (e.g., from position P1 to position P2), the distance to the eye 58 may change. At position P1, the rangefinder 48 may measure the distance D1 to the eye 58. At position P2, the rangefinder 48 may measure the distance D2 to the eye 58.

[0040] The rangefinder 48 may include one or more transmitters such as transmitter 48T and one or more receivers such as receiver 48R. The transmitter 48T may be configured to transmit a signal 56 toward the user's eye 58. The signal 56 may be reflected from the eye 58, and the reflected signal 54 may be detected by the receiver 48R. If desired, the rangefinder 48 may include more than one transmitter 48T and / or more than one receiver 48R. For example, the rangefinder 48 may include a second transmitter 48T and a second receiver 48R for redundancy. Arrangements in which the rangefinder 48 includes three or more transmitters 48T and / or three or more receivers 48R may also be used.

[0041] In some arrangements, one device may be used as both the transmitter 48T and the receiver 48R. For example, a flexible membrane in a transducer may be used to detect an ultrasonic signal (when used as the receiver 48R) and may also be used to transmit an ultrasonic signal (when used as the transmitter 48T). As another example, a self-mixing interferometer may also be used as both the transmitter 48T and the receiver 48R.

[0042] The rangefinder 48 can be any suitable sensor configured to measure distance. In an arrangement where the rangefinder 48 is an optical sensor (e.g., an optical sensor that acquires time-of-flight measurements, a self-mixing sensor, a light detection and ranging (lidar) sensor, a structured light sensor, a phase-based optical coherence tomography sensor, and / or a depth sensor of a stereoscopic imaging device that captures three-dimensional images, etc.), the transmitted signal 56 and the reflected signal 54 can be optical signals. When the rangefinder 48 is formed by a phase-based sensor (such as an optical sensor based on optical coherence tomography), the rangefinder 48 can be configured to achieve a smaller resolvable time interval than a time-of-flight-based sensor. In an arrangement where the rangefinder 48 is an ultrasonic sensor (e.g., one or more capacitive micromachined ultrasonic transducers, piezoelectric micromachined transducers, and / or other suitable ultrasonic transducers for transmitting and / or detecting acoustic signals), the transmitted signal 56 and the reflected signal 54 can be ultrasonic signals.

[0043] If desired, the transmitter 48T can be co-located with the receiver 48R. In other arrangements, the transmitter 48T and the receiver 48R can be mounted at different locations. The camera 42 and the light source 44 can be co-located with each other or can be mounted at different locations. One or both of the transmitter 48T and the receiver 48R can be co-located with the camera 42 and / or the light source 44, or the transmitter 48T and the receiver 48R can be mounted separately from the camera 42 and / or the light source 44.

[0044] During operation, the light source 44 can be used to emit light 50 towards the eye 58. The light 50 can be reflected from the eye 58, and the reflected light 52 can be detected by the camera 42. The emitted light 50 can produce a flash on the eye 58. The camera 42 can capture an image of the eye 58 including the flash produced by the light 50. Based on the captured image, the gaze tracking circuit 62 can determine the position of the flash and the position of the user's pupil. In some arrangements, there can be multiple light sources 44 that produce multiple flashes on the user's eye. If a sufficient number of flashes are produced on the eye 58, the gaze control circuit 62 can determine the shape of the user's eye (e.g., the user's cornea), which in turn can be used to determine the gaze direction (e.g., without the need for the rangefinder 48).

[0045] In some arrangements, such as when fewer light sources 44 are desired, or when an insufficient number of flashes cannot be captured due to the shape of a particular user's eye, the gaze control circuit 62 can combine flash detection with distance sensing to determine the gaze direction. In this type of scenario, a single light source 44 can produce a single flash on the eye 58, and the camera 42 can capture an image of the eye 58 including the single flash. Based on the captured image, the gaze tracking control circuit 62 can determine the position of the flash and the position of the pupil. Since the eye is mostly spherical (e.g., first order), as the eye moves around, the flash on the eye 58 will mostly remain in the same position, but the position of the pupil relative to the flash will change as the gaze direction changes. In particular, as the eyeball moves around to different gaze directions (e.g., from position P1 to position P2), the position of the pupil relative to the flash will change by a scaling factor that depends on the distance to the eyeball. By using the rangefinder 48 to determine the distance to the eyeball at positions P1 and P2, the gaze tracking circuit 62 can determine this scaling factor, and thus can map the pupil and flash positions at P1 to a first gaze direction (e.g., based on distance D1), and map the pupil and flash positions at P2 to a second gaze direction (e.g., based on distance D2).

[0046] The device 10 can include a gaze tracking circuit 62 for each eye 58 (e.g., left eye 58 and right eye 58), or the device 10 can include a gaze tracking circuit 62 for a single eye 58. Figure 5 is a side view of a portion of the device 10 that shows an illustrative example of how a gaze tracking circuit 62 can be implemented in a device including a head-mounted display.

[0047] In Figure 5 the example, the gaze tracking circuit 62 is mounted in the optical module 40 of the device 10. This is merely illustrative. If desired, the gaze tracking circuit 62 can be mounted in other locations of the device 10. The optical module 40 can have a lens barrel 32. The lens 30 can be used to provide an image from the pixel P of the display 14 to the eye comfort zone 13 along the optical axis 60. To provide eye illumination for illuminating the eye located in the eye comfort zone 13, the module 40 can include one or more light sources (e.g., lasers, light-emitting diodes, lamps, etc.), such as one or more light-emitting diodes 44. One or more cameras 42 can be included in each optical module 40 to monitor the eye comfort zone 13. When the user's eye is located in the eye comfort zone 13, the camera 42 can capture an image of the user's eye.

[0048] The light-emitting diode 44 can emit light at one or more wavelengths of interest (e.g., visible light wavelengths and / or infrared light wavelengths, etc.), and the camera 42 can be sensitive to these wavelengths (e.g., visible light wavelengths and / or infrared light wavelengths, etc.). In an illustrative configuration, the light-emitting diode 44 emits infrared light. The infrared light can be used to illuminate the user's eyes in the eye comfort zone 13 in a situation where the user is not aware (or hardly aware) (e.g., because human vision is generally insensitive to infrared light unless the infrared light has an infrared wavelength close to the edge of the visible spectrum extending from 380 nm to 740 nm).

[0049] The electronic components in the module 40 (such as the display 14, the camera 42, the light-emitting diode 44, and the rangefinder 48) can be coupled to a flexible printed circuit or other substrates containing metal traces. The metal traces can form interconnect paths for carrying power signals, data signals, and control signals. As Figure 5 shown, for example, the light-emitting diode 44 can be mounted on an annular substrate such as the flexible printed circuit 46. The printed circuit 46 and the light-emitting diode 44 can extend around some or all of the inner perimeter of the lens barrel 32 (and thus around some or all of the outer perimeter of the display 14).

[0050] During operation, the light from the light-emitting diodes 44 mounted along the edge of the display 14 can travel through the lens 30 to the eye comfort zone 13. When the user is viewing an image presented by the pixel P array on the display 14, the light-emitting diodes 44 are typically outside or almost outside the user's field of view. Some of the light-emitting diodes 44 can produce flashes (e.g., reflections) leaving the surface of the user's eyes in the eye comfort zone 13. These reflections can be captured by the camera 42. The device 10 can process the flash information obtained by the camera 24 to track the user's gaze. For example, the control circuit 20 can analyze the positioning of the flashes to determine the shape of the user's eyes (e.g., the user's cornea). Based on this information, the control circuit 20 can determine the direction of the user's gaze.

[0051] In addition to being used as a flash light source (e.g., a light source for generating flash illumination that is detected as discrete eye flashes by the camera 42), if needed, the light from the light-emitting diode 44 can be used as overall eye illumination. In particular, the light from the light-emitting diode 44 can illuminate parts of each user's eyes (such as the user's iris and the user's pupil).

[0052] During operation, when light from diode 44 illuminates the user's pupil, camera 42 can capture an image of the pupil. The user's pupil will have a shape (e.g., oval) that varies according to the orientation of the user's eye relative to camera 42. For example, if the eye is aligned with camera 42, the pupil will appear circular or near-circular, while if the eye is angled away from camera 42, the pupil will have a higher eccentricity. By analyzing the shape of the pupil, control circuit 20 can determine the direction of the user's gaze.

[0053] It may also be desirable for camera 42 to capture other eye images, such as an image of the user's eye iris. The iris pattern is user-specific, so an iris image can be used to authenticate the user in device 10 (e.g., log the user into a user account, replace a username and / or password, or otherwise serve as a biometric credential for device 10).

[0054] Pupil illumination and illumination for the flash can be produced by light-emitting diode 44 at the same wavelength or at different wavelengths. For example, pupil illumination and flash illumination can be provided by light-emitting diode 44 at a wavelength of 940 nm, 800 nm to 1000 nm, at least 800 nm, at least 850 nm, at least 900 nm, at least 950 nm, less than 950 nm, or other suitable wavelengths. A configuration where the wavelengths of the flash illumination and the pupil illumination are long enough to be invisible to most or all users can help allow continuous flash measurements and pupil measurements and / or other gaze-tracking measurements during operation of device 10 without potentially distracting the user. Iris illumination can be provided by light-emitting diode 44 at the same wavelength and / or a different wavelength as the flash illumination and the pupil illumination. To obtain a desired image contrast when acquiring iris information, it may be desirable to provide iris illumination at a shorter wavelength than the pupil illumination and the flash illumination (e.g., at a visible light wavelength and / or at a shorter infrared wavelength than that used by diode 44 when providing gaze-tracking illumination). Camera 42 can include a single image sensor that captures pupil image data, flash image data, and iris image data, and / or multiple cameras can be provided, each capturing image data at a different wavelength (or band).

[0055] Capturing a sufficient number of flashes on each type of eye can be challenging. If needed, gaze-tracking circuit 62 can be a hybrid gaze tracker that combines flash detection with ranging (distance sensing) to determine the direction of the user's gaze using fewer flashes (e.g., as few as one flash on the eye). As combined with <0, Figure 4As discussed, using the rangefinder 48 to determine the distance to the user's eye allows the gaze tracking circuit 62 to determine the gaze direction using only a single light-emitting diode 44 and a single flash on the eye captured by the camera 42. Additionally, since it can be assumed that if the distance to the eye does not change, the gaze direction does not change, the gaze tracking circuit 62 can selectively activate the camera 42 and the light source 44 only when needed. For example, after determining the gaze direction using the light source 44, the camera 42, and the rangefinder 48, the gaze detection circuit 62 can turn off the light source 44 and the camera 42 (or can otherwise switch the light source 44 and the camera 42 to a low-power state), while continuing to use the rangefinder 48 to acquire distance measurements. If no change is detected in the distance to the eye, the gaze detection circuit 62 can assume that the previously determined gaze direction has not changed. If there is a change in the eye distance detected by the rangefinder 48, the light source 44 and the camera 42 can be activated (e.g., turned on) to illuminate the eye with the light source 44 while using the camera 42 to capture an image of the eye. Based on the new pupil and flash positions and the new distance to the eye measured by the rangefinder 48, the gaze tracking circuit 62 can determine the user's new gaze direction.

[0056] Figure 6 is a top view of an exemplary rangefinder 48 formed using ultrasonic transducers. As Figure 6 shown, the rangefinder 48 can include one or more ultrasonic transducer arrays 170, such as ultrasonic transducer 64. Each array 170 can include a plurality of ultrasonic transducers 64 on a substrate such as substrate 68. The ultrasonic transducers 64 within each array 170 can have different center frequencies such that when used together, the array can collectively achieve the positioning accuracy of a broadband ultrasonic transducer without experiencing the reduction in quality factor typically caused by a single broadband ultrasonic transducer. The arrays 170 can be mounted in the housing 12 of the device 10 and can be distributed at different positions around the user's eye (e.g., around the user's left and right eyes). Any suitable number of arrays 170 (e.g., one, two, three, four, five, six, ten, fifteen, twenty, more than twenty, less than twenty, etc.) can be present in the gaze tracking circuit 62, and each array 170 can include any suitable number of transducers 64 (e.g., five, eight, ten, fifteen, twenty, fifty, one hundred, two hundred, less than two hundred, more than two hundred, etc.).

[0057] One or more of the arrays 170 can be used to form Figure 4 the transmitter 48T and can transmit ultrasonic signals (e.g., Figure 4 the transmitted signal 56). The ultrasonic signal 56 can be reflected from the user's eye 58 (e.g., can be reflected from the cornea of the user's eye). One or more of the arrays 170 can form Figure 4The receiver 48R and can be used to detect the reflected ultrasonic signal 54 after the signal is reflected from the user's cornea. Using the time-of-flight measurement technique, the control circuit 20 can be used to determine the time it takes for the transmitted signal 56 to be reflected back from the eye 58, which in turn can be used to determine the distance to the eye 58 (e.g., the distance to the specular reflection point on the cornea). If desired, the same array 170 can be used for transmitting and detecting ultrasonic signals. Arrangements can also be used in which multiple arrays 170 transmit ultrasonic signals and / or multiple arrays 170 detect ultrasonic signals.

[0058] If desired, one or more of the arrays 170 can be phased transducer arrays. In a phased ultrasonic transducer array, beam steering techniques can be used, in which the phase and / or amplitude of the ultrasonic signal of each transducer in the array 170 is adjusted to perform beam steering. Beam steering can be used to "illuminate" a specific area of interest with ultrasonic signals and / or to illuminate other arrays 170 with calibration signals. Beam steering can also be used to avoid illuminating certain areas with ultrasonic signals (e.g., to avoid directly illuminating other arrays 170 and / or to avoid illuminating certain parts of the user's face). For example, the phased ultrasonic transducer array 170 can be configured to emit a concentrated beam of ultrasonic signals that strikes the cornea but not the user's eyebrows. This type of beam steering arrangement can help improve gaze tracking accuracy by avoiding detecting significant reflections from the surfaces around the user's eyes.

[0059] The use of the time-of-flight based measurement technique is merely illustrative. If desired, other time-based, amplitude-based, and / or phase-based measurement schemes (such as time difference of arrival measurement techniques, angle of arrival measurement techniques, triangulation methods, and / or other suitable measurement techniques) can be used to determine the position of the user's eye using the ultrasonic sensor array 170.

[0060] As Figure 6As shown, the transducer array 170 may include a plurality of transducers 64 on a substrate 68. The array 170 may include transducers 64 having different center frequencies. The center frequency of an ultrasonic transducer may refer to the frequency at the center of the frequency range in which the transducer can operate. For example, the array 170 may include transducers having two, three, four, five, six, seven, ten, more than ten, or fewer than ten center frequencies. The center frequency of the transducers 64 may be, for example, between 750 kHz and 1.25 MHz, between 500 kHz and 1.25 MHz, between 700 kHz and 1 MHz, between 800 kHz and 1.2 MHz, between 900 kHz and 1.1 MHz, between 750 kHz and 1.4 MHz, or between any other suitable frequency ranges. For a given center frequency, any suitable number (e.g., one, two, three, four, five, more than five, fewer than five) of transducers 64 may be present in the array 170. If desired, each transducer 64 may have one or more natural oscillation frequencies for the excitation or detection of ultrasonic waves.

[0061] The bandwidth of each individual transducer 64 may be less than the collective bandwidth spanned by all of the transducers 64 in the array 170. The center frequencies of the individual transducers 64 may be selected such that the collective bandwidth of the entire array 170 spans some or all of a desired frequency range (e.g., from 750 kHz to 1.25 MHz, from 700 kHz to 1 MHz, from 800 kHz to 1.2 MHz, from 900 kHz to 1.1 MHz, from 750 kHz to 1.4 MHz, or any other suitable frequency range). The desired frequency range may depend on the range of distances to be measured. For example, to measure the distance to an object (such as the user's eye 58) within a few centimeters, the array 170 may span a frequency range from 750 kHz to 1.25 MHz (by way of example).

[0062] The substrate 68 may have any suitable dimensions. For example, the lateral dimensions L1 and L2 of the substrate 68 may be between 2 mm and 2.5 mm, between 1 mm and 1.5 mm, between 1 mm and 3 mm, between 2 mm and 4 mm, and / or other suitable lengths. The dimensions L1 and L2 may be equal (such that the substrate 68 has a square coverage area) or unequal (such that the substrate 68 has a rectangular coverage area), or the coverage area of the substrate 68 may have other shapes (e.g., circular, oval, rounded, triangular, etc.).

[0063] The transducers 64 may be arranged on the substrate 68 in a uniformly spaced grid of rows and columns, or may be arranged in any other suitable pattern (e.g., non-uniformly spaced clusters, random patterns, non-grid patterns, etc.). Where the transducers 64 have a circular shape Figure 6The examples are illustrative only. If desired, the transducer 64 can be square, rectangular, oval, rounded, or any other suitable shape.

[0064] In some arrangements, it may be desirable to maximize the amount of space between transducers 64 that share the same center frequency. Maximizing the spacing between co-configured transducers 64 in the array 170 can increase the accuracy of distance measurements made with the array 170. Different rules regarding the placement of different subsets of transducers 64 on the substrate 68 can be implemented to achieve the desired performance from the array 170. As an example, the convex hull of a given set of transducers 64 that share the same center frequency can cover at least 50% of the array 170, can cover at least 80% of the array 170, or can cover other suitable portions of the array 170. As another example, most transducer pairs 64 that share the same center frequency can be separated by transducers 64 of different center frequencies. These examples are merely illustrative. In general, the transducers 64 can be placed on the substrate 68 in any suitable arrangement.

[0065] The operation of the transducers 64 can be controlled by the control circuit 20. The substrate 68 can include interconnects 66 for transmitting signals between the transducers 64 and the control circuit 20. For example, the interconnects 66 can be used to transmit drive signals from the control circuit 20 to the transducers 64 and to transmit sensor signals (e.g., sensor signals associated with ultrasonic waves detected by the transducers 64) from the transducers 64 to the control circuit 20.

[0066] If desired, the transducers 64 in the array 170 can be controlled independently of each other. For example, the frequency, phase, and pulse shape of the drive signal for a given transducer 64 can be different from those of other transducers 64 in the array 170. Each individual transducer 64 in the array 170 can be independently controlled with a different drive signal, or there can be subsets of transducers 64 that are controlled with the same drive signal but independently of other subsets of transducers 64 (e.g., subsets that share the same center frequency or other suitable subsets). However, this is merely illustrative. If desired, the transducers 64 can be not be independently controlled and / or can be controlled with any other suitable drive scheme.

[0067] In some arrangements, transducer 64 may be driven by off-chip control circuitry. In this type of arrangement, interconnect 66 may include leads, contact pads, solder, and / or other conductive elements for conveying signals between array 170 and control circuitry 20 (which is separate from array 170). In other arrangements, substrate 68 may be a multi-layer substrate where transducer 64 is stacked with a control circuitry layer (e.g., an application specific integrated circuit layer) that includes control circuitry 20. With this type of integrated control circuitry, interconnect 66 may include metal vias and / or other conductive elements for conveying signals between transducer 64 and control circuitry 20 located in different layers of substrate 68. These examples are merely illustrative. If desired, interconnect 66 may include metal vias for conveying signals between different layers of substrate 68 and may also include contact pads for conveying signals between array 170 and an external circuit.

[0068] The center frequency of a piezoelectric transducer is determined at least in part by the dimensions of the cavity of the transducer. In some arrangements, transducer 64 may be provided with different center frequencies by using cavities having different dimensions (e.g., different depths, different diameters, etc.) and / or by using different surface features within the cavity to produce a desired acoustic reflection phase at the center frequency. If desired, other structures may be used to produce transducer arrays having different center frequencies. For example, instead of varying the cavity depth, transducer 64 may have a uniform cavity depth (e.g., they may all have a relatively short cavity depth), but have different lateral cavity dimensions (e.g., different diameters, different lengths and widths, etc.). Arrangements where both the cavity depth and the lateral dimensions of the cavity vary may also be used. Generally, any suitable technique for producing transducers having different center frequencies may be used. In addition to or instead of adjusting the cavity dimensions to achieve a desired center frequency, the mechanical resonance of the membrane of transducer 64 may be adjusted to tune the resonance frequency of transducer 64. In particular, one or more openings may be formed in the membrane to adjust the spring constant of the membrane and thereby its resonance frequency.

[0069] Figure 7An exemplary optical self-mixing sensor is shown. The self-mixing sensor 160, sometimes referred to as an optical self-mixing position sensor or a self-mixing orientation sensor, can be used to measure distance and thus determine the relative position between the sensor and the target structure. In some configurations, one or more self-mixing sensors can be used to measure angular orientation. For example, angular tilt can be measured by measuring two or more distances. For example, a pair of distance measurements made at different corresponding positions on a component can be used to measure tilt about one axis, while three such distance measurements can be used to measure tilt about two axes. Arrangements in which the self-mixing sensor is said to measure distance, displacement, or position may sometimes be described herein by way of example. Generally, changes in position, angular orientation, position and / or orientation, and / or other self-mixing sensor measurements can be directly acquired and / or can be derived from measurements of the distance from the self-mixing sensor.

[0070] In some arrangements, the rangefinder 48 can include a self-mixing sensor, such as a self-mixing optical interferometer. Figure 7 is a side view of an exemplary self-mixing interferometer, such as self-mixing interferometer 160. As Figure 7 shown, the self-mixing sensor 160 can be used to measure the spacing (distance D) between the sensor 160 and the eye 58.

[0071] The self-mixing interferometer 160 can include a laser, such as a vertical cavity surface emitting laser 150 (e.g., the self-mixing proximity sensor 160 can be a coherent self-mixing sensor having a diode laser or other coherent or partially coherent source of light or other electromagnetic radiation). The laser 150 can have thin film interference filters 152 (sometimes referred to as Bragg reflectors), each formed by a stack of thin film layers of alternating refractive indices. The active region 154 can be formed between the mirrors 152. The lower mirror 152 in the laser 150 can have a nominal reflectivity of less than 100% to allow some light from the laser 150 to reach the overlapping photodiode 154, or in configurations where the photodiode 154 is located elsewhere in the sensor 160 (e.g., laterally adjacent to the laser 150), the lower mirror 152 can have a nominal reflectivity of 100%. The upper mirror 152 in the laser 150 can have a slightly lower reflectivity such that the laser 150 emits light 158 toward the eye 58. The laser 150 can be controlled by applying a drive signal to the terminal 156 using a control circuit 20 (e.g., a drive circuit in the circuit 20). A sensing circuit (e.g., the photodiode 154 and / or an associated sensing circuit in the circuit 20) can measure the light output of the laser 150 (by way of example).

[0072] The infrared wavelength of the emitted light 158 can be 850 nm - 1200 nm, 800 nm to 1100 nm, 920 nm - 960 nm, at least 800 nm, at least 900 nm, at least 1000 nm, less than 1200 nm, less than 1100 nm, less than 1000 nm, or less than 900 nm, or other suitable wavelengths (e.g., visible wavelength, ultraviolet wavelength, infrared wavelength, near-infrared wavelength, etc.). When the emitted light 158 illuminates the eye 58, some of the emitted light in the emitted light 158 will be reflected back towards the sensor 150 as reflected light 160 (e.g., light specularly reflected from the eye 58 and / or light backscattered from a matte surface in the eye 58).

[0073] Figure 7 The sensor 160 includes photosensitive elements (e.g., a light detector such as a photodiode 154). Figure 7 In the example, the photodiode 154 is located below the laser 150, but if desired, configurations can be used where the photodiode 154 is adjacent to the laser 150, on a separate substrate outside the laser 150, above the active region 154 of the laser 150, and / or has other configurations. The terminals of the photodiode 154 can be coupled to a sensing circuit in the control circuit 20. This circuit acquires the photodiode output signal, which is generated in response to receiving reflected light (the specularly reflected portion and / or the backscattered portion of the emitted light 158) such as the reflected light 160. In addition to using a photodiode, laser junction voltage measurement (e.g., if the laser is driven with a constant bias current) or laser bias current (e.g., if the laser is driven with a constant voltage) can also be used to detect self-mixing.

[0074] A portion of the light 160 that is reflected or backscattered from the eye 58 as the reflected light 160 re-enters the laser cavity of the laser 150 and mixes with the light in the laser cavity, thereby coherently perturbing the electric field and causing a perturbation in the carrier density in the laser 150. These perturbations in the laser 150 result in coherent self-mixing fluctuations of the power of the emitted light 158 and the associated operating characteristics of the laser 150 (such as the laser junction voltage and / or the laser bias current). These fluctuations can be monitored. For example, the power fluctuations of the light 158 can be monitored using the photodiode 154. In Figure 7 the example, the photodiode 154 is an integrated monolithic photodiode formed below the laser 150, but other configurations can be used if desired.

[0075] The control circuit 20 is configured to supply a drive current to the laser 150 and includes circuitry for sensing the response of the photodiode 154. The sensed photodiode output may include measurements of diode current and / or voltage. A modulation scheme may be used to drive the laser 150 for inducing wavelength modulation, and a photodiode output processing scheme (using measurements of photodiode current, junction voltage, bias current, etc.) may be used to process the self-mixing fluctuations of the measured output power to allow the control circuit 20 to determine the distance D between the sensor 160 and the eye 58 based on the principle of self-mixing interferometry.

[0076] Figure 7 and Figure 8 The examples of Figure 8 are merely illustrative examples of the rangefinder 48 that may be included in the gaze tracking circuit 62 of the device 10. If desired, the rangefinder 48 may include additional or different types of distance sensors.

[0077] Figure 8 is a diagram showing an exemplary quantitative model for determining the gaze direction using the camera 42, the light-emitting diode 44, and the rangefinder 48. Although Figure 8 illustrates a two-dimensional model, this two-dimensional model may be generalized to three dimensions. In the gaze tracking circuit 62, the camera 42, the light source 44, and the corneal center of the eye 58 ( Figure 4 ) define three points in space and thus define a plane. The flash generated by the light source 44 on the eye 58 may also be included in the same plane (due to symmetry). In this model, it may be assumed that the cornea is locally spherical around the region where the flash is generated, and the distance is measured from this region. Additionally, it may be assumed that the camera 42 can find the azimuth of the corneal vertex (e.g., the point directly above the pupil) and the azimuth of the flash.

[0078] Figure 8 's model shows how to use geometry to predict the ground truth. By adding errors in the range (distance) data and / or the corneal radius and performing gaze reconstruction using this imperfect data, the sensitivity of the model to errors can be determined. In the forward model, the ground truth optical axis (sometimes called the ground truth gaze direction) may be used to determine the pupil position 88 and the corneal center 92. From the corneal center 92, the corneal surface 86 can be determined. The position on the cornea 86 that minimizes the round-trip travel from the camera 42 at position 80 to the light source 44 at position 82 yields the ground truth time-of-flight path measurement (see Figure 8 's path 84). This position 90 on the cornea 86 is where the flash will appear. The ground truth optical axis is indicated by the line 180 connecting the corneal center 92 and the pupil position 88.

[0079] To determine the sensitivity of the model to errors, distance measurement errors may be assumed. For example, it may be assumed that the camera 42 has negligible error in the flash azimuth, the rangefinder 48 detects an inaccurate eye distance, or there is noise in the flash image captured by the camera 42, which adds an additional length to the light ray between the camera 42 and the true flash. This may result in a modeled flash at position 96 instead of position 90. Based on the modeled flash at position 96, the corneal surface 102 and the estimated corneal center 98 (e.g., based on a known corneal radius of curvature) may be determined. The azimuth 104 of the pupil from the camera 42 may be projected until it intersects the modeled cornea 102 at the modeled pupil position 100. The line 182 connecting the modeled corneal center 98 and the modeled pupil position 100 corresponds to the modeled user's gaze direction. Assuming a time-of-flight error of about 0.5 mm, the model is still able to predict the user's gaze direction accurately enough. Generally speaking, the gaze direction error may be linearly proportional to the distance measurement error.

[0080] Figure 9 is a flowchart of exemplary steps involved in determining a user's gaze direction using the gaze tracking circuit 62.

[0081] During the operation of block 200, the light source 44 may be used to generate a flash on the user's eye 58 ( Figure 4 ). The camera 42 may capture an image of the eye including the flash. Based on the captured image (and based on other known data such as the corneal radius of curvature), the control circuit 20 may determine the position of the user's pupil and the position of the flash.

[0082] During the operation of block 202, the rangefinder 48 may be used to determine the distance to the eye. This may include using a transmitter (e.g., Figure 4 's transmitter 48T) to emit a light signal, an ultrasonic signal, and / or other suitable signals towards the eye and using a receiver (e.g., Figure 4 's receiver 48R) to detect the reflected signal. Using time-of-flight measurement techniques or other suitable measurement techniques, the distance to the eye may be determined.

[0083] During the operation of block 204, the control circuit 20 may determine the user's gaze direction based on the measured pupil position, the measured flash position, and the measured distance to the eye.

[0084] If the distance to the eye also remains unchanged, it can be assumed that the gaze direction remains unchanged. If necessary, after determining the gaze direction, the camera 42 and the light source 44 can be temporarily placed in a low-power state (e.g., off state, sleep state, or other low-power state) to save power, while the rangefinder 48 can be used to monitor changes in the distance to the eye. During the operation of block 206, for example, the camera 42 and the light source 44 can be placed in a low-power state, while the rangefinder 48 continuously or periodically measures the distance to the eye. If no change in distance is detected, the process can loop back to step 206 (see Figure 9 line 208 of Figure 9 ), and the control circuit 20 can continue to use the rangefinder 48 to monitor changes in the distance to the eye. If a change in distance is detected, the camera 42 and the light source 44 can be turned on (e.g., switched from a low-power state to a powered-on state), and the process can loop back to step 200 (see line 210 of

[0085]

[0086] ). If necessary, this allows the camera 42 and the light source 44 to be turned on only when a new gaze direction needs to be measured.

[0087] However, this is only illustrative. If necessary, the camera 42 and the light source 44 can remain powered on and can be used in combination with the rangefinder 48 to determine the gaze direction.

[0088] The present disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with sound privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to privacy policies and measures that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable entity purposes and should not be shared or sold outside of those legitimate purposes. Additionally, such collection / sharing should occur after receiving informed consent from the user. Further, such entities should consider taking any necessary steps to protect and secure access to such personal information data and to ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. Additionally, such an entity may subject itself to third-party assessments to demonstrate its compliance with widely accepted privacy policies and practices. Further, the policies and practices should be adapted to the specific type of personal information data being collected and / or accessed and to the applicable laws and standards, including considerations of special jurisdictions. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Thus, different privacy measures should be asserted for different types of personal data in each country.

[0089] Notwithstanding the foregoing, the present disclosure also anticipates embodiments where users selectively block the use or access of personal information data. That is, the present disclosure anticipates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, the inventive technology may be configured to allow a user to select to “opt-in” or “opt-out” of the collection of personal information data at any time during or after registering for a service. In another example, a user may choose not to provide certain types of user data. In yet another example, a user may choose to limit the length of time that user-specific data is maintained. In addition to providing “opt-in” and “opt-out” options, the present disclosure also anticipates providing notices related to the access or use of personal information. For example, a user may be notified when downloading an application (“app”) that their personal information data will be accessed and then reminded again before the personal information data is accessed by the app.

[0090] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by restricting data collection and deleting the data. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. In appropriate cases, de-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or characteristics of the data stored (e.g., collecting location data at the city level rather than the address level), controlling the way the data is stored (e.g., aggregating data among users), and / or other methods.

[0091] Thus, while the present disclosure broadly covers the use of information that may include personal information data to implement one or more of the various disclosed embodiments, the present disclosure also anticipates that the various embodiments may also be implemented without access to personal information data. That is, the various embodiments of the technology of the present invention will not be unable to function properly due to the lack of all or a portion of such personal information data.

[0092] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the help of an electronic system. Physical environments such as physical parks include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0093] Computer-Generated Reality: In contrast, a computer-generated reality (CGR) environment is a fully or partially simulated environment in which people sense and / or interact via an electronic system. In CGR, a subset of a person's physical movements or their representations are tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner consistent with at least one physical law. For example, a CGR system can detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of the characteristics of virtual objects in a CGR environment can be made in response to a representation of a physical movement (e.g., a voice command). A person can use any of their senses to sense and / or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person can sense and / or interact only with audio objects. Examples of CGR include virtual reality and mixed reality.

[0094] Virtual Reality: A virtual reality (VR) environment is a simulated environment that is designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects with which a person can sense and / or interact. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with the virtual objects in a VR environment by way of a simulation of the person's presence within the computer-generated environment and / or by way of a simulation of a subset of the person's physical movements within the computer-generated environment.

[0095] Mixed Reality: Compared to a VR environment that is designed to be completely based on computer-generated sensory input, a mixed reality (MR) environment is a simulated environment that is designed to include, in addition to computer-generated sensory input (e.g., virtual objects), sensory input or its representation from the physical environment. On the virtual continuum, an MR environment is any condition between a fully physical environment at one end and a virtual reality environment at the other end, but excluding these two ends. In some MR environments, the computer-generated sensory input can respond to changes in the sensory input from the physical environment. Additionally, some electronic systems for presenting an MR environment can track the position and / or orientation relative to the physical environment so that virtual objects can interact with real objects (i.e., physical items from the physical environment or their representations). For example, the system can cause movement such that a virtual tree appears stationary relative to the physical ground. Examples of mixed reality include augmented reality and augmented virtuality. Augmented Reality: An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are superimposed on the physical environment or a representation of the physical environment. For example, an electronic system for presenting an AR environment can have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or translucent display such that a person using the system perceives the virtual objects superimposed on the physical environment. Alternatively, the system can have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with the virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects superimposed on the physical environment. As used herein, a video of the physical environment displayed on an opaque display is referred to as "passthrough video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Further alternatively, the system can have a projection system that projects virtual objects into the physical environment, such as as a hologram or on a physical surface, such that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment is also a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing passthrough video, the system can transform one or more sensor images to impose an alternative perspective (e.g., viewpoint) that is different from the perspective captured by the imaging sensor. Another example is that a representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it such that the modified portions can be a representative but not a true version of the originally captured image. Yet another example is that a representation of the physical environment can be transformed by graphically removing portions of it or blurring portions of it.Augmented Virtuality: An augmented virtual (AV) environment is a simulated environment in which a virtual environment or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but a person's face is a realistic reproduction from an image of the physical person. As another example, a virtual object can adopt the shape or color of a physical item imaged by one or more imaging sensors. As yet another example, a virtual object can adopt a shadow that conforms to the position of the sun in the physical environment.

[0096] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed to be placed on a person's eye (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smart phones, tablet computers, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, the head-mounted system can be configured to receive an external opaque display (e.g., a smart phone). The head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. The head-mounted system can have a transparent or translucent display instead of an opaque display. The transparent or translucent display can have a medium through which light representing an image is directed to a person's eye. The display can utilize digital light projection, OLED, LED, μLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphical image onto a person's retina. The projection system can also be configured to project virtual objects into the physical environment, such as as a hologram or on a physical surface.

[0097] According to one embodiment, there is provided an eye wear that includes a support structure, a first lens and a second lens mounted to the support structure, and a gaze tracking circuit configured to track a gaze direction, the gaze tracking circuit including a light emitting diode configured to generate an eye flash, a camera configured to capture an image of the eye flash, and a rangefinder configured to measure an interpupillary distance, the gaze direction being determined based on the captured image and the measured interpupillary distance.

[0098] According to another embodiment, the rangefinder includes an ultrasonic rangefinder.

[0099] According to another embodiment, the ultrasonic rangefinder includes an ultrasonic transducer array on a common substrate.

[0100] According to another embodiment, the ultrasonic transducer array includes at least some ultrasonic transducers having different center frequencies.

[0101] According to another embodiment, the rangefinder includes an optical rangefinder.

[0102] According to another embodiment, the optical rangefinder includes a self-mixing interferometer.

[0103] According to another embodiment, the first lens and the second lens are adjustable prescription lenses.

[0104] According to another embodiment, the eye wear optionally includes a first display and a second display configured to present images that can be viewed from a first eye comfort zone and a second eye comfort zone through the first lens and the second lens, respectively.

[0105] According to another embodiment, the eye flash is optionally the only eye flash in the captured image.

[0106] According to another embodiment, the pupil position and the flash position are optionally determined based on the captured image, and the gaze direction is determined based on the pupil position, the flash position, and the interpupillary distance.

[0107] According to one embodiment, there is provided a gaze tracking circuit, the gaze tracking circuit including: a light source configured to generate an eye flash; a camera configured to capture an image of the eye flash, the pupil position and the flash position being determined based on the captured image; and a rangefinder configured to measure the interpupillary distance, the gaze direction being determined based on the pupil position, the flash position, and the interpupillary distance.

[0108] According to another embodiment, the light source optionally includes a light emitting diode, and the eye flash is the only eye flash in the captured image.

[0109] According to another embodiment, the rangefinder optionally includes an ultrasonic rangefinder having an ultrasonic transducer array on a common substrate.

[0110] According to another embodiment, the rangefinder optionally includes an optical rangefinder.

[0111] According to another embodiment, the optical rangefinder optionally includes a self-mixing interferometer.

[0112] According to one embodiment, there is provided a head-mounted device including a display configured to present an image, a lens through which the image can be viewed from an eye zone, and a gaze tracking circuit configured to track a gaze direction. The gaze tracking circuit includes a light source and a camera for determining a pupil position and a flash position, and a rangefinder configured to measure an eye distance. The gaze direction is determined based on the pupil position, the flash position, and the eye distance.

[0113] According to another embodiment, the light source includes a light emitting diode that generates an eye flash, the camera captures an image of the eye flash, and the pupil position and the flash position are determined based on the captured image.

[0114] According to another embodiment, the rangefinder optionally includes an ultrasonic rangefinder.

[0115] According to another embodiment, the rangefinder optionally includes an optical rangefinder based on optical coherence tomography.

[0116] According to another embodiment, the rangefinder is optionally configured to monitor a change in the eye distance when the light source and the camera are in a low power state, and the light source and the camera switch from the low power state to an energized state in response to the measured change in the eye distance.

[0117] The foregoing is merely illustrative and various modifications may be made to the described embodiments. The foregoing embodiments may be implemented individually or in any combination.

Claims

1. An eye-wearable device, the eye-wearable device comprising: A support structure; A first lens and a second lens, the first lens and the second lens being mounted to the support structure; And A gaze tracking circuit configured to track a gaze direction, wherein the gaze tracking circuit comprises: A light-emitting diode configured to generate an eye flash; A camera configured to capture an image of the eye flash; and A rangefinder configured to measure an interpupillary distance, wherein the gaze direction is determined based on the captured image and the measured interpupillary distance.

2. The eye-wearable device according to claim 1, wherein the rangefinder comprises an ultrasonic rangefinder.

3. The eye-wearable device according to claim 2, wherein the ultrasonic rangefinder comprises an array of ultrasonic transducers on a common substrate.

4. The eye-wearable device according to claim 3, wherein the array of ultrasonic transducers comprises at least some ultrasonic transducers having different center frequencies.

5. The eye-wearable device according to claim 1, wherein the rangefinder comprises an optical rangefinder.

6. The eye-wearable device according to claim 5, wherein the optical rangefinder comprises a self-mixing interferometer.

7. The eye-wearable device according to claim 1, wherein the first lens and the second lens are adjustable prescription lenses.

8. The eye-wearable device according to claim 1, the eye-wearable device further comprising: A first display and a second display configured to present images, wherein the images can be viewed from a first eye comfort zone and a second eye comfort zone through the first lens and the second lens, respectively.

9. The eye-wearable device according to claim 1, wherein the eye flash is the only eye flash in the captured image.

10. The eye-wearable device according to claim 1, wherein a pupil position and a flash position are determined based on the captured image, and wherein the gaze direction is determined based on the pupil position, the flash position, and the interpupillary distance.

11. A gaze tracking circuit, the gaze tracking circuit comprising: A light source configured to generate an eye flash; A camera configured to capture an image of the eye flash, wherein a pupil position and a flash position are determined based on the captured image; And A rangefinder configured to measure an interpupillary distance, wherein a gaze direction is determined based on the pupil position, the flash position, and the interpupillary distance.

12. The gaze tracking circuit according to claim 11, wherein the light source comprises a light-emitting diode, and wherein the eye flash is the only eye flash in the captured image.

13. The gaze tracking circuit according to claim 11, wherein the rangefinder comprises an ultrasonic rangefinder having an array of ultrasonic transducers on a common substrate.

14. The gaze tracking circuit according to claim 11, wherein the rangefinder comprises an optical rangefinder.

15. The gaze tracking circuit according to claim 14, wherein the optical rangefinder comprises a self-mixing interferometer.

16. A head-mounted device, the head-mounted device comprising: A display configured to present an image; A lens through which the image can be viewed from an eye comfort zone; And A gaze tracking circuit configured to track a gaze direction, wherein the gaze tracking circuit comprises: A light source and a camera for determining a pupil position and a flash position; And A rangefinder configured to measure an eye distance, wherein the gaze direction is determined based on the pupil position, the flash position, and the eye distance.

17. The head-mounted device according to claim 16, wherein the light source comprises a light-emitting diode that generates an eye flash, wherein the camera captures an image of the eye flash, and wherein the pupil position and the flash position are determined based on the captured image.

18. The head-mounted device according to claim 16, wherein the rangefinder comprises an ultrasonic rangefinder.

19. The head-mounted device according to claim 16, wherein the rangefinder comprises an optical rangefinder based on optical coherence tomography.

20. The head-mounted device according to claim 16, wherein the rangefinder is configured to monitor a change in the eye distance when the light source and the camera are in a low-power state, and wherein the light source and the camera switch from the low-power state to an energized state in response to the measured change in the eye distance.

Images