Systems and methods for calibrating an image sensor in a wearable device

Abstract

In one embodiment, a wearable device is provided. The wearable device may include a holographic film disposed on the wearable device, an eye tracking device including an image sensor and an illuminator, and one or more processors. The processor may be configured to: activate the illuminator to illuminate the holographic film; and when the holographic film is illuminated, use the image sensor to capture an image of at least a portion of the holographic film. The processor may also be configured to: determine characteristics of the holographic film based on the image; and determine the position or orientation of the image sensor relative to the holographic film based on the characteristics. The processor may further be configured to: change at least one calibration parameter of the image sensor based on the position or the orientation of the image sensor relative to the holographic film.

Description

[0001] Cross - reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 631,196, filed on February 15, 2018, the entire disclosure of which is incorporated herein by reference for all purposes as fully set forth herein. Technical field

[0003] This application relates to gaze detection systems and methods. Background art

[0004] Interaction with computing devices is a fundamental behavior in today's world. Computing devices such as personal computers are ubiquitous in daily life. Systems and methods for interacting with such devices define how to use them and what they are used for.

[0005] Advances in eye - tracking technology have made it possible to interact with computing devices using an individual's gaze information. In other words, the position on the display that the user is gazing at. This information can be used for interaction either alone or in combination with contact - based interaction techniques (e.g., using user input devices such as keyboards, mice, or touchscreens).

[0006] Gaze - based interaction techniques are finding their way into wearable computing devices such as virtual reality (VR) headsets and augmented reality (AR) headsets. Advantageously, these techniques allow users to control certain aspects of the wearable computing device (e.g., user interface, settings, presented content, etc.). Summary of the invention

[0007] In one embodiment, there is provided an augmented reality device, a virtual reality device, or other wearable device. The device may include: a holographic film disposed on the wearable device, an eye - tracking device including an image sensor and an illuminator, and one or more processors. The processor may be configured to: activate the illuminator to illuminate the holographic film; and when the holographic film is illuminated, use the image sensor to capture an image of at least a portion of the holographic film. The processor may also be configured to: determine characteristics of the holographic film based on the image; and determine at least one of a position or an orientation of the image sensor relative to the holographic film based on the characteristics. The processor may further be configured to: change at least one calibration parameter of the image sensor based on at least one of the position or the orientation of the image sensor relative to the holographic film.

[0008] In another embodiment, a method for changing at least one calibration parameter of an image sensor in an eye tracking device of an augmented reality device, a virtual reality device, or other wearable device is provided. The method may include: activating an illuminator of the eye tracking device to illuminate a holographic film disposed on the wearable device. The method may further include: when the holographic film is illuminated, using the image sensor of the eye tracking device to capture an image of at least a portion of the holographic film. The method may further include: determining a characteristic of the holographic film based on the image. Additionally, the method may include: determining at least one of a position or an orientation of the image sensor relative to the holographic film based on the characteristic. Furthermore, the method may include: changing at least one calibration parameter of the image sensor based on the position or the orientation of the image sensor relative to the holographic film.

[0009] In another embodiment, a non-transitory machine-readable medium storing instructions for changing at least one calibration parameter of an image sensor in an eye tracking device of an augmented reality device, a virtual reality device, or other wearable device is provided. The instructions may be executed by one or more processors to perform a method. The method may include: activating an illuminator of the eye tracking device to illuminate a holographic film disposed on the wearable device. The method may further include: when the holographic film is illuminated, using the image sensor of the eye tracking device to capture an image of at least a portion of the holographic film. The method may further include: determining a characteristic of the holographic film based on the image. Additionally, the method may include: determining at least one of a position or an orientation of the image sensor relative to the holographic film based on the characteristic. Furthermore, the method may include: changing at least one calibration parameter of the image sensor based on the position or the orientation of the image sensor relative to the holographic film.

[0010] In one embodiment, an augmented reality device, a virtual reality device, or other wearable device is provided, including: a holographic film disposed on the wearable device; an eye tracking device including an image sensor and an illuminator; and one or more processors configured to at least: activate the illuminator to illuminate the holographic film; when the holographic film is illuminated, use the image sensor to capture an image of at least a portion of the holographic film; determine a characteristic of the holographic film based on the image; determine at least one of a position or an orientation of the image sensor relative to the holographic film based on the characteristic; and change at least one calibration parameter of the image sensor based on at least one of the position or the orientation of the image sensor relative to the holographic film.

[0011] The image includes: light reflection leaving the holographic film.

[0012] The reflection includes: diffuse reflection from the features of the holographic film.

[0013] The reflection includes: specular reflection of the features of the holographic film.

[0014] The image includes: light diffuse reflection around an area without light reflection.

[0015] The image includes: light diffuse reflection defined by the edge of the holographic film.

[0016] Determining the position or orientation of the image sensor relative to the holographic film includes: determining the position or orientation of the image sensor relative to three orthogonal axes.

[0017] The holographic film includes: three characteristics in two axes arranged in the holographic film, and the three characteristics specularly reflect light towards the image sensor.

[0018] The holographic film includes: only one characteristic that reflects light, and the characteristic is two-dimensional.

[0019] The holographic film includes: at least one characteristic that does not reflect light, at least a part of the at least one characteristic is defined by the reflective part of the holographic film, and the at least one characteristic is two-dimensional.

[0020] In one embodiment, a method for changing at least one calibration parameter of an image sensor in an eye tracking device of an augmented reality device, a virtual reality device, or other wearable devices is provided. The method includes: activating an illuminator of the eye tracking device to illuminate a holographic film disposed on the wearable device; when the holographic film is illuminated, using the image sensor of the eye tracking device to capture an image of at least a part of the holographic film; determining the characteristics of the holographic film based on the image; determining at least one of the position or orientation of the image sensor relative to the holographic film based on the characteristics; and changing at least one calibration parameter of the image sensor based on the position or orientation of the image sensor relative to the holographic film.

[0021] The image includes: light reflection leaving the holographic film, and the reflection includes diffuse reflection from the features of the holographic film or specular reflection of the features of the holographic film.

[0022] The image includes: light diffuse reflection around an area without light reflection.

[0023] The image includes: light diffuse reflection defined by the edge of the holographic film.

[0024] Determining the position or orientation of the image sensor relative to the holographic film includes: determining the position or orientation of the image sensor relative to three orthogonal axes.

[0025] In one embodiment, a non-transitory machine-readable medium is provided, having instructions stored thereon for changing at least one calibration parameter of an image sensor in an eye tracking device of an augmented reality device, a virtual reality device, or other wearable device, wherein the instructions are executable by one or more processors to perform at least the following operations: activating an illuminator of the eye tracking device to illuminate a holographic film disposed on the wearable device; when the holographic film is illuminated, capturing an image of at least a portion of the holographic film using the image sensor of the eye tracking device; determining characteristics of the holographic film based on the image; determining at least one of the position or orientation of the image sensor relative to the holographic film based on the characteristics; and changing at least one calibration parameter of the image sensor based on the position or orientation of the image sensor relative to the holographic film.

[0026] The image includes: light reflection leaving the holographic film, and wherein the reflection includes diffuse reflection from features of the holographic film or specular reflection of features of the holographic film.

[0027] The image includes: light diffuse reflection around an area without light reflection.

[0028] The image includes: light diffuse reflection bounded by the edge of the holographic film.

[0029] Determining the position or orientation of the image sensor relative to the holographic film includes:

[0030] Determining the position or orientation of the image sensor relative to three orthogonal axes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Embodiments of the present invention are described in conjunction with the accompanying drawings:

[0032] Figure 1 An exemplary eye tracking system is shown;

[0033] Figure 2 An example image of an eye captured by an image sensor is shown;

[0034] Figure 3 A dedicated computer system is shown;

[0035] Figure 4 An exemplary wearable computing device is shown;

[0036] Figure 5 Example components of a virtual reality head-mounted device are shown;

[0037] Figure 6 Shows an example of a mark on a lens;

[0038] Figure 7 Shows an example of using a mark on a lens;

[0039] Figure 8 Shows a method for updating calibration parameters;

[0040] Figure 9 Shows an example of the distance from the center of the cornea to the lens;

[0041] Figure 10 Shows an example of pincushion distortion of an image;

[0042] Figure 11 Shows another method for updating calibration parameters;

[0043] Figure 12 Shows an example of an optimized Fresnel lens;

[0044] Figure 13 Shows an example of a two - layer Fresnel lens;

[0045] Figure 14 Shows an example of manipulating an object in a virtual reality environment;

[0046] Figure 15 Shows a method for manipulating an object in a virtual environment;

[0047] Figure 16 Shows an example of manipulating multiple objects in a virtual reality environment;

[0048] Figure 17 Shows a method for manipulating multiple objects in a virtual reality environment;

[0049] Figure 18 Shows another example of manipulating multiple objects in a virtual reality environment;

[0050] Figure 19 Shows another method for manipulating multiple objects in a virtual reality environment;

[0051] Figure 20 Shows an example of replacing a controller in a virtual environment;

[0052] Figure 21 Shows a method for replacing a controller in a virtual environment;

[0053] Figure 22 Shows a schematic example of a wearable device, which demonstrates an embodiment of the present invention; and

[0054] Figures 23A to 23F An example holographic film with reflective markings thereon is shown.

[0055] In the drawings, like parts and / or features may have the same numerical reference marks. Further, each part of the same type may be distinguished by a letter following the reference mark to distinguish like parts and / or features. If only the first numerical reference mark is used in the specification, the description applies to any one of the like parts and / or features having the same first numerical reference mark without considering the letter suffix. DETAILED DESCRIPTION

[0056] This application relates to gaze detection systems and methods. In an example, such systems and methods are embodied in virtual reality devices, augmented reality devices, and / or other computing devices, and support using gaze information for user interaction with such computing devices.

[0057] Interaction with computing devices is a fundamental behavior in today's world. Computing devices such as personal computers are ubiquitous in daily life. The systems and methods for interacting with such devices define how to use them and what they are used for.

[0058] Advances in eye tracking technology have made it possible to interact with computing devices using an individual's gaze information. In other words, the position on the display that the user is gazing at. This information can be used for interaction either alone or in combination with contact-based interaction techniques (e.g., using user input devices such as keyboards, mice, or touchscreens).

[0059] Previously proposed interaction techniques using gaze information can be found in U.S. Patent 6,204,828, U.S. Patent Application Publication 20130169560, U.S. Patent 7,113,170, U.S. Patent Application Publication 20140247232, and U.S. Patent 9,619,020. The entire specifications of these patents and applications are incorporated herein by reference.

[0060] Gaze-based interaction techniques are looking for ways to be applied to wearable computing devices such as virtual reality (VR) headsets and augmented reality (AR) headsets. Advantageously, these techniques allow users to control certain aspects of the wearable computing device (e.g., user interface, settings, presented content, etc.).

[0061] Nevertheless, given the portability of these types of computing devices, a number of technical challenges have emerged. For example, the accuracy of gaze information may be reduced for different reasons. Wearable computing devices undergo repeated movements, which can change the alignment of the optical processing hardware components. Over time, the misalignment reduces the accuracy of the information derived from processing the measured optical data. Another reason is related to the size of some of these optical processing hardware components (e.g., the lenses used). The size is typically limited due to the size of the wearable computing device. In turn, the limitation may affect how well the hardware components process light (e.g., transmit, reflect, distort, scatter).

[0062] In addition, many applications available on wearable computing devices involve the use of an external controller. Using an external controller to interact accurately and intuitively with virtual objects affects the usability, success, and failure of such applications. Integrating the virtual world with the external controller and gaze information is challenging.

[0063] Embodiments of the present disclosure relate to improving the accuracy of detecting gaze information and the usability of applications that rely on an external controller. In an example context of a wearable computing device using gaze-based interaction techniques, accuracy can be improved based on component calibration over time and / or based on a specific component construction that takes into account overall size constraints. Component calibration generally involves updating gaze-related parameters. In a first example, updating gaze-related parameters relies on processing an image to determine the position and rotation of a camera and a lens based on markers located on the lens. In a second example, updating gaze-related parameters relies on processing an image to determine the distance between the center of the cornea and the center of the lens and using this distance to compensate for distortion (e.g., pincushion distortion). In a third example, updating gaze-related parameters relies on processing an image to determine multiple flash positions and updating stereo parameters (e.g., the center and radius of the cornea, and the distance to the camera). Regarding the specific component construction, a Fresnel lens is used in the example. The central region of this lens does not have any Fresnel steps. Instead, the Fresnel steps are arranged in an annular manner around the central region and have different groove depths and step sizes. These and other examples are described further below. For clarity of explanation, the components of the gaze detection system are first described, followed by a description of how such components are implemented in a wearable portable device. Thereafter, details of example component calibration and an example Fresnel lens construction are described. In an example context of a wearable computing device using gaze-based interaction techniques and an external controller, the gaze of a user at a virtual object in a virtual environment is tracked and a virtual representation of the external controller is presented in the virtual environment. When it is detected that the user has gazed at a particular object for a period of time, the wearable computing device selects and locks the object. Taking into account the user touch area of the external controller, the wearable computing device presents a virtual menu to manipulate the locked virtual object. The options on the menu depend on the user touch area. In this way, the user can accurately and intuitively manipulate the locked virtual object by moving his or her finger around the touch area of the external controller.

[0064] Eye tracking

[0065] Figure 1 An eye tracking system 100 (which may also be referred to as a gaze tracking system) according to an embodiment is shown. The system 100 includes illuminators (or light emitters) 111 and 112 for illuminating (or irradiating) a user's eyes, and an image sensor 113 for capturing an image of the user's eyes. The illuminators 111 and 112 can be, for example, light-emitting diodes that emit light in the infrared band or the near-infrared band. The image sensor 113 can be, for example, a camera, such as a complementary metal oxide semiconductor (CMOS) camera or a charge-coupled device (CCD) camera. The camera is not limited to an IR camera or a depth camera or a light field camera. The shutter mechanism of the image sensor can be a rolling shutter or a global shutter.

[0066] The first illuminator 111 is arranged coaxially with (or near) the image sensor 113 such that the image sensor 113 can capture a bright pupil image of the user's eye. Due to the coaxial arrangement of the first illuminator 111 and the image sensor 113, the light reflected from the retina of the eye passes back through the pupil and towards the image sensor 113, such that the pupil appears brighter than the iris surrounding it in the image of the eye illuminated by the first illuminator 111. The second illuminator 112 is arranged non - coaxially with (or further away from) the image sensor 113 for capturing a dark pupil image. Due to the non - coaxial arrangement of the second illuminator 112 and the image sensor 113, the light reflected from the retina of the eye cannot reach the image sensor 113, and the pupil appears darker than the iris surrounding it in the image of the eye illuminated by the second illuminator 112. The illuminators 111 and 112 can, for example, alternately illuminate the eye such that each first image is a bright pupil image and each second image is a dark pupil image.

[0067] The eye tracking system 100 further includes circuitry 120 (e.g., including one or more processors) for processing the images captured by the image sensor 113. The circuitry 120 can be connected to the image sensor 113 and the illuminators 111 and 112, for example, via a wired or wireless connection. In another example, the circuitry 120 in the form of one or more processors can be disposed in one or more stacked layers beneath the photosensitive surface of the image sensor 113.

[0068] Figure 2 An example of an image of the eye 200 captured by the image sensor 113 is shown. The circuitry 120 can, for example, employ image processing (such as digital image processing) to extract features in the image. The circuitry 120 can, for example, employ pupil - center corneal - reflection (PCCR) eye tracking to determine the position where the eye 200 is looking. In PCCR eye tracking, the processor estimates the position of the center of the pupil 210 and the position of the center of the glint 220 at the eye 200. The glint 220 is caused by the reflection of light from one of the illuminators 111 and 112. The processor uses the glint 220 to calculate the position of the user in space and uses the pupil 210 to calculate the position towards which the user's eye 200 is pointing. Since there is typically an offset between the optical center of the eye 200 and the fovea, the processor performs fovea - offset calibration in order to be able to determine the position where the user is looking. Then the gaze directions obtained from the left and right eyes can be combined to form a combined estimated gaze direction (or viewing direction). As will be described below, many different factors can affect how the gaze directions of the left and right eyes should be weighted relative to each other when forming this combination.

[0069] In reference Figure 1In the described embodiments, illuminators 111 and 112 are arranged in an eye tracking module 110 placed below a display viewed by a user. This arrangement is for illustrative purposes only. It will be understood that any number of more or fewer illuminators and image sensors may be employed for eye tracking, and such illuminators and image sensors may be distributed in a variety of different ways relative to the display viewed by the user. It will be understood that, for example, the eye tracking scheme described in the present disclosure may be employed for remote eye tracking (e.g., in a personal computer, a smart phone, or integrated in a vehicle), or for wearable eye tracking (such as in virtual reality glasses or augmented reality glasses).

[0070] Figure 3 is a block diagram showing a dedicated computer system 300 in which embodiments of the present disclosure may be implemented. This example shows a dedicated computer system 300, which may be used, in whole, in part, or with various modifications, to provide the component functions described herein.

[0071] The dedicated computer system 300 is shown as including hardware elements that may be electrically coupled via a bus 390. The hardware elements may include one or more central processing units 310, one or more input devices 320 (e.g., a mouse, a keyboard, an eye tracking device, etc.), and one or more output devices 330 (e.g., a display device, a printer, etc.). The dedicated computer system 300 may also include one or more storage devices 340. By way of example, the (multiple) storage devices 340 may be disk drives, optical storage devices, solid state storage devices (such as programmable, flash-updatable random access memory (“RAM”) and / or read-only memory (“ROM”)) and / or the like.

[0072] In addition, the dedicated computer system 300 may include a computer-readable storage medium reader 350, a communication system 360 (e.g., a modem, a network card (wireless or wired), an infrared communication device, a cellular communication device, a Bluetooth TM device, etc.), and a working memory 380, which may include RAM devices and ROM devices as described above. The working memory 380 includes the (multiple) operating systems 384 and other code 388. In some embodiments, the dedicated computer system 300 may also include a processing acceleration unit 370, which may include a digital signal processor, a dedicated processor, and / or the like.

[0073] Wearable computing device implementing eye tracking

[0074] Figure 4 shows an example of a wearable computing device 400 that implements as described in connection withFigures 1 to 2 Some or all of the above components of the described eye tracking system. The wearable computing device 400 can be a VR headset or an AR headset that can be worn by a user. As shown, the wearable computing device 400 includes a set of lenses 410 (such as Fresnel lenses), a set of cameras 420, a set of hot mirrors 430 (for example, as Figures 12 to 14 further shown in, in various embodiments this set of hot mirrors includes two hot mirrors for each eye), and a set of displays 440. The cameras 420 can include Figure 1 the image sensor 113. Although not shown in Figure 4 , the wearable computing device 400 can also include a set of illuminators and processing circuitry. These and other components can be integrated within the housing 450 of the wearable computing device 400. In this way, when the user mounts the wearable computing device 400 on his or her head, this set of lenses 410 will be relatively close to the user's eyes, and this set of displays will be relatively far from the user's eyes, and the remaining components can be located in between. The arrangement of these components allows for detecting the user's fixation point 460 in a three-dimensional virtual or real space.

[0075] Figure 5 Shows an example of components of a VR headset that implements some or all of the above components of the eye tracking system and the dedicated computer system described in connection with Figures 1 to 5 As shown, the components include a lens cup 510 (which can also be referred to as a lens barrel). In an example, the lens cup 510 is a housing made of a rigid material such as plastic and includes two opposite openings 512 and 514 along a transverse axis. The shape of the lens cup 510 is substantially a semi-conical (or funnel-shaped) housing along the axis, where the first opening 512 replaces the apex of a full cone, and the second opening 514 is at the base. A head-mounted (HMD) lens 520 is mounted in the first opening 512 and is protected by an illuminator cover 530. A light illuminator 540 is disposed within the cover 530 and outside the lens cup 510. Alternatively, the illuminator 540 can be exposed without the cover 530. The light illuminator 540 can include a set of light-emitting diodes (LEDs) that emit light in the infrared or near-infrared spectrum. One side of the lens cup 510 includes an opening and a wall that form a chamber 516. A camera module 550 is mounted within the chamber and its optical sensor faces the opening of the chamber 516. The components also include a hot mirror 560, a VR display 570, and an electronic board 580. The hot mirror 560 is mounted outside the lens cup 510 at a position close to the second opening 514. The VR display 570 is mounted behind the hot mirror 560 (e.g., further away from the second opening 514) relative to the second opening 514. The electronic board 580 includes Figure 3Some or all components of the dedicated computer system 300 (e.g., CPU, storage device, computer-readable storage medium reader, communication system, processing acceleration unit, and working memory).

[0076] Hardware calibration based on the position and rotation of the camera and lens during use

[0077] Generally, wearable computing devices (such as Figures 4 to 5 the wearable computing device shown in) are unstable (shaky) due to the random movement of the user. For example, a sudden drop may cause movement of internal components, especially the camera and lens. Misalignment of the camera and lens results in inaccurate eye tracking. To address the misalignment and improve accuracy, calibration can be used.

[0078] In existing systems, calibration would involve manually disassembling the wearable computing device and reinstalling these components in the correct position. And then starting the gaze calibration process from the beginning, which means that various parameters (e.g., focal length of the lens, lens diameter, focal length / FOV of the camera, camera position and orientation, position of the hot mirror, imaging parameters of the camera, lighting parameters, etc.) are required for the calibration process.

[0079] In contrast, embodiments of the present disclosure significantly simplify the calibration process. For example, calibration includes: determining misalignment based on the position and / or rotation of the camera and lens; and considering the position and / or the rotation when calculating gaze information.

[0080] The technical challenge of using this type of calibration is that there is no direct way to determine the position and rotation of the camera and lens during use. Embodiments of the present disclosure use an indirect method by placing markers on the lens and relying on the markers on the lens. Instead of considering the above parameters, the distance between the camera (e.g., the position of the image sensor plane or surface) and the markers on the lens (e.g., the x and y coordinates of the markers) is calculated based on one or more images captured by the camera. By using the distance values, the eye tracking system can adjust the calibration parameters of the camera (e.g., imaging parameters, focal length, distortion parameters, and lighting parameters). The camera can be: (i) at least one image sensor with an optical arrangement; or (ii) at least one lensless camera without an optical arrangement but with an ultrathin optical phased array (OPA), which manipulates incident light to capture images without using a traditional lens.

[0081] In an example, embodiments of the present disclosure include storing original distances and newly updated distance history information. The storage can be local or remote for the wearable computing device (e.g., retrievable from a remote server upon user identification). Distances are estimated by using images captured by an eye-tracking camera (e.g., an IR camera), which is referred to herein as the "camera". Each of these images captured for hardware calibration is stored locally or remotely in non-volatile (e.g., flash) memory. Whenever the wearable computing device is started, new images can be captured as described herein. The newly captured images can be used in conjunction with the most recently stored (at least one) image to identify whether any relative movement of the marker has occurred. If so, the newly captured image is at least further analyzed to perform calibration. Otherwise, calibration is skipped.

[0082] The markers are on the same side of the camera and preferably can be placed on and / or embedded within one of the lenses. If there are two or more lenses, one or more markers are placed on and / or embedded within each of the lenses because any accidental drop may cause inconsistent movement of each of the lenses and its corresponding components. This type of calibration can also be used in other scenarios. For example, one or more of the lenses can be moved to compensate for visual artifacts such as myopia or hyperopia. In such scenarios, the optical settings can be visually detected in a similar manner (e.g., by relying on the distance from the camera to the marker and / or the x and y coordinates of the marker). The marker can be a cross marker, a dot pattern (e.g., an LED illuminator pattern within the lens), or a line. The marker can be placed in each or some of the optics of the lens group. For example, the marker can be placed on one or more layers of a Fresnel lens and even on a heat mirror. Different markers can be used in different lens elements. For example, a cross can be used as the marker on one lens, a circle as the marker on another lens, and a triangle as the marker on a third lens. In this way, the type of marker (e.g., as detected in the image) can be used to associate the marker with the lens.

[0083] Since the original relative position and alignment of the camera and the lens are known, once the lens or the camera is moved, by using the estimated position of the marker, we can "reconstruct" the new relative position and alignment of the camera and the lens. Thus, the eye-tracking camera and the illuminator will be automatically and immediately recalibrated.

[0084] Calibration can be triggered in different ways. In one way, calibration is always on during the life of the head-mounted device as long as the head-mounted device is powered on or when the user's eyes are detected by the eye-tracking camera. In another way, sensors can be used to trigger calibration. For example, an accelerometer and / or a gyroscope are used for sudden movement detection and then trigger the calibration process. In yet another way, the user can use a manual control (e.g., a physical button on the VR head-mounted device) to initiate the recalibration process. In some or all ways, warning feedback (audible or visible warning feedback or vibration) will be given in case of imperfect eye tracking (e.g., in response to sensor triggering). In a further example, the trigger is the detection of the user's presence (or absence). For example, calibration can be performed when the user's presence is detected after a predefined period of time (such as 20 seconds) without seeing the user.

[0085] Before performing hardware calibration or before the runtime of the eye-tracking head-mounted device, the position and alignment (factory default) information or data of each component inside the eye-tracking head-mounted device are stored in a computer-readable medium and are known or input into the system. The components at least include a hot mirror 560, an illumination module (530 + 540), a camera module 550, a lens cup 510, and a VR display 570. The position and alignment data of the components are represented by a 3D coordinate system.

[0086] In an embodiment, an augmented reality device, a virtual reality device, or other wearable devices are provided. Figures 4 to 5 A wearable computing device is an example of such a device. In an example, the device includes an eye-tracking device and one or more processors. The eye-tracking device includes an image sensor (such as a camera) and a first lens. The first lens includes a first marker. For example, the first marker is on the surface of the first lens or embedded in the first lens. The position of the first marker is within the field of view (FOV) of the camera. The first marker is selected from the group consisting of a cross marker, a dot pattern, a dot, a line, and a geometric figure. Additionally or alternatively, the first marker includes a light-emitting element (e.g., one or more LEDs). The device or the eye-tracking device further includes a second lens, and the second lens includes a second marker. The second marker is located at a layer of the lens. Additionally or alternatively, the device further includes a hot mirror, and the hot mirror includes another marker.

[0087] The one or more processors are configured to (e.g., based on computer-readable instructions stored on one or more non-transitory computer-readable storage media) at least: receive a first image from an image sensor, wherein the first image shows a first marker; determine a first distance from the image sensor to the first marker based on the first image; and change at least one calibration parameter of an eye tracking algorithm used with the eye tracking device based on the first distance.

[0088] In an example, changing the at least one calibration parameter is further based on the first distance being different from a second distance, wherein the second distance is a previously measured and stored distance. In an example, changing the at least one calibration parameter based on the first distance includes: determining another distance from the image sensor to a first lens based on the first distance and a known position of the first marker at the first lens; and changing the at least one calibration parameter of the eye tracking algorithm used with the eye tracking device based on the second distance. The one or more processors may further be configured to determine an orientation of the image sensor relative to the first marker based on the first image. If so, then changing the at least one calibration parameter is further based on the orientation.

[0089] In a further example, the change to the calibration parameter is performed when: (a) the newly determined distance is different from a previously determined value; and (b) the adjustment to the calibration parameter is based not only on the newly determined and different distance value, but also on input position and alignment data of components.

[0090] In an example, the first image shows a second marker if the second marker is included in the first lens. The one or more processors are further configured to: determine a second distance from the image sensor to the second marker based on the first image; and change at least one calibration parameter of the eye tracking algorithm based on the second distance. In another example, if the second marker is included in a second lens, the one or more processors are further configured to: receive a second image from the image sensor, wherein the second image shows the second marker; determine a second distance from the image sensor to the second marker based on the second image; and change at least one calibration parameter of the eye tracking algorithm based on the second distance. In yet another example, if the second marker is included in a hot mirror, the first image or the second image received from the image sensor shows the second marker. The one or more processors are further configured to: determine a second distance from the image sensor to the second marker based on the first image or the second image; and change at least one calibration parameter of the eye tracking algorithm used with the eye tracking device based on the second distance. In these different examples, the at least one calibration parameter is selected from the group consisting of: imaging parameters, focal length, distortion parameters, and illumination parameters.

[0091] Combine Figures 6 to 8 These and other features are further described. Figure 6 Examples of marks on or in the lens are shown. Figure 7 Examples of an image of the mark and examples of calculating distance and direction based on the image are shown. Figure 8 An example of a process for updating calibration parameters is shown.

[0092] Figure 6 Examples of marks 610 on or in lens 620 according to certain embodiments of the present disclosure are shown. As shown, lens 620 may have a geometry (e.g., circles and ellipses) and have dimensions suitable for being assembled in a wearable computing device such as a VR headset or an AR headset. For example, lens 620 is a Fresnel lens.

[0093] Mark 610 may be attached (e.g., glued) or embedded (e.g., etched or textured) on the outer surface of lens 620, or may be embedded within the internal volume of lens 620. In an example, mark 610 may have a shape such as a cross mark, a dot pattern, dots, lines, geometries, etc., and / or may include light-emitting elements (e.g., an illumination matrix including a plurality of LEDs arranged in a predefined pattern on or embedded in the lens). The shape and / or any pattern conveyed by the shape and / or light-emitting elements may encode or present information about mark 610. This information may be used to calculate distance and direction. For example, the information may indicate the size of mark 610 (which allows establishing a pixel scale based on image analysis, and the pixel scale is available during calculation, as further described in combination with Figures 7 to 8 ). Further, the information may indicate the original position of mark 610 on or in lens 620 (e.g., the top of the lens, within five millimeters of the outer perimeter of lens 620), and the original position may also be used during calculation. The information may also identify mark 610 or lens 620 (which may be used to retrieve relevant information about size, position, and / or previous distance and direction from memory).

[0094] Different techniques may be used to adjust one or more calibration parameters based on the position and alignment of components of a wearable computing device (e.g., a lens and an image sensor). Generally, existing position and alignment data of the components may be obtained from memory. An image is captured by the image sensor and shows the mark on the lens. The distance between the mark and the lens is calculated based on the image. The (multiple) calibration parameters are adjusted based on the existing position and alignment data of the internal components and based on the calculated distance.

[0095] Different techniques can be used to calculate the relative movement between components of a wearable computing device based on image data. In an example, the position and alignment data of each component are mapped into a three-dimensional coordinate system. The system also includes the position of a marker. The relative distances and positions of the components are initially known to the system or stored based on previously determined results. Once the image sensor detects the movement of the marker, the movement can be calculated based on information from the image and information obtainable from the three-dimensional coordinate system. Since all relative positions and alignments are mapped on the same three-dimensional coordinate system, the new relative positions and alignments of each of the components can be automatically calculated by remapping the components to the three-dimensional coordinate system based on the image data generated by the image sensor. The difference between the new mapping and the previous mapping indicates the new relative positions and alignments for changing the (multiple) calibration parameters.

[0096] Figure 7 Examples of images of a marker and examples of calculating distance and direction based on the images are shown in accordance with certain embodiments of the present disclosure. In an example, the distance is the straight-line distance between the marker (e.g., a point on the surface of the marker) and the camera 710 (or a point on the surface of the image sensor of the camera 710). This straight-line distance indicates the position of the camera 710 and the lens (e.g., a value representing the distance interval between the two). The direction is the direction in which the marker has moved from a previous position. This direction indicates the rotation of the lens relative to the camera 710 (or vice versa). The determination of the marker movement is not limited to the foregoing direction calculation. The reasons for such movement may vary due to different manufacturing methods of the components.

[0097] The number of images used to derive the distance and direction may depend on the information conveyed by the marker and / or related information obtainable from the memory. In an example, when the marker encodes or presents its size and original position, a single image is sufficient. Similarly, when the marker identifies itself or the lens and information about the size and original position can be obtained from the memory based on the identifier, a single image is sufficient. In these examples, the single image shows the marker in its current position based on the movement of the camera 710 and / or the lens. Otherwise, two or more images can be used. One of these images shows the previous position, which corresponds to the previous distance obtainable from the memory. Another one of the images shows the current position for which the distance and direction should be calculated. The difference between the previous position and the current position indicates the direction. The use of a single image is described in more detail next, followed by a description of using two images.

[0098] Generally, distance and direction can be defined relative to a coordinate system (e.g., a coordinate system centered at the center of the image sensor of camera 710), and can be represented as a vector (the distance is the magnitude of the vector and the direction is the direction of the vector). The position of the camera (as determined by the (multiple) images) is mapped to coordinates in the coordinate system. Further, in the coordinate system, the vector can start at camera 710 and terminate at the marked coordinates. Geometric reconstruction is applied to calculate the direction and magnitude of the vector based on the (multiple) images.

[0099] In the case of a single image, the image shows the marker in its current position. The size of the marker can be identified directly from the encoded information in the image or indirectly from the relevant information stored in the memory. The size is compared with the number of pixels occupied by the marker in the image. This comparison establishes a pixel ratio (e.g., if the marker is ten millimeters wide and occupies ten pixels in the image, then each pixel in the image corresponds to one millimeter).

[0100] Thereafter, the distance and direction are calculated based on geometric reconstruction using the pixel ratio. For example, a right triangle is formed between camera 710 and the current position of the marker in the image (the hypotenuse is the position from camera 710 to the current position) and between the current position and the edge of the image (e.g., the right angle side). The (multiple) interior angles and the right angle side are determined from the image based on the pixel ratio. Trigonometric operations are used to calculate the hypotenuse (e.g., the distance).

[0101] To calculate the direction from a single image, the current position of the marker in the image is determined, and the original position is accessed directly (e.g., from the encoded information shown in the image) or indirectly (e.g., from the relevant information in the memory). These two positions are compared, and the pixel ratio is applied to the difference between the two positions to derive the direction.

[0102] In the case of using two or more images, the one or more previous positions of the marker in one or more previous images are compared with the current position of the marker in the current image. The previous positions correspond to the previous known directions and positions. The difference with the current position can be used to generate a difference vector. The pixel ratio can be calculated as above or based on the previous position and previous distance. The pixel ratio is applied to the difference vector. The magnitude and direction of the difference vector represent the direction of the marker. Geometric reconstruction is applied to derive the direction from the previous direction and the difference vector.

[0103] In Figure 7The use of two images is further shown in [description]. The difference between the two images is shown as the difference image 720. A marker moves from its previous position 730 (as shown in the previous image) to its current position 740 (as shown in the current image). The difference between the two establishes the difference vector 750. Since the previous distance 760 between the previous position 730 and the camera 710 is known, a pixel scale can be derived and applied to the difference vector 750 to calculate the magnitude of the difference vector. The magnitude and direction indicate the movement of the marker to the new position 740. Then geometric reconstruction is applied to calculate the rotation angle 780 and the new distance 790 between the marker at the current position 740 and the camera 710.

[0104] Of course, the marker may not have moved. Instead, the camera 710 may have moved and a similar approach (single or multiple images) can be used. Similarly, if both components have moved, a similar approach can still be used.

[0105] Figure 8 An example of a process for updating calibration parameters in accordance with certain embodiments of the present disclosure is shown. The process begins at operation 802, in which an image is received by a computer system (e.g., Figure 5 the electronic board 580). The image shows at least one marker. The marker is on or in the lens. The image is generated by a camera (e.g., captured by an image sensor of the camera).

[0106] At operation 804, the computer system determines the distance and / or direction from the camera (e.g., from the image sensor) to the marker based on the image. As described in connection with Figures 6 to 7 various approaches are possible. In one approach, a single image may be sufficient. In another approach, a history of the distance and / or position is accessed. The history may include previous images showing the marker at previous positions. Generally, geometric reconstruction is applied in both approaches to measure the distance and direction.

[0107] At operation 806, the computer system determines the distance and / or direction from the camera (e.g., from the image sensor) to the lens based on the known position of the marker on the lens. For example, this distance and direction is from the center of the image sensor to the center of the lens. The known position of the marker identifies a vector between the center of the marker and the center of the lens. This vector is used to derive the distance and direction between the camera and the lens based on the distance and direction in operation 804.

[0108] At operation 808, the computer system changes at least one calibration parameter of an eye tracking algorithm used with an eye tracking device based on the distance and / or orientation relative to the marker and / or lens as calculated at operations 804 and 806. In an example, changing the calibration parameter includes changing one or more of an imaging parameter, a focal length, a distortion parameter, and an illumination parameter. The change can be specified in a table stored in a local memory of the computer system or in a remote memory accessible by the computer system via a data network. The table associates values of the calibration parameter with the distance and / or orientation between the camera and another component (e.g., a lens, a heat mirror, etc.). The values in the table can be derived during the design, development, and testing phases of the wearable computing device in a controlled laboratory environment or during a trial operation phase.

[0109] At operation 810, the computer system determines whether any other markers are available in the image. The presence of another marker can be based on image analysis or on prior knowledge about the expected number of markers. The other marker can be located on or in the same lens, on another lens, or on a heat mirror. If so, operation 804 is then performed. In this way, additional distances and / or orientations can be calculated to update the calibration parameter. More specifically, if the other marker is on the same lens, the accuracy of the distance calculation and the orientation calculation can be increased (e.g., by averaging the two vectors, one vector for each marker on the lens). If the other marker is on a different lens or on a heat mirror, the (multiple) distances and (multiple) orientations from the camera to the other marker can be calculated and used similarly to further update the (multiple) associated calibration parameters as specified in the table. If no other markers are shown in the image, the process moves to operation 812.

[0110] At operation 812, the computer system determines whether any other images showing other markers are available. If so, operation 804 is then performed. Again in this way, additional distances and / or orientations can be calculated to update the calibration parameter. Otherwise, the process moves to operation 814.

[0111] At operation 814, the computer system determines whether there is a trigger for repeating the calibration. Various types of triggers are possible. In one example type, if the wearable computing device is powered on or when the user's eyes are detected by its eye tracking system, the calibration is repeated. In another example type, a sensor can be used to trigger the calibration. For example, an accelerometer or a gyroscope is used for sudden movement detection and then triggers the calibration process. In yet another example type, the user can use a manual control (e.g., a physical button on a VR headset) to initiate the recalibration process. If a trigger is received, the process loops back to operation 802 (e.g., starts and repeats based on the trigger operation 802). Otherwise, the process ends.

[0112] Distance determination is not limited to using a camera-based sensing system. A camera-less sensing system is also possible. For example, a specific type of microelectromechanical system (or MEMS) can also be implemented to achieve a similar result.

[0113] Hardware calibration based on distortion compensation

[0114] Generally, lenses in wearable computing devices (such as Figures 4 to 5 the wearable computing device shown in suffer from pincushion distortion. This distortion strongly and directly affects eye image capture and indirectly affects gaze tracking performance. Technically, due to lens pincushion distortion, the captured eye images may have a lot of distortion, especially when the eye moves away from the lens (relative to a threshold).

[0115] In other cases, there may be other types of distortion, which are generally classified as barrel distortion and mustache distortion. The distortion can be irregular or follow many patterns. In VR implementations, the most common distortion is radially symmetric. However, depending on the relative placement and alignment between the camera and the lens, the eye images may also suffer from asymmetric distortion. In summary, any type of geometric distortion is possible and is strongly affected by the lens design. And the distortion correction model adapts to various types of distortion.

[0116] A distortion compensation model is needed to correct the captured eye images for gaze tracking. This distortion correction model depends on the distance from the corneal center to the lens center (shown as D in Figure 9 and Figure 10 ). This means that the distortion model should be adapted to the corneal-to-lens distance at each frame to compensate the eye images for gaze tracking. Further, the distortion model to be used can depend on the distance to the eye and the position of the eye in the x and y coordinates relative to the lens and the display. 0 )).

[0117] Therefore, the updated value of the corneal center to the lens center (estimated per frame) is fed into the distortion correction model. To do this, a nominal (default) value is set for the corneal center to the lens center (based on statistical averages). The distortion correction model is established based on the default corneal center to the lens center. During eye tracking, the corneal center to the lens center is automatically estimated at a certain rate, such as based on each eye image or alternative frames or based on a trigger (which can be manually adjusted by the user or automatically adjusted according to the movement of the head-mounted device (e.g., whether the user frequently moves the head-mounted device), and the movement is detected by eye image capture and on-board accelerometers and gyroscopes) to estimate the corneal center to the lens center (such as the distance D shown in Figures 9 to 10 and 1 D,2 , D 3 ) value. Estimate and update the distance from the corneal center to the lens center. Use the newly estimated and updated distance from the corneal center to the lens center to establish a new / updated distortion correction model. And the process loops back and forth between establishing the distortion correction model and using the new estimate to update this model.

[0118] If the distance from the corneal center to the lens center changes, as Figures 9 to 10 shown, the distance changes, and assuming the original distance (e.g., D 0 ) is the optimal distance. Generally speaking, the longer the distance (e.g., D 3 ), the more pincushion distortion exists.

[0119] In an embodiment, an augmented reality device, a virtual reality device, or other wearable device is provided. Figures 4 to 5 The wearable computing device of is an example of such a device. In the example, the device includes an eye tracking device and one or more processors. The one or more processors are configured to (e.g., based on computer-readable instructions stored in one or more non-transitory computer-readable storage media) at least: receive image data from an image sensor, where the image data corresponds to an image observed through a lens; determine the degree and pattern of pincushion distortion in the image based on the image data; and change at least one calibration parameter of an eye tracking algorithm used with the eye tracking device based on the degree and pattern of pincushion distortion.

[0120] In the example, changing the at least one calibration parameter based on the degree or pattern of the pincushion distortion includes: determining the relative position or relative orientation of the image sensor based on the degree or pattern of the pincushion distortion; and changing the at least one calibration parameter of the eye tracking algorithm used with the eye tracking device based on the relative position or the relative orientation of the image sensor.

[0121] In the example, the device further includes a motion sensor. The at least one processor is further configured to: receive a signal from the motion sensor; determine based on the signal that the device has accelerated or decelerated beyond a threshold amount; and initiate at least one action in response to determining that the device has accelerated or decelerated beyond the threshold amount. The at least one action includes selecting at least one of the group consisting of: issuing an alert to a user of the device, and changing at least one calibration parameter of an eye tracking algorithm.

[0122] In combination with Figures 9 to 11 These and other features are further described. Figure 9 An example of the distance from the corneal center to the lens center is shown. Figure 10An example of an image showing an example of pincushion distortion. Figure 11 An example of a process for updating calibration parameters based on an eye image, where the image suffers from pincushion distortion.

[0123] Figure 9 An example of the distance from the corneal center to the lens center according to certain embodiments of the present disclosure. Although Figure 9 Shown as D 0 , ……, D 3 Four distances, but depending on the degree or pattern of pincushion distortion, any number of distances are possible and can generally be referred to as D i , where "i" is a positive integer. The corneal center represents the center of the cornea. The corneal center is determined in three-dimensional space. To determine the corneal center, several parameters need to be considered, including the position of the camera and the positions of at least two illuminators, at least two flash positions on the spherical region of the cornea (caused by at least two different illuminators), and the corneal radius. The lens center represents the center of the lens. The distance from the corneal center to the lens center represents the distance between the center of the cornea and the center of the lens.

[0124] As shown, the lens 910 is mounted in a wearable computing device (not shown). The nominal value of the distance between the center 912 of the lens and the center 924 of the cornea 922 of the user's human eyeball 920 is defined as D 0 930. This nominal value represents the default distance from the corneal center 924 to the lens center 912 and when the wearable computing device is mounted on the user's head, and the nominal value is defined based on the statistical average of different users wearing such a device (or by using data modeling and / or actual data).

[0125] When the eye moves away from this default distance, the degree and / or pattern of distortion can change. The movement can be caused by different reasons, including a change in the position of the lens 910 in the wearable computing device (e.g., intentionally based on user adjustment, or unintentionally due to general movement, and sometimes a sudden drop of the user's wearable computing device).

[0126] When the human eyeball 920 moves further away from the center 912 of the lens, the degree and / or pattern of pincushion distortion change. The degree indicates the amount of distortion that will be present in the image of the human eyeball 920, where this image is used for eye tracking. The pattern indicates how the distortion is presented in the image.

[0127] Therefore, at a distance D 1 940 representing another value of the distance from the corneal center to the lens center, the degree and pattern of pincushion distortion are different from those at distance D 0The degree and pattern at 930. Similarly, at a distance D representing another value from the corneal center to the lens center 2 950, the degree and pattern of pincushion distortion are different from those at distance D 0 930 and D 1 940. Similarly, at a distance D representing another value from the corneal center to the lens center 3 960, the degree and pattern of pincushion distortion are different from those at distance D 0 930, D 1 940, D 2 950. These distances D 1 940, and D 2 950, and D 3 960 each represent a change in the default distance D 0 930. The greater the distance, the more distortion there is (e.g., the amount of distortion increases and the pattern becomes more prominent).

[0128] Figure 10 An example image showing an example of pincushion distortion according to certain embodiments is presented. As Figure 9 shown, four distances D 0 , ……, D 3 (corresponding to the distances in Figure 9 ) are presented to show pincushion distortion. However, depending on the degree or pattern of pincushion distortion, any number of distances are possible and can generally be referred to as D i , where "i" is a positive integer. Although not shown in Figure 10 , the presented distortion is of an image of the user's eye wearing a wearable computing device. And this image is used for eye tracking.

[0129] As shown, the pincushion pattern varies with distance D i . The greater the distance D i , the greater the amount of distortion (e.g., degree), resulting in different patterns. A specific shape such as a distorted rectangle (e.g., the vertical sides of the triangle are convex lines instead of straight lines) is used to show the distorted pattern. The degree of distortion is shown by the size, area, and / or perimeter of the specific shape.

[0130] Thus, at a distance D 0 1010 representing the default value from the corneal center to the lens center, the default distortion can be identified through the eye image (shown as a distorted rectangle at the center). At a distance D 1 1020 representing another value from the corneal center to the lens center, the degree and / or pattern of distortion differ from that at distance D 0Degree and / or pattern changes at 1010 (the changes are shown using a larger distorted rectangle). Similarly, at a distance D representing another value from the corneal center to the lens center 2 At 1030, the degree of distortion and / or the pattern vary from the distance D 0 Between 1010 and D 1 Degree and / or pattern changes at 1020 (the changes are shown using an even larger distorted rectangle). Similarly, at a distance D representing another value from the corneal center to the lens center 3 At 1040, the degree of distortion and / or the pattern vary from the distance D 0 Between 1010, D 1 Between 1020, and D 2 Degree and / or pattern changes at 1030 (the changes are shown using the largest distorted rectangle).

[0131] Figure 11 An example of a process for updating calibration parameters based on an eye image according to certain embodiments of the present disclosure is shown, where the image suffers from pincushion distortion. The process starts at operation 1102, in which a default distance from the corneal center to the lens center is set by a computer system (e.g., Figure 5 electronic board 580). This default value can be stored in the local memory of the computer system and can represent the average distance from the corneal center to the lens center.

[0132] At operation 1104, the computer system establishes a distortion correction model based on the default distance from the corneal center to the lens center. For example, the computer system loads this model from memory for eye tracking. In an example, a prediction model is used, and the prediction model associates the degree and / or pattern of pincushion distortion with the values of one or more calibration parameters (e.g., values of imaging parameters, focal length, distortion parameters, and illumination parameters, etc.). The association can include a mapping between known images of potential pincushion distortion, the degree and / or pattern of potential pincushion distortion, the distance from the corneal center to the lens center, and the values of the calibration parameters in the prediction model. These known images can be stored in the prediction model itself, or separately in the local memory of the computer system or in a remote memory accessible by the computer system through a network. The values can be derived during the design, development, and testing phases of the wearable computing device in a controlled laboratory environment, or during a trial operation phase. Establishing the distortion correction model includes: for example, accessing values corresponding to the pincushion distortion that should be observed at the default distance from the corneal center to the lens center from the prediction model; and specifying that these values should be used for eye tracking to compensate for the distortion.

[0133] At operation 1106, the computer system begins eye tracking. Different triggers are possible. In an example, the tracking begins in response to the wearable computing device being turned on (either based on sensing that the device has been worn on the user's head or based on a user manual input received at the wearable computing device). Generally, eye tracking involves analyzing an image of at least one of the user's eyes. Such an eye image may suffer from pincushion distortion, depending on the distance from the center of the cornea to the center of the lens.

[0134] At operation 1108, the computer system receives image data from an image sensor (e.g., a camera) of the wearable computing device. This image data represents an image of the user's eye and is received as part of the eye tracking.

[0135] At operation 1110, the computer system determines the degree and / or pattern of pincushion distortion in the image based on the image data. In an example, the computer system applies pattern recognition or image analysis algorithms to identify the degree and pattern. In another example, the computer system matches the image data with one of the known images available from a prediction model, local memory, or remote memory. For example, the best match is identified and the matched known image is used. The known image is associated with the degree and pattern of pincushion distortion in the prediction model.

[0136] At operation 1112, the computer system estimates the updated distance from the center of the cornea to the center of the lens. In an example, this updated distance is derived by a pattern recognition or image analysis algorithm (if one of the pattern recognition or image analysis algorithms was used in operation 1110). In another example, the updated distance is determined by a prediction model, where this model associates the degree and pattern of pincushion distortion with the updated distance from the center of the cornea to the center of the lens. Generally, the distance indicates the relative position or relative orientation of the image sensor based on the degree or pattern of pincushion distortion.

[0137] At operation 1114, the computer system changes the distortion correction model. In an example, the computer system accesses the values of the calibration parameters associated with the updated distance from the center of the cornea to the center of the lens (or equivalently, the degree and / or pattern of pincushion distortion) and updates the distortion correction model by specifying that these values should be used in conjunction with the eye tracking.

[0138] At operation 1116, the computer system changes at least one calibration parameter of the eye tracking algorithm used with the eye tracking device based on the degree or pattern of pincushion distortion. For example, the values of the relevant calibration parameters are changed as identified by the prediction model.

[0139] At operation 1118, the computer system determines whether there is a trigger for repeating the change. If so, the process loops back to operation 1108. Otherwise, the process ends. Various triggers are possible. In one example, the change is automatically repeated at a frame interval (e.g., for each received image, or for every other image). In another example, the computer system receives a signal from a motion sensor of the wearable computing device. Based on the signal, the computer system determines that the wearable computing device has been moved and that the movement indicates that the wearable computing device has accelerated or decelerated beyond a threshold amount. The computer system then initiates at least one action in response to determining that the device has accelerated or decelerated beyond the threshold amount. The actions include repeating the change and issuing an alert to the user. For example, the alert is regarding the change and can be presented visually on a display and / or auditorily on a speaker.

[0140] Optimized Fresnel lens

[0141] Many wearable computing devices use Fresnel lenses. A typical construction of a Fresnel lens is a lens with rings having a large and constant step size. The problem with these Fresnel steps is that they are clearly visible in an eye tracking image (i.e., an eye image) because they locally distort the image and they scatter light from an illuminator that can be interpreted as a flash. Thus, the eye image is useless or difficult to use for gaze determination.

[0142] Embodiments of the present disclosure relate to an optimized Fresnel lens that improves eye tracking and enhances the user experience. In an example, the Fresnel steps are smaller than a certain step size. In a central region of the Fresnel lens, there may be no Fresnel steps at all. By not having any Fresnel steps in the central region, stray light is minimized (thereby enhancing the user experience), the image quality is improved, and there are no distorted flashes (thereby improving eye tracking). The Fresnel steps / rings gradually surround the clear central region with an increasing groove depth and possibly an increasing step size.

[0143] Accordingly, the Fresnel lens includes a flat central region without gaps and variable-sized Fresnel lens steps outside the central region. Regarding the smaller size of the Fresnel lens steps, the step size / spacing of the Fresnel rings is less than: the pixel size of the camera pixels (or, if pixel binning is used, the size of the binned pixels) divided by the focal length of the camera, multiplied by the optical distance between the camera and the lens, and then multiplied by two. A Fresnel ring spacing much smaller than this value may generate unnecessary stray light. Outside the central flat region, the groove depth of the Fresnel lens gradually increases and possibly the step size increases.

[0144] Furthermore, a two-layer (or multi-layer) Fresnel lens configuration is possible. Each layer has different lens steps and sizes. The size of the central flat area can also be different.

[0145] In addition, stray light can be reduced by introducing anti-reflection (AR) technology. Standard anti-reflection coatings require complex coating processes that are expensive and may not be well-suited for high production volumes. Other technologies such as moth-eye structures (ME) or etc. may be more suitable. The resulting surfaces are sensitive to contaminants, so these technologies should not be used on user-facing surfaces, but can be used on all other surfaces. These AR technologies can even be used on Fresnel surfaces.

[0146] Figure 12 An example of an optimized Fresnel lens 1200 according to an embodiment of the present disclosure is shown. As shown, the Fresnel lens 1200 includes a central (or center) region 1210 and a Fresnel ring 1220 surrounding the central region 1210. The central region 1210 does not have any such rings and is optically clear so that light can pass through the region. Different shapes and sizes of the central region 1210 are possible, depending on the shape and size of the Fresnel lens 1200 and its placement relative to other components in the wearable computing device (e.g., a camera, an area for receiving the user's eye, etc.). In the example, the Fresnel lens 1200 has a circular cross-section with a diameter in the range of ten to fifty millimeters. The central region 1210 is also circular and has a diameter in the range of 1% to 25% of the diameter of the Fresnel lens 1200.

[0147] In the example, the Fresnel ring 1220 includes a plurality of rings that circumferentially surround the central region 1210. The number and size of these rings 1220 depend on the shape and size of the Fresnel lens 1200 and / or the central region 1210. Generally, each of the Fresnel rings 1220 has a groove depth and a step size. The groove depth and / or the step size increase as the diameter of the Fresnel ring 1220 increases. The groove depth can be in the micrometer range. The step size can be in the millimeter range.

[0148] Figure 13Shows an example of a two - layer Fresnel lens 1300 according to an embodiment of the present disclosure. The first layer 1310 includes a central region 1312 and a plurality of Fresnel rings 1314 surrounding the central region 1312. Similarly, the second layer 1320 includes a central region 1322 and a plurality of Fresnel rings 1324 surrounding the central region 1322. Generally, these central regions 1312 and 1322 are aligned, and their centers may, but need not, be on the same axis (the horizontal axis passes through their centers). As shown, the central regions 1312 and 1322 are of different sizes. The sizes of the Fresnel rings 1314 and 1324 may also be different.

[0149] Virtual environment control

[0150] Figures 14 to 21 Shows different examples for manipulating virtual objects in a virtual environment. For the sake of brevity, virtual objects are referred to as "objects" in this document. Generally, objects can be represented in different layers of the virtual environment. Each object has a set of parameters (referred to as rendering parameters in this document) that control how the object is presented and manipulation operations applicable to the object (referred to as manipulation parameters in this document).

[0151] The external controller is presented as a virtual controller in the virtual environment. Also for the sake of brevity, the virtual controller is referred to as a "controller" in this document (while the external controller is a physical controller). The external controller can have one or more touch areas, such as a touch - sensitive surface, tactile buttons, key switches, etc. These touch areas can be presented as virtual menus (referred to as "menus" in the virtual environment in this document).

[0152] Placing the menu in a virtual reality or augmented reality environment is important for the user experience. Especially in a complex virtual background, placing the menu in the appropriate position / alignment makes the user's operations more natural and intuitive. In the present disclosure, a combination of gaze - based interaction with objects and controls from the external controller is used to trigger, place, and operate the menu.

[0153] In an example, the user's gaze in the virtual environment is tracked. If the user gazes at an object for a period longer than a time threshold, the object is selected. The selection can lock the object so that the user can move their line of sight or gaze away from the object, while the object will still be selected and available for manipulation. Thereafter, depending on the user's hold on the touch area on the external controller, a menu specific to the touch area is presented in the virtual environment. The menu provides options for manipulating the object according to the object's parameters. The user operates the touch area on the external controller to view and select the available menu options and manipulate the object in the virtual environment.

[0154] In an example, the rendering parameters of an object indicate the virtual environment layer in which the object should be rendered, as well as the look and feel of the object. The manipulation parameters of the object indicate editable visual characteristics (e.g., size, shape, color), constraints for moving and placing the object in the virtual environment (e.g., can the object be placed adjacent to another object), and associations with other objects. A menu corresponding to a touch area allows the object to be edited and manipulated as set in the rendering parameters and manipulation parameters of the object.

[0155] Different types of associations can be defined. These associations affect how an object can be manipulated based on other objects in the virtual environment and thus can control how an external controller can be used to manipulate these objects. "Relationship association" is an example and specifies how two objects are related. For example, in Figure 16 a parent-child association is described. In Figure 16 , a 3D printer is associated with a printed object, where the printer is the parent and the printed object is the child. In this case, the 3D printer can be manipulated to create the printed object. Another example of an association is a "manipulation association", which specifies whether the manipulation of an object should be initiated from or transferred to another object and how the two objects can be manipulated together. For example, in Figure 14 a tree is described. The manipulation of the tree is independent of other objects in the environment based on its manipulation association, indicating that the tree is not associated with other objects. In contrast, in Figure 18 a card game is shown. In Figure 18 , a card is associated with a slot on a table for receiving the card. The manipulation of the card can be transferred to the slot so that the card can be placed in the slot.

[0156] Figure 14 An example of manipulating an object 1410 in a virtual environment 1400 according to an embodiment of the present disclosure is shown. A wearable computing device (such as a VR or AR headset as shown in Figures 4 to 5 ) presents the virtual environment 1400. The wearable computing device is communicatively coupled to an external controller via a data network. The external controller can include a plurality of touch areas, each of which provides controls. The data network can be a wired or wireless network, including a local area network, a peer-to-peer network, a communication bus, or any other data network suitable for exchanging control information between the external controller and the wearable computing device. The virtual environment 1400 can be a virtual reality environment in the case of a VR headset or an augmented reality environment in the case of an AR headset.

[0157] As shown, the virtual environment 1400 presents an object 1410. This object 1410 has a set of parameters that enable its interaction. Specifically, a user can interact with the object 1410 in the virtual environment, where the interaction involves manipulating the object by changing one or more of the characteristics of the object 1410. The object 1410 is shown as a three-dimensional tree.

[0158] The wearable computing device presents the virtual environment 1400 on a set of displays and tracks the user's gaze as he or she observes the set of displays and looks at the presented virtual environment 1400. In an example, based on the tracking and the mapping between the set of displays and the virtual environment 1400, the wearable computing device presents an indication 1420 of the user's gaze position in the virtual environment. For example, the indication 1420 is presented as a geometric shape (e.g., a colored circle) in a layer of the virtual environment 1400. In another example, the gaze indication 1420 may be invisible to the user or may not be displayed. In fact, during such gaze-based interaction, the user already knows what he and she is looking at. Thus, it may not be necessary to present the indication 1420.

[0159] Further, when the user moves an external controller, the wearable computing device detects the movement based on data communication with the external controller over a data network and displays a virtual controller 1430 in the virtual environment 1400. This controller 1430 can have an appearance (relative shape and size) that mimics the external controller.

[0160] If the user holds or touches the touch area of the physical controller, the computing device gets an indication of this "physical integration" in the physical environment from the external controller and updates the controller 1430 to highlight the corresponding virtual touch area 1432. As Figure 14 shown, when the user interacts with the physical touch-sensitive area on the external controller, the corresponding area 1432 is highlighted on the controller 1430 in the virtual environment.

[0161] If the wearable computing device detects the user's gaze on the object 1410 (e.g., the three-dimensional tree) for a predefined period of time, the wearable computing device may lock the gaze to the object 1410. Thereafter, the user can move the gaze away from the object 1410 and still control the object 1410 via the external controller.

[0162] Other techniques for locking object 1410 are possible. For example, a gaze zone around object 1410 is predefined. If the wearable computing device detects that the user's gaze has fallen within this zone and, in parallel, receives an indication of a user action from an external controller, the wearable computing device locks target 1410. The gaze zone can be predefined based on parameters of the object. For example, the gaze zone is a geometric region around object 1410 and is thus typically larger than the size of object 1410.

[0163] Once object 1410 is locked, the wearable computing device presents menu 1440 in the virtual environment. In Figure 14 the example shown, menu 1440 is positioned close to (e.g., adjacent to) object 1410 such that the user can see both object 1410 and menu 1440 when manipulating object 1410. This menu 1440 is in a different layer in the virtual environment than object 1410 (as Figure 14 shown, the menu is in the top layer and partially encompasses object 1410). However, both elements 1410 and 1440 can alternatively be in the same layer.

[0164] In the example, menu 1440 itself is to some extent a virtual object and has a set of parameters that control the look and feel of menu 1440 and the manipulation operations supported by menu 1440. The wearable computing device sets these parameters based on the touch area on the external controller with which the user interacts (rather than on the external controller itself). For example, if the physical touch area is a circular touch-sensitive area that enables a rotation operation, menu 1440 is set to be a circular menu that also enables a rotation operation. If the physical touch area is a tactile button or a pushbutton switch, menu 1440 is set to be a virtual power switch.

[0165] In some embodiments, the look and feel of menu 1440 can match the physical touch area. For example, when the physical touch area is a button or switch-shaped controller, a circular menu 1440 is presented. Nevertheless, menu 1440 can still be used to provide a rotation operation, but this operation will require an operation in the physical environment different from just clicking a button / switch. For example, the user can move the external controller in a circular manner in the physical environment. The wearable computing device can receive information about this movement from the external controller and use this information to reflect the control exerted on object 1410 via menu 1440.

[0166] Once menu 1440 is presented, the user can operate the corresponding touch area (or, as explained in the previous paragraph, in the case of a mismatch, the entire controller) to manipulate object 1410 according to the parameters of object 1410 and the functions supported by menu 1440 (or similarly, the physical touch area / controller). For example, the user can edit the color, shape, size, position, animation, etc. of object 1410 in the virtual environment. In this way, the wearable computing device receives relevant control information from the external controller, updates the presentation of menu 1440 to indicate that the corresponding controls are being applied, and updates object 1410 and its display to show the changes to object 1410.

[0167] Figure 15 An example process for manipulating an object in a virtual environment based on gaze information and physical controls on an external controller (e.g., a physical controller) in accordance with an embodiment of the present disclosure is shown. The example process begins at operation 1502, in which the wearable computing device displays a virtual environment that includes an object. At operation 1504, the wearable computing device displays an indication of the user's gaze position in the virtual environment. For example, the wearable computing device tracks the user's gaze and maps the user's gaze to the virtual environment, and displays a geometry at the gaze position as mapped in the virtual environment.

[0168] At operation 1506, the wearable computing device locks the virtual object. For example, the wearable computing device detects that the gaze position is on the object for a predefined period of time. In another example, the wearable computing device detects that the gaze position is within a predefined gaze zone around the object and, in parallel, receives information from the external controller that the user is interacting with the physical controller. In both examples, the wearable computing device selects the object and updates its parameters to indicate that the object has been selected and is subject to manipulation.

[0169] At operation 1508, the wearable computing device displays a menu associated with manipulating the object. In the example, the wearable computing device presents a menu in the virtual environment near the object. This menu can correspond to a physical touch area on the physical controller that the user holds or interacts with. For example, the wearable computing device receives an identifier of the touch area from the external controller and accesses a virtual model of the external controller and the physical touch area from memory. The virtual model can define the shape, size, look and feel, and functions of the corresponding virtual touch area. The wearable computing device presents this virtual touch area as a menu.

[0170] At operation 1510, the wearable computing device manipulates an object. In an example, the wearable computing device receives control information from an external controller based on a user's interaction with a physical touch area. Since the object is locked, the wearable computing device presents a virtual representation of the user interaction on a menu, manipulates the object based on the control information, and presents the change of the object as being manipulated in a virtual environment.

[0171] Figure 16 An example of manipulating multiple objects in a virtual environment according to an embodiment of the present disclosure is shown. Here, the following two objects are associated with each other: a three-dimensional (3D) printer 1610 and a printed object 1630 (e.g., a cartridge). The 3D printer 1610 can be manipulated in the virtual environment to print the printed object 1630. In other words, the creation of the second object 1630 depends on the manipulation of the first object 1610.

[0172] As Figure 16 shown in the upper left corner of, the wearable computing device detects a user's gaze on the object 1610 (virtual 3D printer). Based on the parameters of this object, a guidance 1620 for using this object is presented. The guidance 1620 presents information in the virtual environment regarding operating an external controller to interact with the object 1610.

[0173] In an example, the wearable computing device sets the guidance 1620 to at least replicate the shape of a virtual controller (and thus the shape of the external controller). The operation information can be shown relative to the replicated shape. As Figure 16 shown in, the guidance 1620 notifies the user to "press" the blue touch area.

[0174] Once the user presses the corresponding touch area on the external controller, the wearable computing device receives an indication of this press. Due to the association between the virtual 3D printer and the virtual printed object, the wearable computing device generates and presents the object 1630 (e.g., a virtual printed object) in the object 1610 (e.g., in the virtual 3D printer), as Figure 16 shown in the upper right corner of.

[0175] At this time, the user can gaze at the object 1630 to lock the object. Alternatively, since the user has pressed the blue touch area to print the printed object in the virtual environment, the wearable computing device automatically locks the object (without a specific gaze) based on the generation of the object 1630. Once locked, the object 1630 can be manipulated.

[0176] Accordingly, the wearable computing device presents a menu for manipulating an object 1630. In an example, the wearable computing device receives an indication of the user's interaction with a physical touch area of an external controller from the external controller. In response, the wearable computing device invokes and presents the menu. The presentation may be progressive such that the menu transitions from an initial state to a stable state (e.g., the menu transitions forward towards the user in a virtual environment).

[0177] Figure 16 The lower left corner of [Figure] shows a menu in transition from an initial state (shown as the transitioning menu 1640). During the transition, the menu is displayed at a first tilt angle. The wearable computing device updates the presentation of the menu such that it gradually moves towards the user's forward field of view. During the movement of the menu, the tilt angle is adjusted to a second tilt angle until it is finally stably placed in front of the user in the virtual environment. Figure 16 The lower right corner of [Figure] shows the menu in a stable state at the second tilt angle (shown as the stable menu 1650).

[0178] Once fully displayed, the menu includes a plurality of components that may be arranged on one or more layers of the virtual environment. Some of these components may be objects that can be locked and manipulated. For example, based on detecting a gaze position on a "width" component, the wearable computing device locks this component for manipulation. The manipulation may occur via the user's interaction with the touch area of the external controller.

[0179] Figure 17 [Figure] shows an example process for manipulating multiple objects in a virtual environment based on gaze information and physical controls on an external controller (e.g., a physical controller) according to an embodiment of the present disclosure. The example process begins at operation 1702, where the wearable computing device detects a gaze position on a first object (e.g., a virtual 3D printer) in the virtual environment. At operation 1704, the wearable computing device displays a guide associated with using the first object. In an example, the guide is displayed based on the parameters of the first object and its association with a second object (e.g., a virtual printed object). The guide shows information about using the external controller to manipulate the first object. For example, the guide notifies the user to press a physical control (e.g., interact with a physical touch area) to generate the second object.

[0180] At operation 1706, the wearable computing device displays a second object in the virtual environment based on the interaction information. In an example, the interaction information includes control information that is provided by the external controller based on the user pressing the physical control in response to the guide information. The wearable computing device receives the interaction and thus generates and displays the second object.

[0181] At operation 1708, the wearable computing device locks a second object. In an example, the second object is automatically locked (e.g., without additional user input or user gaze) based on generating the second object in response to receiving interaction information.

[0182] At operation 1710, the wearable computing device displays a menu associated with manipulating the second object. In an example, the menu corresponds to a physical touch area of an external controller and facilitates manipulation of the second object based on user interaction with the physical touch area. Some or all components of the menu are also objects that can be locked based on detection of a gaze thereon, and once locked, the object can be manipulated based on user interaction with the physical touch area on the external controller.

[0183] Figure 18 Another example of manipulating multiple objects in a virtual environment in accordance with an embodiment of the present disclosure is shown. Here, the following two objects are associated with each other: a card 1810 and a slot 1850 of a table. The card 1810 is a first object that has parameters specifying that it can move in the virtual environment and be placed in the slot 1850. The slot is a second object that has parameters specifying that it is static in the virtual environment and that it can be a destination object for other objects (e.g., can receive the card 1810). In other words, manipulation of the first object 1810 can involve manipulation of the second object 1850.

[0184] As Figure 18 shown in the upper left corner of, the wearable computing device detects a user's gaze 1820 on an object 1810 (the card) for a predefined period of time. Based on the parameters of this object, the object 1810 is selected. The wearable computing device shows the selection 1830 by highlighting the object 1810.

[0185] Next, the wearable computing device receives information from the external controller about a user's interaction with a touch area of the external controller (e.g., a press of a button). The wearable computing device determines that this information corresponds to holding the object 1810 and updates the rendering of the object 1810 accordingly to indicate that the object is being held in the virtual environment.

[0186] Once held, the wearable computing device detects that the user's gaze has moved away from the object 1810 (e.g., the card) to the destination object 1850 (e.g., the slot on the table for receiving the card). Figure 18 The rightmost corner of shows an updated gaze 1840 when the element 1810 is being positioned on the destination object 1850.

[0187] After the user gazes at the destination object 1850, the wearable computing device may allow the user to move their gaze away and operate an external controller to move object 1810 to the destination object 1850. Upon receiving information from the external controller about the user's interaction with the touch area of the external controller (e.g., another button press), a determination is made that object 1810 should be placed in the destination object 1850, and thus the virtual environment is updated to show object 1810 falling within the target object 1850, as shown by element 1860 in the lower central portion of Figure 18 shown.

[0188] Figure 19 Another example of a process for manipulating multiple objects in a virtual environment in accordance with an embodiment of the present disclosure is presented. Here, two objects are associated with each other such that the first object can be moved near the second object (e.g., placed within, on, below, etc. the second object) in the virtual environment.

[0189] The example process begins at operation 1902, in which the wearable computing device detects a position gaze on a first object in the virtual environment for a predefined period of time. As a result, the wearable computing device locks the first object.

[0190] At operation 1904, the wearable computing device detects a user interaction for holding the first object in the virtual environment. For example, the computing device receives information about the user interaction from the external controller, where the user interaction is with the touch area of the external controller and indicates the holding. As a result, the wearable computing device updates the parameters of the first object and its rendering in the virtual environment to reflect the holding.

[0191] At operation 1906, the wearable computing device detects a position gaze on a second object in the virtual environment for a predefined period of time. As a result, the wearable computing device locks the second object.

[0192] At operation 1908, the wearable computing device detects a user interaction for placing the first object on the second object. For example, the computing device receives information about the user interaction from the external controller, where this interaction is with the touch area of the controller and indicates the placing.

[0193] At operation 1910, the wearable computing device places the first object on the second object based on the detected user. For example, the wearable computing device updates the parameters of the first object and its rendering in the virtual environment to reflect the placing.

[0194] Figure 20Shows an example of replacing a controller in a virtual environment according to an embodiment of the present disclosure. Here, the following two objects are associated with each other: the current controller 2010 and the available controller 2020. The use of these two objects in the virtual environment is mutually exclusive. In other words, the user is allowed to use only one of the controllers in the virtual environment at any given time.

[0195] In the example, the user can use an external controller, and the wearable computing device can present the corresponding controller in the virtual environment. The user can replace the external controller with another external controller. The wearable computing device will accordingly replace the controller in the virtual environment. Alternatively, the user can use a general external controller, which can be mapped to different virtual models stored in the local memory or accessed by the wearable computing device via a remote memory. The wearable computing device presents these different virtual models as available controllers in the virtual environment and allows the user to switch between them.

[0196] As Figure 20 shown in the upper left corner of [], the wearable computing device displays the current controller 2010 that the user is operating in the virtual environment and the other available controller(s) 2020 for selection. As Figure 20 shown in the upper right corner of [], to switch from the current controller 2010 to the available controller 2020, the user operates the external controller such that the current controller 2010 moves towards the position of the available controller 2020 in the virtual environment. For example, the wearable computing device receives information indicating the user's interaction with the touch area of the external controller or the external controller itself from the external controller, where the user interaction is used to move the current controller 2010 towards the available controller 2020. Based on this information, the wearable computing device updates the parameters and the presentation of the current controller 2010 to show this movement in the virtual environment.

[0197] To switch to the available controller 2020, the user may need to punch or tap on the available controller. And the user operates the external controller to perform this operation in the virtual environment, and the wearable computing device receives the relevant information from the external controller and updates the virtual environment. As Figure 20 shown in the upper right corner of [], the wearable computing device highlights the virtual punch 2030 by changing the visual characteristics of the available controller 2020 (e.g., by setting its color to bright or flashing).

[0198] Once the available controller 2020 is selected (shown as Figure 20In the selected controller 2040 in the lower central part of [], the selected controller 2040 is used to replace the current controller 2010. In the example, the current controller 2010 is an object presented at the top layer of the virtual environment. The available controller 2020 is another object presented in the lower layer of the virtual environment before the selection. Based on the selection, the wearable computing device eliminates (e.g., removes from the presentation) the object representing the current controller 2010 from the top layer and adds the object representing the available controller 2020 to this top layer. Alternatively, instead of removing and adding, the wearable computing device changes the parameters of the object in the top layer from the parameters of the current controller 2010 to the parameters of the available controller 2020.

[0199] Figure 21 Shows an example of a process for replacing a controller in a virtual environment according to an embodiment of the present disclosure. The example process starts at operation 2102, in which the wearable computing device detects the movement of a first controller towards a second controller in the virtual environment. For example, the computing device receives information from an external controller regarding the user's interaction with a touch area of the external controller or the external controller itself, where the user interaction indicates the movement. Accordingly, the wearable computing device updates the parameters of the first controller and its presentation in the virtual environment to reflect the movement.

[0200] At operation 2104, the wearable computing device detects the contact (e.g., virtual contact) of the first controller with the second controller in the virtual environment. This contact corresponds to a virtual tap. For example, the computing device receives information from an external controller regarding the user's interaction with a touch area of the external controller, where the user interaction indicates the virtual contact.

[0201] At operation 2106, the wearable computing device selects the second controller based on the virtual contact. At operation 2108, the wearable computing device replaces the first controller with the second controller in the virtual environment.

[0202] Figure 22 Shows a schematic example of a wearable device 2200, which demonstrates an embodiment of the present invention. In some embodiments, the wearable device 2200 can be an augmented reality or virtual reality head-mounted device. In this embodiment, the wearable device 2200 is shown as a type of virtual reality head-mounted device.

[0203] The wearable device 2200 may include an eye tracking device 2210, and the eye tracking device 2210 may include at least one image sensor 2213 and at least one illuminator 2216 (which are respectively referred to as the image sensor 2213 and the illuminator 2216 herein). The eye tracking device 2210 may further include one or more processors 2219, or a connection to one or more processors (collectively referred to as processors 2219 herein) that are remote from the eye tracking device 2210 and / or the device 2200 but perform the same and / or similar functions.

[0204] The illuminator 2216 may emit light of one or more wavelengths, and the image sensor 2213 may be configured or selected to respond to the corresponding wavelengths. In some embodiments, the light provided by the illuminator 2216 may be in the infrared wavelength, and thus the image sensor 2213 may respond to infrared wavelength light.

[0205] The eye tracking device 2210 may be configured to selectively activate the illuminator 2216 to illuminate at least one eye 2220 of the user of the wearable device 2200. The processor 2219 may analyze the image of the illuminated eye 2220 and determine the gaze direction of the eye 2220 based thereon.

[0206] The wearable device 2200 may further include at least one holographic film 2230 disposed at a certain position on the wearable device 2200. In the illustrated example, the holographic film 2230 is shown at three possible positions on the wearable device 2200. In the first possible position, the holographic film 2230a is shown on the lens 2240 of the wearable device 2200. In the second possible position, the holographic film 2230b is shown on the display 2250 of the wearable device 2200. In the third possible position, the holographic film 2230c is shown on the freestanding (lensless) portion 2260 of the wearable device 2200. In practice, there may be only one holographic film 2230. In some embodiments, there may be multiple different holographic films 2230. In various embodiments, the image sensor 2213 may be positioned at an angle (θ) with respect to the normal of the holographic film 2230 at which the gaze ray is incident on the holographic film 2230. In some embodiments, θ may be between about 5 degrees and about 75 degrees.

[0207] In addition, it should be noted that the orientation and placement of the eye tracking device 2210 (and thus the image sensor 2213 and / or the illuminator 2216) and the holographic film 2230 can be modified based on this example in a given embodiment. For example, the illuminator 2216 can be positioned relatively farther away from the image sensor 2213 and more directly addressed to the eye 2220, where the light reflected from the eye is again reflected by the holographic film 2230 towards the image sensor 2213. Additionally, in a device different from the virtual reality headset, the holographic film 2230 can be arranged on other parts of such a device (i.e., the light guide of the augmented reality headset).

[0208] The holographic film 2230 can be any kind of film that allows holographic patterns and images to be presented thereon. Example materials for the holographic film 2230 include polyester, polypropylene, nylon, and / or other polymers. In some embodiments, surface reflection holograms are employed, where patterns and images can possibly be embossed on the surface reflection holograms by creating minute grooves cut in the film at various angles and different shapes. This allows the presentation of patterns and images that can have a three-dimensional effect. In other embodiments, volume reflection holograms can be employed, where the holographic film is exposed by light that selectively changes the refractive index of the film to create the hologram.

[0209] The reflection of the patterns and images can be specular reflection or diffuse reflection. Specular reflection travels from the holographic film 2230 in a single direction, while diffuse reflection travels from the holographic film 2230 in multiple / many directions. A holographic film 2230 having one or both types of reflectivity can be employed, as will be discussed below.

[0210] When a holographic film 2230 having specular reflection is employed, the holographic film 2230 must be set on a part of the device 2200 where the reflection angle with respect to the image sensor 2213 and the illuminator 2216 is such that the image sensor 2213 will indeed be able to detect the reflection from the illuminator 2216 on the holographic film 2230. When a holographic film 2230 having diffuse reflection is employed, a wider range of relative positions of the image sensor 2213, the illuminator 2216, and the holographic film 2230 is possible.

[0211] During activities outside the normal process of using the eye tracking device 2210 to determine the gaze direction of the eye 2220, the processor 2219 can also be configured to activate the illuminator 2216 to illuminate the holographic film 2230, and use the image sensor 2213 to capture an image of at least a portion of the holographic film 2230 when the holographic film 2230 is illuminated. The processor 2219 can then determine the characteristics of the holographic film 2230 based on the image. For example, the processor 2219 can determine the sizes and relative positions of different reflective portions of the holographic film 2230.

[0212] Based on the determined characteristics of the holographic film 2230, the processor 2219 can then determine the position, orientation, and / or other characteristics of the image sensor 2213 relative to the holographic film 2230 (or a key sub-component of the image sensor 2213; e.g., a lens, a charge-coupled device (CCD), an active pixel sensor, etc.). The processor 2219 can then be configured to change at least one calibration parameter of the image sensor 2213 based on at least one of the position, orientation, or other characteristics.

[0213] Alternatively, the processor 2219 can send data regarding the position, orientation, and / or other characteristics to the image sensor 2213 for the image sensor 2213 to use to change at least one of its own calibration parameters. In a further alternative, the processor 2219 can use the data regarding the position, orientation, and / or other characteristics to calibrate the algorithms used by the processor 2219 in order to determine the gaze direction using the image sensor 2213 and the illuminator 2216. The adjusted calibration parameters can include, by way of example only: imaging parameters, focal lengths, distortion parameters, illumination parameters, the orientation of the image sensor relative to the hologram, the field of view, the distance to the hologram, and / or the angle of the image sensor relative to the holographic film 2230.

[0214] The holographic film 2230 can have different reflective characteristics, each of which can be sufficient to implement the described embodiments of the present invention. In order for the image of the reflected light from the holographic film 2230 to be sufficient to determine the position and / or orientation of the image sensor 2213 from it, there must be characteristics along at least two orthogonal axes. In some embodiments, a single feature with two-dimensional characteristics will be provided. In other embodiments, there can be two or more features with combined two-dimensional characteristics.

[0215] Figures 23A to 23FShows various holographic films with different characteristics that can be used to implement embodiments of the present invention. Other variations are possible. The shaded portions of these figures represent their first reflective portions, while the non - shaded portions represent the second reflective portions. However, the reflection angles of the first reflective portion may be different from those of the second reflective portion. Alternatively, the shaded or non - shaded portions of the figure may represent non - reflective regions of the film. That is, the holes or cavities in the holographic film / mirror can be calibrated based on their characteristics (size, position, etc.). In some embodiments, the non - shaded portion can reflect an image of the eye for gaze detection, while the shaded portion can provide a specular reflection towards the image sensor 2213 for calibration as discussed herein. Although more complex and larger markings on the holographic film can provide more information and result in a more accurately determined position / orientation of the image sensor 2213, smaller markings will reduce the likelihood of the image sensor 2213 saturating due to reflected light, which saturation may render the data obtained therefrom less or unavailable.

[0216] Figure 23A Shows a holographic film with a marking that has two - dimensional characteristics. This marking can be presented in a diffusive or specular manner depending on the placement of the illuminator 2216 and the image sensor 2213. Diffusive markings can be made in any way known in the art or that becomes known in the art in the future.

[0217] Figure 23B Shows an alternative marking that also has two - dimensional characteristics. This marking can also be presented in a diffusive or specular manner depending on the placement of the illuminator 2216 and the image sensor 2213.

[0218] Figure 23C Shows another marking, whereby at least three holographic spheres are presented. The reflection of the illumination of the spheres provides three general point positions in a two - dimensional layout. Due to the circular nature of the holographic spheres, the overall reflection from the spheres will be diffused in different directions. Larger spheres will provide more reflected light but lower accuracy, while the smaller spheres will provide less reflected light but with increased accuracy. Other three - dimensional holographic markings can also be employed.

[0219] Figure 23D Shows another possible marking, where both the size of the illuminated region and the implicit size of the non - illuminated region can be used as characteristics that can be utilized by the systems and methods of the present invention to determine the position and / or orientation of the image sensor 2213. This marking can also be presented in a diffusive or specular manner depending on the placement of the illuminator 2216 and the image sensor 2213.

[0220] Figure 23EAnother marker is shown, whereby the overall dimensions of the integral reflective holographic film provide the necessary two-dimensional characteristics. This marker can also be presented in a diffused or specular manner depending on the placement of illuminator 2216 and image sensor 2213.

[0221] Figure 23F Another marker is shown having two one-dimensional (or approximately one-dimensional) markers, whereby the length of the markers and their relative placement with respect to each other provide the necessary two-dimensional characteristics. This marker can also be presented in a diffused or specular manner depending on the placement of illuminator 2216 and image sensor 2213.

[0222] Execute Figure 8 、 11 The programs or instructions of the flowcharts shown in 15, 17, 19, and 21 can be stored in (a) computer-readable storage medium reader 350 and / or in (a) other code 388 of working memory 380, and can be executed by Figure 3 the (a) CPU 310 shown in

[0223] For purposes of clarity and understanding, the present disclosure has been described in detail. However, it will be understood that certain changes and modifications can be practiced within the scope of the appended claims.

[0224] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the above description of the exemplary embodiments will provide those skilled in the art with a description for enabling the implementation of one or more exemplary embodiments. It should be understood that various changes can be made to the functions and arrangements of the elements without departing from the spirit and scope of the present disclosure as set forth herein.

[0225] For example, any details discussed with respect to one embodiment may or may not be present in all contemplated versions of that embodiment. Similarly, any details discussed with respect to one embodiment may or may not be present in all contemplated versions of other embodiments discussed herein. Finally, any details not discussed with respect to the embodiments herein should be assumed such details may or may not be present in any version of any embodiment discussed herein.

[0226] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments can be practiced without these specific details. For example, the circuits, systems, networks, processes, and other elements in the present disclosure can be shown in block diagram form as components so as not to obscure the embodiments with unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details to avoid obscuring the embodiments.

[0227] Similarly, it should be noted that each embodiment may be described as a process, which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of the operations may be rearranged. The process may terminate when its operations are completed, but may have additional steps that are not discussed or not included in the figure. Furthermore, not all operations in any particular described process may occur in all embodiments. The process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When the process corresponds to a function, its termination corresponds to the function returning to the calling function or the main function.

[0228] The term "machine-readable medium" includes, but is not limited to, transient or non-transient, portable or fixed storage devices, optical storage devices, wireless channels, and various other media capable of storing, containing, or carrying instructions and / or data. A code segment or machine-executable instruction may represent a process, a function, a subroutine, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0229] Furthermore, embodiments of the present disclosure may be implemented, at least in part, manually or automatically. The manual or automatic implementation may be performed or at least assisted by using a machine, hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a machine-readable medium. One or more processors may perform the necessary tasks.

[0230] As used herein, phrases such as "the first thing based on the second thing" may mean that the first thing is based solely on the second thing; or the first thing is based on the second thing and one or more additional events.

Claims

1. A wearable device (2200), characterized in that, the wearable device (2200) includes: an eye tracking device (2210), the eye tracking device including at least one image sensor (2213), at least one illuminator (2216) and at least one processor (2219); at least one lens, the at least one lens including a first lens (2240); and at least one holographic film, the at least one holographic film including a first holographic film (2230a) disposed on the first lens (2240); wherein, the at least one image sensor (2213) is positioned at an angle with respect to the normal of the gaze ray incident on the at least one holographic film; wherein, the first holographic film (2230a) contains a marker, wherein, the at least one processor (2219) is configured to perform the following steps: activate the at least one illuminator (2216) to illuminate the at least one holographic film; when the at least one holographic film is illuminated, use the at least one image sensor (2213) to capture an image of at least one part of the at least one holographic film; determine the characteristics of the at least one holographic film based on the image; determine at least one of the position or orientation of the at least one image sensor (2213) relative to the at least one holographic film based on the determined characteristics; and change at least one calibration parameter of the at least one image sensor (2213) based on at least one of the position or the orientation of the at least one image sensor (2213) relative to the at least one holographic film.

2. The wearable device (2200) according to claim 1, further including: a display (2250); the at least one holographic film includes a second holographic film (2230b); the second holographic film (2230b) is disposed on the display (2250); the second holographic film (2230b) contains a marker.

3. The wearable device (2200) according to claim 2, further including: a lensless portion (2260); the at least one holographic film includes a third holographic film (2230c); the third holographic film (2230c) is disposed on the lensless portion (2260); the third holographic film (2230c) contains a marker.

4. The wearable device (2200) according to claim 3, wherein: each of the first holographic film (2230a), the second holographic film (2230b) or the third holographic film (2230c) contains a marker, and the markers are in a cross shape in a two-dimensional layout.

5. The wearable device (2200) according to claim 3, wherein: each of the first holographic film (2230a), the second holographic film (2230b) or the third holographic film (2230c) contains a marker having a square portion in a two-dimensional layout.

6. The wearable device (2200) according to claim 3, wherein: Each of the first holographic film (2230a), the second holographic film (2230b), or the third holographic film (2230c) contains markings, and the reflection of illuminating the markings provides three point positions in a two-dimensional layout.

7. The wearable device (2200) according to claim 3, wherein: Each of the first holographic film (2230a), the second holographic film (2230b), or the third holographic film (2230c) contains markings, and both the size of the illuminated area and the implicit size of the non-illuminated area can be used to determine the position and orientation of the at least one image sensor (2213).

8. The wearable device (2200) according to claim 3, wherein: Each of the first holographic film (2230a), the second holographic film (2230b), or the third holographic film (2230c) contains markings, and the overall size of the overall reflective film provides two-dimensional characteristics.

9. The wearable device (2200) according to claim 3, wherein: Each of the first holographic film (2230a), the second holographic film (2230b), or the third holographic film (2230c) contains two one-dimensional markings, and the length of the two markings and their relative placement with respect to each other provide two-dimensional characteristics.

10. The wearable device (2200) according to any one of claims 1 to 9, wherein: Each of the first holographic film (2230a), the second holographic film (2230b), or the third holographic film (2230c) contains markings presented in a diffused or specular reflection manner.

Images