Dynamic Camera Rotation Calibration

Abstract

The present invention relates to a method and apparatus for calibrating a camera of an eye tracking device and compensating for a potential angular offset of the camera. The method includes: a step of capturing an eye image of a user, wherein the eye image contains a plurality of flashes formed by a plurality of light emitters in an eye tracking system; detecting the flashes in the eye image; projecting the light emitter positions onto the eye image to determine expected flash positions; determining an angular offset between the expected flash positions and the detected flash positions for corresponding pairs of expected flash positions and detected flash positions, determining an angle correction for the eye tracking camera using the determined angular offset angle; and applying the angle correction for the eye tracking camera to an eye tracking camera model.

Description

Technical Field

[0001] The present invention relates to a method for calibrating a camera for eye tracking capabilities Background Art

[0002] In devices that support eye tracking, such as in an extended reality head-mounted display (XR HMD), algorithms are used to create a mathematical camera model to establish a connection between the image space (2D) and the world space (3D). So far, for eye tracking integration in XRHMDs, etc., the process of calculating the camera model has been used for hardware calibration devices. An example of such a calibration device can be found in U.S. Patent No. 10,928,891 (Rosell), which is incorporated herein by reference. The calibration process is expensive as part of HMD production, and it also increases production time and may affect the yield ratio.

[0003] Eye tracking integration with HDMs has shown that the quality of the camera model has a significant impact on the performance of the accuracy of the eye tracking signal. Therefore, historically, in order to obtain sufficient eye tracking quality, each unit of hardware needs to be calibrated.

[0004] Therefore, for some eye tracking integrations, a default camera model needs to be used without calibrating each unit during its production. The main benefit of doing this is that it can reduce the cost and complexity of integrating and running the hardware calibration at the production site. However, the problem for each produced HDM is that installation tolerances will result in individual installation errors. Therefore, by using the default calibration, the default camera model for all HMDs will not fit each individual HMD perfectly.

[0005] Through simulation and experimentation, it has been found that the parameter that most affects the accuracy of the eye tracking signal is the rotation of the camera around its own axis. Therefore, the present invention attempts to alleviate the problem of using the default camera model by adding an algorithm to the eye tracking system that will use the data collected during the personal user calibration, where the system can dynamically calibrate the camera rotation outside of each user and production site. This process does not add additional user interaction steps beyond those already required for the eye model user calibration. Summary of the Invention

[0006] An embodiment disclosed by the present invention describes a method for calibrating a camera of an eye tracking system. The method includes the steps of capturing an eye image of a user, where the eye image includes multiple flashes formed by multiple light emitters in the above-mentioned eye tracking system; detecting the flashes in the eye image; projecting the light emitter positions onto the eye image to form expected flash positions; for corresponding pairs of expected flash positions and captured flash positions, determining the angular offset between the expected flash position and the captured flash position; using the determined angular offset angle to determine the angular correction of the eye tracking camera; and applying the angular correction of the eye tracking camera to the eye tracking camera model.

[0007] In another embodiment, the eye tracking system is included within a head-mounted display device. Additionally, the calibration steps can be repeated for each camera of the system to calibrate all the cameras of the system. In another embodiment, the multiple light emitters are disposed around the lens of the HMD. In a further embodiment, the detected flashes are used to determine the corneal model of the user. Additionally, the light emitter positions are projected onto the corneal model of the user to form expected flash positions.

[0008] Another alternative embodiment includes the step of determining the center point of the expected flash position, where the center point is used to determine the angular offset of each flash relative to its corresponding expected flash position. Additionally, there is an embodiment that includes the step of calculating the median value of the angular offset angles of the detected flashes, and using the median value of the angular offset angles to determine the angular correction of the eye tracking camera.

[0009] In another alternative embodiment, the disclosed invention may include an apparatus for calibrating a camera of an eye tracking system. The apparatus includes the eye tracking system, which has at least one eye tracking camera and multiple light emitters; and a calibration processor, which is arranged to capture an eye image of the user, where the eye image includes multiple flashes formed by multiple light emitters in the above-mentioned eye tracking system, detect the flashes in the eye image, project the light emitter positions onto the eye image to form expected flash positions, for corresponding pairs of expected flash positions and captured flash positions, determine the angular offset between the expected flash position and the captured flash position, use the determined angular offset angle to determine the angular correction of the eye tracking camera; and apply the angular correction for the eye tracking camera to the eye tracking camera model.

[0010] Another embodiment of the disclosed invention includes a non - transitory computer - readable storage medium storing instructions that, when executed by an eye - tracking system, cause the system to perform the steps of capturing an eye image of a user, where the eye image includes multiple flashes formed by multiple light emitters in the eye - tracking system, detecting the flashes in the eye image, projecting the light - emitter positions onto the eye image to form expected flash positions, determining, for corresponding pairs of expected flash positions and captured flash positions, the angular offset between the expected flash position and the captured flash position, using the determined angular - offset angle to determine an angle correction for the eye - tracking camera; and applying the angle correction for the eye - tracking camera to the eye - tracking camera model.

[0011] In another alternative embodiment, the eye - tracking system is included within a head - mounted display device, such as a VR / AR / XR headset, a peripheral device, such as an eye - tracker connected to a personal computer, or a head - mounted device without a display, such as wearable eye - tracking glasses. Brief Description of the Drawings

[0012] Specific embodiments are described in detail with reference to the accompanying drawings, in which:

[0013] Figure 1 Depicts a user's eye observing through the lens of an HMD with eye - tracking integration;

[0014] Figure 2 Is a depiction of an eye image with captured flashes and expected flash positions superimposed thereon;

[0015] Figure 3 Depicts a flowchart of the method according to the present disclosure.

[0016] Detailed Description

[0017] As discussed previously, in manufacturing eye - tracking camera - integrated devices such as XR HMDs, significant cost savings can be achieved if per - unit hardware calibration is omitted. By using the default camera model, the problem that arises is that the camera models of each individual HMD unit are different. These differences are because the installation tolerances of the cameras, lenses, and other hardware are not ideal and vary from unit to unit. Therefore, the present invention attempts to form a dynamic calibration method that can compensate for production errors and installation tolerances. The described invention will also achieve further advantages in that the method will improve the accuracy of the eye - tracking signal by calculating and compensating for the rotation of each HMD camera, as well as compensating for changes in the hardware and installation over time, such as due to wear and distortion of the HMD.

[0018] Figure 1Depicts an HMD device with integrated eye tracking capabilities. The optical path from the user's eye 102 through the lens 130, the hot mirror 135 to the eye tracking camera 120 is depicted. The initialization process of the HMD for the user will include dynamic camera rotation calibration required using the default camera model. In particular, the dynamic camera rotation calibration will seek to compensate for the rotation of the eye tracking camera 120 about the z-axis along the optical path as shown in Figure 1 shown.

[0019] To seek rotation compensation, the eye tracking system will capture data to establish a corneal model of the user. This is done by capturing eye images when a plurality of light emitters (not shown in Figure 1 ) produce a flash pattern on the user's cornea. Although the method does not require the user's eyes to look straight ahead, which typically occurs in conventional eye tracking calibration, it is understood that it is best to use eye images in which many flashes can be seen. Then, the eye tracking system will use the flash data to establish a user eye movement model. The model can be a three-dimensional model of the user's eye based on the detected flash positions captured in the eye images. Methods for calculating the corneal position and shape are well known in the eye tracking field and do not require further discussion in the present invention.

[0020] The default camera model will include the default positions of a plurality of light emitters used in a specific eye tracking system corresponding to the default camera model. Using the default positions of the light emitters, the system superimposes the positions of the light emitters onto the eye image. The emitter superimposition projection is performed using the corneal model. Although in an optimal embodiment, the eye tracking system will include six or more light emitters for each eye, it is understood that the system of the present invention can generally function with at least two light emitters. The superimposition projection is used by the system to determine the expected flash positions based on the default camera model.

[0021] As discussed in the present invention, projection refers to the process of using the default camera model to calculate where something (a light emitter) at a known position in the world will be visible in the image. The default camera model is used to describe the relationship between the image (2D) and the world (3D). Those skilled in the art can understand that the projection step does not necessarily form a superimposed image as seen in Figure 2 , but can use computational methods, visual superimposition methods, and other known ray tracing methods.

[0022] To determine the compensation required for the camera, the system will compare each observed flash position with its corresponding expected flash position. This comparison provides an angular offset, where the angle can be measured from the center point of the expected flash pattern. To ensure the robustness of the system and reduce noise, outlier angles can be discarded by calculating the median angular difference of multiple offset angles.

[0023] Figure 2 depicts an eye image with a captured flash 206 and an overlaid expected flash position 208. In Figure 2 , the eye image of user 200 has been captured by the eye tracking camera 120, which shows the user's pupil 204 and the flash 206 detected on the user's cornea 202. The detected flash 206 allows the system to determine the position of the user's cornea, based on which the expected flash 208 is overlaid onto the eye image. Using the center of the flash pattern (shown as a cross in the middle of the pupil 204) as a reference point, the flash 206 is compared with the expected flash position 208, and an offset angle 210 is measured for each corresponding pair of detected flash 206 and expected flash position 208.

[0024] Given the median offset angle, the camera angle correction will be determined and applied to the camera model. Thus, the updated camera model will correct any angular distortion from the eye tracking camera, thereby improving the camera rotation calibration. It can be further envisioned that the camera rotation calibration process can be used after the initialization of the eye tracking system and can be repeated after the detection of a possible device damage, after a set time period, or at the user's active request.

[0025] Figure 3 depicts a flowchart of the method steps for calibrating the eye tracking camera for a camera model updated due to rotational distortion 300. When starting and initializing the system, the eye tracking camera captures an eye image containing a flash 310. Then, the system detects the flash in the eye image and determines the cornea position of user 320. Using the cornea position and the default camera model illuminator position, the expected illuminator position of the camera 330 is determined. Then, the system compares the expected illuminator position with the detected flash and measures the offset angle between the corresponding pair 340. Using the offset angle data, the system determines the camera angle offset correction and applies the correction to the camera model 350. After the camera model is updated, the calibration process ends.

Claims

1. A method for calibrating at least one eye tracking camera of an eye tracking system and updating a camera model, the method comprising the steps of: a) Capturing an eye image of a user, wherein the eye image includes a plurality of flashes formed by a plurality of light emitters in the eye tracking system, b) Detecting the flashes in the eye image, c) Projecting the light emitter positions onto the eye image using a corneal model to determine expected flash positions, d) Determining an angular offset between the expected flash positions and the detected flash positions for corresponding pairs of expected flash positions and detected flash positions, e) Using the determined angular offset angle to determine an angle correction for the at least one eye tracking camera, and f) Applying the angle correction for the at least one eye tracking camera to the eye tracking camera model.

2. The method according to claim 1, wherein the eye tracking system is included in a head-mounted display device (HDM).

3. The method according to claim 2, wherein steps a to f are repeated for each eye tracking camera in the system.

4. The method according to claim 2, wherein the plurality of light emitters are disposed around a lens of the head-mounted display device.

5. The method according to claim 1, wherein the detected flashes are used to determine the corneal position of the user.

6. The method according to claim 5, wherein the light emitter positions are projected onto the determined corneal position of the user to determine the expected flash positions.

7. The method according to claim 1, further comprising the step of determining a center point of the expected flash positions, the center point being used to determine the angular offset of each flash relative to its corresponding expected flash position.

8. The method according to claim 1, further comprising the steps of: calculating a median value of the angular offset angles of the detected flashes, and using the median value of the angular offset angles to determine the angle correction for the eye tracking camera.

9. An apparatus for calibrating an eye tracking camera of an eye tracking system, the apparatus comprising: The eye tracking system, the eye tracking system having at least one eye tracking camera and a plurality of light emitters; wherein the eye tracking system is configured to: a) Capture an eye image of a user, wherein the eye image includes a plurality of flashes formed by a plurality of light emitters in the eye tracking system, b) Detect the flashes in the eye image, c) Project the light emitter positions onto the eye image using a corneal model to determine expected flash positions, d) Determine an angular offset between the expected flash positions and the detected flash positions for corresponding pairs of expected flash positions and detected flash positions, e) Use the determined angular offset angle to determine an angle correction for the eye tracking camera, and f) Apply the angle correction for the eye tracking camera to the eye tracking camera model.

10. The apparatus for calibrating an eye tracking camera of an eye tracking system according to claim 9, wherein the eye tracking system is included in a head-mounted display device (HDM).

11. The apparatus for calibrating an eye tracking camera of an eye tracking system according to claim 9, wherein steps a to f are repeated for each eye tracking camera in the system.

12. The apparatus for calibrating an eye tracking camera of an eye tracking system according to claim 10, wherein the plurality of light emitters are disposed around a lens of the head-mounted display device.

13. The apparatus for calibrating an eye tracking camera of an eye tracking system according to claim 9, wherein the detected flash is used to determine the corneal position of the user.

14. The apparatus for calibrating an eye tracking camera of an eye tracking system according to claim 13, wherein the light emitter position is projected onto the determined corneal position of the user to determine the expected flash position.

15. The apparatus for calibrating an eye tracking camera of an eye tracking system according to claim 9, wherein the eye tracking system is further configured to determine a center point of the expected flash position, and the center point is used to determine an angular offset of each flash relative to its corresponding expected flash position.

16. The apparatus for calibrating an eye tracking camera of an eye tracking system according to claim 9, wherein the eye tracking system is further configured to calculate a median value of the angular offset angles of the detected flashes, and use the median value of the angular offset angles to determine the angle correction for the eye tracking camera.

17. A non-transitory computer-readable storage medium storing instructions, which when executed by an eye tracking system, cause the system to perform the steps of: a) capturing an eye image of a user using an eye tracking camera, wherein the eye image includes a plurality of flashes formed by a plurality of light emitters in the eye tracking system, b) detecting the flashes in the eye image, c) projecting the light emitter positions onto the eye image using a corneal model to determine the expected flash positions, d) determining an angular offset between the expected flash position and the detected flash position for corresponding pairs of expected flash positions and detected flash positions, e) determining an angle correction for the eye tracking camera using the determined angular offset angles, and f) applying the angle correction for the eye tracking camera to an eye tracking camera model.

18. The non-transitory computer-readable storage medium storing instructions according to claim 17, wherein the eye tracking system is included in a head-mounted display device, or a peripheral device, or a head-mounted device.

Images