Hybrid eye tracking device and laboratory and field calibration methods for calibrating hybrid eye tracking device
By adopting a hybrid eye tracking device in data glasses, combined with camera sensors and LFI sensors, fast and accurate eye position and speed measurements are achieved, solving the problem of inefficient complex calibration and tracking in the prior art, reducing energy consumption and improving user-friendliness.
Patent Information
- Application Number
- CN202380073933.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-10-20
- Filing Date
- 2023-07-12
- Publication Date
- 2025-05-30
AI Technical Summary
Eye tracking devices in existing data glasses require complex calibration processes during initial assembly and on-site calibration, and it is difficult to achieve fast and precise eye position tracking.
A hybrid eye tracking device is used, which combines camera sensors and laser feedback interferometry (LFI) sensors to achieve fast and accurate measurement of eye position and speed through the fusion of camera images and laser points. The device can be accurately calibrated by laboratory calibration methods and field calibration methods.
Fast and accurate eye tracking is achieved, support for update rates greater than 1kHz, reduces energy consumption, and simplifies calibration processes, improving user-friendliness and device robustness.
Smart Images
Figure CN120077317A_ABST
Abstract
Description
Background Art
[0001] Data glasses with eye tracking devices are known. Generally, these data glasses require initial calibration during assembly and, if necessary, re - calibration on site. Summary of the Invention
[0002] A hybrid eye tracking device, in particular a hybrid high - speed eye tracking device, is proposed, which is preferably in data glasses and is used to determine the instantaneous eye position, in particular the instantaneous eye position of the eyes of the user of the data glasses. The hybrid eye tracking device includes at least one camera sensor and has at least one laser feedback interferometry (LFI) sensor, in particular at least two LFI sensors. The camera sensor is configured to at least determine the eye position from the camera image, and the LFI sensor, in particular a plurality of LFI sensors, is configured to at least determine the instantaneous speed, in particular the instantaneous speed of eye movement. Advantageously, this enables particularly fast and very precise eye tracking. In particular, eye tracking with an update rate greater than 1 kHz, preferably greater than 10 kHz and preferably greater than 100 kHz can be achieved. By fusing the position of the eye from the camera images (frames) of the camera sensor, the absolute position of the eye can be determined in particular by means of the hybrid eye tracking device. Advantageously, between the capture of the frames of the camera sensor, the intermediate position of the eye can be integrated by the speed measurement of the LFI sensor. In particular, the data glasses include a computer unit, which is configured to, in particular by means of the integration, determine the instantaneous eye position occupied between successive camera images (frames) of the camera sensor from the instantaneous speed measured by the LFI sensor. A "high - speed eye tracking device" should in particular be understood as an eye tracking device having an eye position update rate greater than 1 kHz, preferably greater than 10 kHz and preferably greater than 100 kHz.
[0003] "Data glasses" should in particular be understood as wearable devices (head-mounted displays) by means of which information can be added to the user's field of vision. Preferably, the data glasses are capable of implementing augmented reality applications and / or mixed reality applications. As is well known, data glasses are also referred to as smart glasses. In particular, the data glasses have a virtual retinal display, in particular a virtual retinal display familiar to those skilled in the art (also referred to as a retinal scanning display or a light field display). In particular, the data glasses, preferably the virtual retinal display, include a laser projector. In particular, the LFI sensor can be constructed in a way integrated into the laser projector or arranged separately from the laser projector in the data glasses. The data glasses can include at least one or more additional LFI sensors. One or more LFI sensors and camera sensors can be arranged in a common plane, in particular in the lens plane of the data glasses. Alternatively, one or more LFI sensors and camera sensors can be arranged in different planes. Preferably, one or more LFI sensors and camera sensors are arranged and / or oriented relative to each other such that the LFI sensors and camera sensors are oriented such that the LFI sensors and camera sensors can detect data (images or speed or / and distance) of the eyes. The LFI sensor is in particular a sensor that is based on the self-mixing or back-injection laser interferometry effect known to those skilled in the art. In particular, in the LFI sensor, a sensor signal is generated by reflecting a laser signal into, in particular by the user's eye into, the laser resonator of the LFI sensor, preferably the laser resonator that has already generated the output laser signal, and thereby modulating the output laser signal, in particular the amplitude and / or frequency of the output laser signal. Advantageously, the cost can be reduced by using the LFI sensor. In particular, the LFI sensor can have a photodiode that can detect the reflected laser signal. In particular, the photodiode is integrated into the laser projector, in particular into the laser (infrared laser), preferably into the laser resonator of the laser (infrared laser). For example, the laser (infrared laser) can include a ViP-VCSEL (vertical cavity surface emitting laser with integrated photodiode) or can be constructed as a ViP-VCSEL. Preferably, the LFI sensor is set to detect the instantaneous speed of the eye movement of the user's eyes. Preferably, the LFI sensor is set to detect the speed change of the eye movement of the user's eyes transverse to the emission direction of the output laser signal. Therefore, in order to determine the instantaneous speed of the eye movement of the user's eyes, it may be necessary to track the angle of incidence between the eye surface of the user's eyes and the axis of rotation of the user's eyes. For this purpose, in particular, the exact position of the laser of the LFI sensor in the optical system, in particular in the data glasses, preferably in the frame of the data glasses, needs to be known. In particular, the laser of the LFI sensor is constructed as an infrared laser."Set" and / or "established" should in particular be understood as specifically programmed, designed and / or equipped. An object being set for a determined function should in particular be understood as meaning that the object satisfies and / or implements the determined function in at least one application state and / or operating state.
[0004] Furthermore, it is proposed that the LFI sensor has a significantly lower resolution and a significantly higher sampling rate relative to the camera sensor. Advantageously, as a result, the energy consumption can also be kept low in the case of a particularly high update rate. In particular, the LFI sensor is configured as a high-speed LFI sensor. In particular, the camera sensor is configured as a high-precision camera sensor, preferably as a high-precision infrared camera sensor, which in particular has an update rate of 0.1 Hz to a maximum of 100 Hz. Since in particular the camera sensor has a relatively high energy consumption and the LFI sensor has a relatively low energy consumption or current consumption, the proposed combination of the two sensors can advantageously optimize the energy consumption while having a high update rate.
[0005] Furthermore, a laboratory calibration method for a hybrid eye tracking device and / or a hybrid eye tracking method is proposed, wherein the position of the LFI sensor in the coordinate system of the camera sensor is determined by determining the position of the laser spot generated by the LFI sensor on the laboratory calibration target in the coordinate system of the camera sensor at different predefined known distances between the laboratory calibration target and the camera sensor with the aid of the camera sensor. The laboratory calibration target is preferably flat and preferably monochromatic. Advantageously, this enables a fast and simple possibility for an accurate end-of-line calibration of the hybrid eye tracking device. Advantageously, this can reduce the tolerancing effort during production, especially since the hybrid eye tracking device, preferably data glasses, can be accurately calibrated at the end of production. In particular, the laboratory calibration target can here be a simple monochromatic (e.g., black) surface or can include a checkerboard pattern (e.g., ChArUco pattern). Preferably, the laboratory calibration method is carried out at the end of the production line for a hybrid eye tracking device, especially for a virtual retinal display, preferably for data glasses. In particular, the laboratory calibration method is configured as an end-of-line calibration method for data glasses. In particular, to carry out the laboratory calibration method, first the hybrid eye tracking device, especially the virtual retinal display, preferably data glasses, are positioned in front of the laboratory calibration target. In particular, the laboratory calibration target is photographed by the camera sensor while the LFI sensor radiates onto the laboratory calibration target. In particular, the camera sensor here detects the irradiation point of the output laser signal on the laboratory calibration target. In particular, when the laboratory calibration target is moved by means of a precision linear axis, the irradiation point of the output laser signal moves. It is conceivable that multiple LFI sensors radiate onto the laboratory calibration target simultaneously and are calibrated simultaneously or sequentially in turn with respect to the camera sensor by means of the laboratory calibration method. In particular, all LFI sensors of the hybrid eye tracking device of the data glasses are calibrated simultaneously or sequentially in turn by means of the laboratory calibration method. In particular, the data glasses can include at least two LFI sensors, wherein preferably each LFI sensor is assigned to the other of the two user eyes of the user of the data glasses.
[0006] If, in a laboratory calibration method, a laboratory calibration target is moved on a precision linear axis during calibration such that the corresponding relative spacing between the laboratory calibration target and the camera sensor can be directly read out or determined, then advantageously, fast, simple, and precise end calibration of a hybrid eye tracking device can be achieved. In particular, the movement path of the precision linear axis can be accurately read out and / or determined. In particular, the reading out can be automated, for example, by means of a computing unit. Alternatively, a manual reading can also be provided. In particular, in a laboratory calibration method, if the axis of movement of the precision linear axis and the axis of the camera of the camera sensor are not to be located on the same axis, then only the change in the relative spacing between the laboratory calibration target and the camera sensor is known. If the axes coincide, then even the absolute change in spacing can be determined, but this is not compulsorily required for the execution of the laser calibration method.
[0007] Furthermore, if, in a laboratory calibration method, in at least one laboratory calibration step, the distance between the LFI sensor and the laboratory calibration target at the location of the laser spot, in particular at the irradiation point of the output laser signal on the laboratory calibration target, is determined by means of an LFI sensor, then advantageously, fast, simple, and precise end calibration of a hybrid eye tracking device can be achieved. In particular, for this purpose, the LFI sensor is operated by means of triangular modulation, for example, in a manner similar to an FMCW radar (continuous wave radar), whereby the LFI sensor can measure the spacing between the laser exit facet of the LFI sensor and the irradiation point of the output laser signal on the laboratory calibration target.
[0008] Furthermore, if in the laboratory calibration method, in at least one laboratory calibration step, a unit vector of the beam direction of the LFI sensor is determined from the horizontal and / or vertical pixel pitch between images of the same laser point at different distances between the laboratory calibration target and the camera sensor, then a fast, simple and accurate end calibration of the hybrid eye tracking device can be advantageously achieved. In particular, the camera is configured as a calibrated camera. In particular, the position of the laser point has been moved in the vertical direction (Δy) and in the horizontal direction (ΔX) along a precision linear axis (Δz) by this method. Advantageously, Δz is already known from the movement path of the precision linear axis, while Δy and Δx are measured in pixels from the camera image (relative to the center of the laser point on the laboratory calibration target). Then, by using the calibrated camera, the distance in pixels can be advantageously converted into an absolute distance and thus the unit vector in the direction of the laser beam outputting the laser signal can be determined. Then, in at least one additional calibration step, the position of the LFI sensor relative to the coordinate system of the camera sensor is determined from an overview of the determined distance between the LFI sensor and the laboratory calibration target and the determined unit vector of the beam direction of the LFI sensor. In particular, here, the measured distance from the LFI sensor to the laboratory calibration target is used as the absolute vector length in the coordinate system of the camera sensor. In particular, here, the unit vector is used as the vector direction in the coordinate system of the camera sensor.
[0009] Furthermore, a field calibration method for calibrating a hybrid eye tracking device is proposed, wherein the position of the LFI sensor, in particular at least one LFI sensor among a plurality of LFI sensors, in the coordinate system of the camera sensor is determined by obtaining, by means of the camera sensor, the position in the coordinate system of the camera sensor of the laser points generated by the LFI sensor on a defined and known calibration pattern of the field calibration target at different unknown distances between the field calibration target and the camera sensor. The field calibration target is in particular flat, and the field calibration target is preferably a checkerboard pattern target, preferably a ChArUco pattern target. Advantageously, a fast and simple possibility for precise field calibration of a hybrid eye tracking device can thus be achieved. Advantageously, a high level of user-friendliness can be achieved. For example, advantageously, the recalibration that becomes necessary, for example, due to a fall or an adjustment of the data glasses, can be performed by means of the field calibration method. In particular, the calibration pattern of the field calibration target can be printed (by means of a commercially common printer). Alternatively, the calibration pattern of the field calibration target can also be shown on a screen. In particular, the defined and known calibration pattern of the field calibration target is set to at least enable the calibration of the camera, for example, by a shape or pattern having known and prescribed dimensions. In particular, the defined and known calibration pattern of the field calibration target is set to at least enable the determination of the orientation of the calibration target in space relative to the LFI sensor, for example, by regularly and repeatedly arranging defined and identical pattern elements. The ChArUco pattern target is a special checkerboard pattern that is combined with elements from the Augmented Reality Library (ArUco) of the University of Cordoba. First, for this purpose, the calibration (recalibration) of the camera sensor is performed in the field calibration method. For this purpose, the markers of the defined and known calibration pattern are identified in the camera image of the camera sensor, for example, the ArUco elements in the ChArUco pattern. Then, by means of these identified markers, the calibration matrix K of the pinhole camera model and the associated distortion vector b are calculated.
[0010]
[0011] b = [k 1 k 2 p 1 p 2 k 3
[0012] Next, the laser of the LFI sensor is turned on, and the laser points on the field calibration target are detected from the camera image by means of the camera sensor.
[0013] Then, if in at least one in-situ calibration step, the distance between the LFI sensor and the checkerboard pattern target at the location of the laser spot is determined by the LFI sensor, it is advantageously possible to achieve a fast, simple and precise in-situ calibration of the hybrid eye tracking device. The determination of the distance to the in-situ calibration target is carried out in a manner similar to the determination of the distance to the laboratory calibration target in the laboratory calibration method.
[0014] Furthermore, if in at least one additional in-situ calibration step, the spatial orientation of the in-situ calibration target in the coordinate system of the camera sensor is determined by means of checkerboard camera calibration based on the camera image of the camera sensor, the in-situ calibration target is preferably a checkerboard pattern target, preferably a ChArUco pattern target, then it is advantageously possible to achieve a fast, simple and precise in-situ calibration of the hybrid eye tracking device. For this purpose, preferably, the normal vector of the in-situ calibration target in the camera coordinates is determined from a plurality of different marker vectors t m1 to t m4 pointing to the in-situ calibration target, for example, the markers at the corners of the outermost checkerboard squares. In particular, the marker vectors and the illumination points of the marker vectors on the in-situ calibration target are determined from the pinhole camera model and the virtual image plane l. Subsequently, the normal vector (and thus the pose of the in-situ calibration target in the camera coordinate system) of the in-situ calibration target can be determined, in particular, from the marker vectors, especially by means of the following formula.
[0015]
[0016] Preferably, next, the laser vector t l
[0017]
[0018] pointing to the illumination point of the output laser signal on the in-situ calibration target in the camera coordinates and the intersection point of this laser vector with the image plane are determined.
[0019]
[0020] It is also proposed that in at least one additional in-situ calibration step, a virtual sphere is unfolded in the coordinate system of the camera sensor according to the determined spatial orientation of the in-situ calibration target, especially the checkerboard pattern target. The center point of the virtual sphere is formed by the laser spot on the in-situ calibration target, especially the laser spot located in the camera coordinates, and the radius of the virtual sphere is formed by the distance between the LFI sensor and the in-situ calibration target, especially the checkerboard pattern target, at the location of the laser spot, especially the measured distance. Advantageously, a fast, simple and precise in-situ calibration of the hybrid eye tracking device can thus be achieved.
[0021] Here, if in a further on-site calibration step, for at least three unknown spacings, especially the attitude, between the on-site calibration target, especially the checkerboard pattern target, and the camera sensor, virtual spheres are respectively unfolded in the coordinate system of the camera sensor such that the position of the LFI sensor in the coordinate system of the camera sensor can be determined from the intersections of these spheres, especially in a manner similar to trilateration, then it is advantageously possible to achieve fast, simple, and precise on-site calibration of the hybrid eye tracking device. In addition to trilateration using three spheres, another alternative for determining the attitude of the LFI sensor can be an iterative method in which an error function is minimized in the form of a gradient until the ideal / optimal position is found, for example, in the case of using the Gauss-Newton algorithm (Gauβ-Newton-Algorithmus) or the Levenberg-Marquardt algorithm (Levenberg-Marquardt-Algorithmus).
[0022] Furthermore, a data glasses is proposed, which has a hybrid eye tracking device that can preferably be calibrated or is preferably calibrated by means of a laboratory calibration method and / or by means of an on-site calibration method. Advantageously, a cost-effective, robust, user-friendly, and precise data glasses can thus be obtained.
[0023] Here, the hybrid eye tracking device according to the invention, the laboratory calibration method according to the invention, the on-site calibration method according to the invention, and the data glasses according to the invention should not be limited to the applications and embodiments described above. In particular, in order to implement the principle of operation described here, the hybrid eye tracking device according to the invention, the laboratory calibration method according to the invention, the on-site calibration method according to the invention, and the data glasses according to the invention can have a number different from the number mentioned here for the respective elements, components, and units as well as method steps. In addition, for the numerical ranges indicated in this disclosure, the values within the mentioned limits should also be considered as disclosed and can be used arbitrarily. Description of the Drawings
[0024] Other advantages result from the following description of the drawings. Embodiments of the invention are shown in the drawings. The drawings, the description, and the claims include a large number of feature combinations. A person skilled in the art will also appropriately observe the individual features and combine the features into other meaningful combinations.
[0025] The drawings show:
[0026] Figure 1 A schematic view of a data glasses having a hybrid eye tracking device
[0027] Figure 2Schematically shows the measurement process of an eye by an LFI sensor with a hybrid eye tracking device.
[0028] Figure 3 Shows a schematic flowchart of a laboratory calibration method for calibrating a hybrid eye tracking device.
[0029] Figure 4 Schematically shows the structure of a laboratory calibration device for performing the laboratory calibration method.
[0030] Figure 5 Shows a schematic front view of a laboratory calibration target for the laboratory calibration method.
[0031] Figure 6 Shows another schematic view of a part of the laboratory calibration device.
[0032] Figure 7 Shows a schematic flowchart of a field calibration method for calibrating a hybrid eye tracking device.
[0033] Figure 8 Shows a schematic front view of a field calibration target for the field calibration method.
[0034] Figure 9 Shows a schematic diagram of a field calibration step of the field calibration method in the coordinate system of the camera sensor of the hybrid eye tracking device.
[0035] Figure 10 Shows a schematic diagram of another field calibration step of the field calibration method in the coordinate system of the camera sensor. Detailed implementation
[0036] Figure 1Schematic illustration of data glasses 10. The data glasses 10 have a spectacle frame 64. The data glasses 10 have spectacle lenses 66. The data glasses 10 are set up to directly display a digital display into the user's field of view, for example via a virtual retinal display (not explicitly shown). The data glasses 10 have a hybrid eye tracking device 16. The hybrid eye tracking device 16 is configured as a hybrid high-speed eye tracking device. The hybrid eye tracking device 16 is set up to determine the instantaneous eye position of the eyes 68 of the user of the data glasses 10. The hybrid eye tracking device 16 has a camera sensor 12. The camera sensor 12 is configured as an infrared camera sensor. The camera sensor 12 is integrated into the spectacle frame 64. The camera sensor 12 is set up to take camera images (frames). The camera sensor 12 is set up to at least determine the eye position of the eyes 68 from the taken camera images. The data glasses 10, in particular the hybrid eye tracking device 16, have a computing unit 70. The computing unit 70 can be set up to analyze and process the camera images of the camera sensor 12. The computing unit 70 is integrated into the spectacle frame 64. However, it is also conceivable that the computing unit 70 is arranged outside the data glasses 10 and communicates with the camera sensor 12 only wirelessly. The hybrid eye tracking device 16 has a laser feedback interferometry (LFI) sensor 14. The hybrid eye tracking device 16 has a further LFI sensor 14'. The LFI sensors 14, 14' are set up to at least determine the instantaneous speed of the eye movement of the eyes 68. Compared to the camera sensor 12, the LFI sensors 14, 14' have a significantly lower resolution. Compared to the camera sensor 12, the LFI sensors 14, 14' have a significantly higher sampling rate.
[0037] Figure 2 Schematically shows the measurement process with the LFI sensors 14, 14'. The LFI sensor 14 radiates an infrared laser beam in the beam direction 36 onto the user's eyes 68. Since the LFI sensor 14 can only measure the speed changes transverse to the beam direction 36 of the laser of the LFI sensor 14, the angle of incidence 72 of the laser onto the surface of the eyes 68 must be continuously tracked. Therefore, Figure 2 marked with S 1 the LFI sensor 14 incident on the surface of the eyes 68 at the focus I 1 measures the following speed: The speed is incorrect due to the angle of incidence 72 (n l between S 1 and I 1 the angle between the extension line connecting them).
[0038] The hybrid eye tracking device 16 of the data glasses 10 can be calibrated by means of laboratory calibration methods (see Figure 3 ) and / or by means of on-site calibration methods (seeFigure 7 ) is calibrated. The data glasses 10 are calibrated by means of a laboratory calibration method (see Figure 3 ) and / or by means of an on-site calibration method (see Figure 7 ).
[0039] Figure 3 Fig. shows a schematic flow chart of a laboratory calibration method for calibrating the hybrid chamber eye tracking device 16. Figure 4 Schematically shows the construction of a laboratory calibration device 18 for performing the laboratory calibration method. The laboratory calibration device 18 includes a laboratory calibration target 22. The laboratory calibration target 22 can be configured as a flat monochromatic (e.g., black) plate. However, alternatively, a laboratory calibration target 22 with patterns and / or multiple colors can also be considered. In addition, the laboratory calibration device 18 includes a precision linear axis 26. The laboratory calibration target 22 is fastened to the precision linear axis 26. The laboratory calibration target 22 can be moved by means of the precision linear axis 26. The laboratory calibration target 22 can be moved along the z-direction by means of the precision linear axis 26. The laboratory calibration target 22 can move away from the camera sensor 12 and / or move towards the camera sensor 12 by means of the precision linear axis 26.
[0040] In the laboratory calibration method, the positions of the LFI sensors 14, 14' are determined in the coordinate system of the camera sensor 12. For this purpose, the laser points generated by the LFI sensors 14, 14' on the laboratory calibration target 22 at different predefined known distances 24, 24' between the laboratory calibration target 22 and the camera sensor 12 are obtained by means of the camera sensor 12 (see Figure 5 ) in the coordinate system of the camera sensor 12. Here, different distances 24, 24' are set by means of the precision linear axis 26. For this purpose, during laboratory calibration, the laboratory calibration target 22 is moved on the precision linear axis 26 so that the corresponding relative distances 24, 24' between the laboratory calibration target 22 and the camera sensor 12 can be directly read out or measured. In at least one laboratory calibration step 74, the laboratory calibration target 22 is arranged at a first distance 24 from the camera sensor 12. In at least one further laboratory calibration step 28, the first distance 30 between the LFI sensors 14, 14' and the laboratory calibration target 22 at the location of the laser point 20 on the laboratory calibration target 22 is obtained by means of the LFI sensors 14, 14'. In at least one further laboratory calibration step 76, the laboratory calibration target 22 is arranged at a second distance 24' from the camera sensor 12. In at least one further laboratory calibration step 28', the second distance between the LFI sensors 14, 14' and the laboratory calibration target 22 at the location of the laser point 20 on the laboratory calibration target 22 is obtained by means of the LFI sensors 14, 14'. Figure 5Exemplarily, laser points 20 on a laboratory calibration target 22 are shown for two different spacings 24, 24'. Here, the center of the laser point 20 moves along the x-direction and y-direction on the laboratory calibration target 22. In at least one additional laboratory calibration step 32, a unit vector 34 of the beam direction 36 of the LFI sensors 14, 14' is determined from the horizontal and / or vertical pixel spacings (Δx and Δy) between camera images of the same laser point 20 for different spacings 24, 24' of the laboratory calibration target 22 from the camera sensor 12 (see Figure 6 ). In Figure 6 , the measured distance 30 and the determined unit vector 34 are denoted by the abbreviations d meas and e l . The coordinate system of the camera sensor 12 is denoted by the abbreviations R and t. Then, in at least one additional laboratory calibration step 78, from an overview of the determined distance 30 between the LFI sensors 14, 14' and the laboratory calibration target 22 and the determined unit vector 34 of the beam direction 36 of the LFI sensors 14, 14', the position of the LFI sensors 14, 14' relative to the coordinate system of the camera sensor 12 is determined. Thus, the hybrid eye tracking device 16 is calibrated. By using the combination of the LFI sensor 14 and the camera sensor 12, it is shown that the name "hybrid" eye tracking is correct.
[0041] Figure 7 Fig. shows a schematic flow chart of a field calibration method for calibrating a hybrid eye tracking device 16. In the field calibration method, the positions of the LFI sensors 14, 14' in the coordinate system of the camera sensor 12 are determined. For this purpose, with the aid of the camera sensor 12, the positions of the laser points 62 generated by the LFI sensors 14, 14' on a defined and known calibration pattern 38 of the field calibration target 40 are determined in the coordinate system of the camera sensor 12 for different unknown spacings 42 of the field calibration target 40 from the camera sensor 12. The field calibration target is in particular flat. Figure 8 Fig. shows an exemplary and advantageous embodiment of a field calibration target 40. The field calibration target 40 is constructed flat. The field calibration target 40 is constructed as a chessboard pattern target. The field calibration target 40 has a chessboard pattern. The field calibration target 40 is constructed as a ChArUco pattern target. The field calibration target 40 has a ChArUco pattern. The field calibration target 40 can be printed by the user himself. When performing the field calibration method, the user places the field calibration target 40 in such a way that it has different, unknown spacings 42 from the camera sensor 12.
[0042] In at least one on-site calibration step 80, the on-site calibration target 40 is arranged to have a first spacing 42 from the camera sensor 12. In at least one further on-site calibration step 44, a first distance 46 between the LFI sensors 14, 14' and the on-site calibration target 40 at the location of the laser spot 62 on the on-site calibration target 40 is determined by the LFI sensors 14, 14'. In at least one further on-site calibration step 82, the on-site calibration target 40 is arranged to have a second spacing from the camera sensor 12. In at least one further on-site calibration step 44', a second distance 46' between the LFI sensors 14, 14' and the on-site calibration target 40 at the location of the laser spot 62 on the on-site calibration target 40 is determined by the LFI sensors 14, 14'. In at least one further on-site calibration step 84, the on-site calibration target 40 is arranged to have a third spacing from the camera sensor 12. In at least one further on-site calibration step 44", a third distance 46" between the LFI sensors 14, 14' and the on-site calibration target 40 at the location of the laser spot 62 on the on-site calibration target 40 is determined by the LFI sensors 14, 14'.
[0043] In at least one further on-site calibration step 48, the spatial orientation of the on-site calibration target 40 in the coordinate system of the camera sensor 12 is determined by means of chessboard camera calibration based on the camera image of the camera sensor 12 (see also Figure 9 or Figure 10 ). For this purpose, (here, for example, four) markers 90, 90', 90", 90"' of a defined and known calibration pattern 38 are identified in the camera image, and the markers are analyzed to determine the orientation. In order to determine the orientation of the on-site calibration target 40 in space, marker vectors 88, 88', 88", 88"' (denoted by t Figure 9 to t m1 in m4 ) pointing to the respective 90, 90', 90", 90"' are formed in the coordinate system of the camera sensor 12, and the marker vectors are analyzed. From the determined range of the on-site calibration target 40 in space, the normal vector 86 of the on-site calibration target 40 is determined (in Figure 9In the following, n is used to denote). In at least one further on-site calibration step 50, a virtual sphere 52, 52', 52" is unfolded in the coordinate system of the camera sensor 12 according to the sought spatial orientation of the on-site calibration target 40. The center points 54, 54', 54" of these virtual spheres 52, 52', 52" are formed by the respective positions of the laser points 62, 62', 62" on the on-site calibration target 40 at different, initially unknown spacings 42. The radii 56, 56', 56" of the virtual spheres 52, 52', 52" are formed by the respective distances 46, 46', 46" between the LFI sensors 14, 14' and the on-site calibration target 40 at the locations of the respective laser points 62, 62', 62" on the on-site calibration target 40. In at least one further on-site calibration step 58, for at least three unknown spacings 42 between the on-site calibration target 40 and the camera sensor 12, respective virtual spheres 52, 52', 52" are unfolded in the coordinate system of the camera sensor 12, such that the positions of the LFI sensors 14, 14' in the coordinate system of the camera sensor 12 can be determined from the intersections 60 of these virtual spheres 52, 52', 52". Thus, the positions of the LFI sensors 14, 14' in the coordinate system of the camera sensor 12 are determined by a trilateration with respect to three spatially virtual spheres 52, 52', 52". In Figure 9 the laser vector 92 of the LFI sensor 14 in the coordinate system of the camera sensor 12 is denoted by the abbreviation t l Thus, the hybrid eye tracking device 16 is calibrated or re-calibrated. The on-site calibration method is preferably used for re-calibration, while the laboratory calibration method is preferably used for initial calibration.
Claims
1. A hybrid eye tracking device (16), in particular a hybrid high-speed eye tracking device, preferably a hybrid eye tracking device in data glasses (10), for determining an instantaneous eye position, in particular the instantaneous eye position of an eye (68) of a user of the data glasses (10), the hybrid eye tracking device comprising at least one camera sensor (12) and having at least one laser feedback interferometry (LFI) sensor (14, 14'), wherein, the camera sensor (12) is at least configured to determine an eye position from a camera image, and wherein the LFI sensor (14, 14') is at least configured to determine an instantaneous velocity, in particular the instantaneous velocity of eye movement.
2. A hybrid eye tracking device (16), characterized in that, relative to the camera sensor (12), the LFI sensor (14, 14') has a significantly lower resolution and a significantly higher sampling rate.
3. A laboratory calibration method for calibrating the hybrid eye tracking device (16) according to any one of claims 1 or 2, characterized in that, the position of the LFI sensor (14, 14') in the coordinate system of the camera sensor (12) is determined by obtaining, by means of the camera sensor (12): the position of a laser spot (20) generated by the LFI sensor (14, 14') on a laboratory calibration target (22) in the coordinate system of the camera sensor (12) at different predefined known distances (24, 24') between the laboratory calibration target (22) and the camera sensor (12), the laboratory calibration target being preferably flat and preferably monochromatic.
4. The laboratory calibration method according to claim 3, characterized in that, during the calibration, the laboratory calibration target (22) is moved on a precision linear axis (26) such that the corresponding relative distances (24, 24') between the laboratory calibration target (22) and the camera sensor (12) can be directly read out or measured.
5. The laboratory calibration method according to claim 3 or 4, characterized in that, in at least one laboratory calibration step (28, 28'), the distance (30) between the LFI sensor (14, 14') and the laboratory calibration target (22) at the location of the laser spot (20) is obtained by means of the LFI sensor (14, 14').
6. The laboratory calibration method according to any one of claims 3 to 5, characterized in that, in at least one laboratory calibration step (32), a unit vector (34) of the beam direction (36) of the LFI sensor (14, 14') is obtained from the horizontal and / or vertical pixel distances between camera images of the same laser spot (20) at different distances (24, 24') between the laboratory calibration target (22) and the camera sensor (12).
7. A method for on-site calibration of a hybrid eye tracking device (16) according to any one of claims 1 or 2, characterized in that the position of the LFI sensor (14, 14') in the coordinate system of the camera sensor (12) is determined by the camera sensor (12) by determining: in different unknown distances (42) between the on-site calibration target (40) and the camera sensor (12), the positions in the coordinate system of the camera sensor (12) of the laser points (62, 62', 62") generated by the LFI sensor (14, 14') on a defined and known calibration pattern (38) of the on-site calibration target (40), the on-site calibration target being in particular flat, the on-site calibration target preferably being a checkerboard pattern target, preferably a ChArUco pattern target.
8. The on-site calibration method according to claim 7, characterized in that in at least one on-site calibration step (44, 44', 44"), the distance (46, 46', 46") between the LFI sensor (14, 14') and the on-site calibration target (40) at the location of the laser point (62, 62', 62") is determined by the LFI sensor (14, 14').
9. The on-site calibration method according to claim 7 or 8, characterized in that in at least one on-site calibration step (48), based on the camera image of the camera sensor (12), the spatial orientation of the on-site calibration target (40) in the coordinate system of the camera sensor (12) is determined by means of checkerboard camera calibration.
10. The on-site calibration method according to claims 8 and 9, characterized in that in at least one further on-site calibration step (50), based on the determined spatial orientation of the on-site calibration target (40), a virtual sphere (52, 52', 52") is unfolded in the coordinate system of the camera sensor (12), the center point (54, 54', 54") of the virtual sphere being formed by the laser points (62, 62', 62") on the on-site calibration target (40), and the radius (56, 56', 56") of the virtual sphere being formed by the distance (46, 46', 46") between the LFI sensor (14, 14') and the on-site calibration target (40) at the location of the laser point (62, 62', 62").
11. The on-site calibration method according to claim 10, characterized in that In the additional on-site calibration step (58), for at least three unknown spacings (42) between the on-site calibration target (40) and the camera sensor (12), virtual spheres (52, 52', 52") are respectively unfolded in the coordinate system of the camera sensor (12), so that the positions of the LFI sensors (14, 14') in the coordinate system of the camera sensor (12) can be obtained from the intersections (60) of these virtual spheres (52, 52', 52").
12. A data glasses (10), the data glasses having a hybrid eye tracking device (16) according to any one of claims 1 or 2, and the hybrid eye tracking device is preferably calibrated by means of a laboratory calibration method according to any one of claims 3 to 6 and / or an on-site calibration method according to any one of claims 7 to 11.