Method and device for determining direction of sight of eye, and data glasses
By arranging multiple static laser sensors on the data glasses, combining classifiers and integration techniques, the offset and error problems of line of sight direction determination in the prior art are solved, and high-precision and low-cost eye tracing are achieved.
Patent Information
- Application Number
- CN202380036545.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-27
- Filing Date
- 2023-03-02
- Publication Date
- 2024-12-06
AI Technical Summary
Prior arts have difficulty in achieving accurate tracking without relying on high-quality eye size basics and geometric models when determining eye sight direction, especially when equipment slips or eye movements, with offsets and errors.
Multiple static laser sensors are used to arrange them on the data glasses. By reading the spacing value and velocity value, combining classifiers and estimation methods, the reflection point of the laser beam on the eye is determined, and the simplified eye model and integration technology are used to calculate the none. offset line of sight direction and high-precision measurements are achieved through laser self-mixing effect and resonator.
Achieve rapid and accurate determination of eye sight direction without the need for cameras and high-quality geometric models, adapting to different numbers and locations of laser sensors, reducing hardware costs and current consumption, ensuring that when the device slides down or blinks Robustness.
Smart Images

Figure CN119096219A_ABST
Abstract
Description
Technical Field
[0001] The invention proceeds from a generic device or method according to the independent claims. The subject matter of the invention is also a computer program. Background Art
[0002] The goal of eye tracking (also called gaze detection) with the aid of portable devices is to precisely estimate the gaze angle in order to derive information about the gaze direction or to adapt display information depending on the eye position. Technical implementations range from video nystagmus systems (VOG) to special tracking devices based on the pupil-corneal reflection.
[0003] For general classification tasks, a number of methods are known which are suitable for a wide range of application cases. Compared to the related (unsupervised) clustering task, the classification task requires problem-related knowledge about how the clusters should be named. Summary of the invention
[0004] Against this background, the solution proposed here provides a method for determining the direction of sight of an eye according to the main claim, furthermore a device using the method, and finally a corresponding computer program and a pair of data glasses. Advantageous embodiments and improvements of the device described in the independent claim can be achieved by the measures listed in the dependent claims.
[0005] The method for determining the direction of eye gaze comprises the following steps:
[0006] reading in a first distance value and a first speed value, wherein the first distance value represents the distance between a first laser sensor for emitting a first laser beam and a first surface intersection point of the laser beam on the eye, and wherein the first speed value represents the surface speed at the first surface intersection point,
[0007] reading in a second distance value and a second speed value, wherein the second distance value represents the distance between a second laser sensor for emitting a second laser beam and a second surface intersection point of the laser beam on the eye, and wherein the second speed value represents the surface speed at the second surface intersection point,
[0008] reading in at least one third distance value and a third speed value, wherein the third distance value represents the distance between a third laser sensor for emitting a third laser beam and a third surface intersection point of the laser beam on the eye, and wherein the third speed value represents the surface speed at the third surface intersection point;
[0009] When the distance value and the speed value are used, a sight direction value representing the sight direction is obtained.
[0010] Advantageously, the method can be used in the context of a device that can be worn by a person, such as data glasses. The laser sensor can be arranged, for example, on a frame of a portable device. The distance value and the speed value can represent measured values or values determined from the measured values. For example, a run time measurement, a detection of a Doppler shift or an analysis of changes in signal characteristics due to interference can be performed for this purpose. The surface intersection point can represent a position on a part of the eye at which the laser beam is reflected. Such a part can be, for example, the iris or the retina. The surface velocity can therefore be caused by the movement of the following part of the eye: the corresponding surface intersection point is located on this part.
[0011] The method may include the following steps: classifying the distance values and the speed values. Through this classification, a first classification value (Klassifikationswert) can be obtained, a second classification value can be obtained, and a third classification value can be obtained, the first classification value showing the part of the eye assigned to the first surface intersection, the second classification value showing the part of the eye assigned to the second surface intersection, and the third classification value showing the part of the eye assigned to the third surface intersection. In the step of determining, the line of sight direction value can be determined using the classification values. With the help of a suitable classifier, it is possible to determine in a simple manner where the individual laser beams are reflected on the eye. Possible parts of the eye that can be displayed by the classification values include, for example, the sclera, iris, retina and eyelid of the eye or a subset of the mentioned parts.
[0012] The method may include the steps of determining a first absolute value representing a pose of the first laser sensor in head-fixed coordinates of the head-fixed coordinate system, a second absolute value representing a pose of the second laser sensor in head-fixed coordinates, and a third absolute value representing a pose of the third laser sensor in head-fixed coordinates. The absolute values may be determined using spacing values and classification values.
[0013] Optionally, the method may include the following steps: estimating a first position value, a second position value, and a third position value, wherein the first position value represents the position of the first laser sensor, the second position value represents the position of the second laser sensor, and the third position value represents the position of the third laser sensor. The estimating step may be performed using a spacing value, a speed value, and a classification value. In the obtaining step, a sight direction value may be obtained using the position value, the spacing value, and the speed value. The estimating may be performed using a suitable estimation method.
[0014] For example, the position and orientation of the sensor in head-fixed coordinates can then be determined from the position values and the classification values. For this optional step, the distance measurement values are used, which are assigned as measurement points to the sclera and thus to the eyeball with a known fixed radius r_sclera. From the known position values in eye-fixed coordinates and the measured distances, the irradiation point on the sclera can be determined, and with the known radius r_sclera, the eyeball and its center can be determined with reference to the position values of the laser sensor. For this purpose, a sphere with r_sclera is fitted to a collection of sclera measurements.
[0015] Advantageously, the eyeglass coordinate system can be set and tracked with reference to the head coordinate system, so that the gaze direction can be carried out robustly and without errors even if the eyeglasses slip off the nose.
[0016] The method may include the step of determining an angular velocity, the angular velocity representing an eye surface velocity of the eye. The angular velocity may be determined using a surface intersection and a correlation between a head-fixed coordinate system and an eyeglass coordinate system. Furthermore, the method may include the step of determining a gaze direction with an offset using the angular velocity.
[0017] Therefore, in a further step, the eye surface velocity can be determined for all measured values as a projection of the measured velocity in the direction of the laser beam. For this purpose, the connection of the coordinate system determined in the previous step and the surface intersection point are used. Next, the sight direction with the offset is determined, for example, by integration of the surface velocity.
[0018] In order to finally determine the non-deflected and therefore correct gaze direction, in a third step, a simplified eye model is optionally used, from which the deviation of the gaze angle is determined. In this case, the eye model is determined or fitted from the distance measurement points and the velocity measurement points and the previously determined surface intersection points. In this case, the iris can be regarded as a simple plane with a normal vector that corresponds to the absolute gaze angle in the eye coordinate system.
[0019] Finally, the correct gaze direction can be determined by using the gaze direction with offset obtained from the integrated velocity measurement and correcting the offset using a simplified eye model.
[0020] Therefore, the method may include the step of obtaining a gaze direction value representing the gaze direction using the gaze direction having the offset and an offset for correcting the gaze direction having the offset.
[0021] In the step of determining, using the position values, the distance values and known geometric assumptions of the eye, parameters of the connection between the head-fixed coordinate system and the eyeglass coordinate system can be determined. The geometric assumption can be based on a model of the eye, for example.
[0022] Optionally, the sight direction with the offset may be determined by integrating the angular velocity.
[0023] The offset can be determined using an eye surface model which is determined from distance values, velocity values and surface intersection points.
[0024] Optionally, the eye surface model may be learned over time from measurements. Thus, the method may comprise the step of: the eye surface model is continuously learned over time from detected data points. Advantageously, the data points may be based on measurements of a laser sensor.
[0025] The method may include the steps of receiving a first reflected beam, a second reflected beam, and a third reflected beam, wherein the first reflected beam represents a reflection or scattering of the first laser beam at a first surface intersection, the second reflected beam represents a reflection or scattering of the second laser beam at a second surface intersection, and the third reflected beam represents a reflection or scattering of the third laser beam at a third surface intersection. In addition, the method may include the steps of determining a first spacing value and a first speed value using fluctuations in the radiation intensity of the first reflected beam, determining a second spacing value and a second speed value using fluctuations in the radiation intensity of the second reflected beam, and determining a third spacing value and a third speed value using fluctuations in the radiation intensity of the third reflected beam. Advantageously, fluctuations based on interference effects can be analyzed here.
[0026] The method may include the steps of emitting a first laser beam at a constant wavelength, emitting a second laser beam at a constant wavelength, and emitting a third laser beam at a constant wavelength. In the step of determining, during the emission of the first laser beam at a constant wavelength, a first velocity value may be determined using fluctuations in the radiation intensity of the first reflected beam, during the emission of the second laser beam at a constant wavelength, a second velocity value may be determined using fluctuations in the radiation intensity of the second reflected beam, and during the emission of the third laser beam at a constant wavelength, a third velocity value may be determined using fluctuations in the radiation intensity of the third reflected beam. This enables a simple determination of the respective surface velocity.
[0027] Accordingly, in the step of emitting, the first laser beam can be emitted at a changed wavelength during a further first time interval, the second laser beam can be emitted at a changed wavelength during a further second time interval, and the third laser beam can be emitted at a changed wavelength during a further third time interval. Here, in the step of determining, during the emission of the first laser beam at a changed wavelength, the first distance value can be determined using fluctuations in the radiation intensity of the first reflected beam, during the emission of the second laser beam at a changed wavelength, the second distance value can be determined using fluctuations in the radiation intensity of the second reflected beam, and during the emission of the third laser beam at a changed wavelength, the third distance value can be determined using fluctuations in the radiation intensity of the third reflected beam. A changed wavelength can be understood as a change in the wavelength of the laser radiation emitted by the respective laser sensor. Thus, a simple determination of the distance between the respective sensor and the respective reflection point can be achieved.
[0028] The method may include the following steps: determining a first beat frequency using fluctuations in the radiation intensity of the first reflected beam during emission of the first laser beam at a constant and a changing wavelength, determining a second beat frequency using fluctuations in the radiation intensity of the second reflected beam during emission of the second laser beam at a constant and a changing wavelength, and determining a third beat frequency using fluctuations in the radiation intensity of the third reflected beam during emission of the third laser beam at a constant and a changing wavelength. In the determining step, a movement direction value representing the movement direction of the eye and a movement speed value representing the movement speed of the eye can be determined using an average value of the beat frequencies. In the ascertaining step, a sight direction value can be ascertained using the movement direction value and the movement speed value of the eye. In this way, the effects resulting from the constant and the changing wavelength of the emitted laser beam can be used to ascertain the sight direction and the movement of the eye.
[0029] The method can be implemented, for example, in the form of software or hardware or in a hybrid form consisting of software and hardware, for example in a control unit.
[0030] In addition, the solution proposed here realizes a device for determining the direction of sight of an eye. The device is designed to perform, control or implement a variant of the method proposed here in a corresponding device. Through this embodiment variant of the invention in the form of a device, the task on which the invention is based can also be solved quickly and efficiently.
[0031] For this purpose, the device may have at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or actuator, and / or at least one communication interface, the interface being used to read a sensor signal of a sensor or to output a data signal or a control signal to an actuator, the communication interface being used to read or output data embedded in a communication protocol. The computing unit may be, for example, a signal processor, a microcontroller or the like, wherein the memory unit may be a flash memory, an EEPROM or a magnetic memory unit. The communication interface may be configured to read in or output data wirelessly and / or wired, wherein a communication interface that can read in or output wired data may, for example, read in these data from a corresponding data transmission line electrically or optically or output these data to a corresponding data transmission line.
[0032] In the present case, the device can be understood as an electric appliance, which processes sensor signals and outputs control signals and / or data signals according to the sensor signals. The device can have an interface, which can be constructed in hardware and / or in software. In the hardware-based construction scheme, the interface can be, for example, a part of a so-called system ASIC, which contains various functions of the device. However, it is also possible that the interface is a self-owned integrated circuit or is at least partially composed of discrete structural elements. In the software-based construction, the interface can be, for example, a software module that coexists with other software modules on a microcontroller.
[0033] Advantageously, a corresponding device can be implemented in the context of data glasses which, in addition to the device, also comprise at least three laser sensors for emitting laser radiation in the direction of the eye and for receiving reflected beams of the laser radiation reflected at the eye.
[0034] A computer program product or a computer program with a program code is also advantageous, wherein the program code can be stored on a machine-readable carrier or storage medium, such as a semiconductor memory, a hard disk memory or an optical memory, and in particular when the program product or the program is executed on a computer or a device, the program code is used to execute, implement and / or control the steps of the method described in one of the preceding embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] An exemplary embodiment of the solution proposed here is shown in the drawings and explained in more detail in the following description. The drawings show:
[0036] Figure 1 shows a section of data glasses according to one embodiment;
[0037] Figure 2 shows the spectrum of an unmodulated sensor in case of constant motion according to one embodiment;
[0038] Figure 3a and 3b shows a spectrum of motion of an object for scattering while modulating a sensor according to one embodiment;
[0039] Figure 4 shows a model of glasses according to one embodiment;
[0040] Figure 5 A classifier for surface classification according to one embodiment is shown;
[0041] Figure 6 An estimator for estimating absolute sensor position and intersection point in head-fixed coordinates is shown according to one embodiment;
[0042] Figure 7 A flow chart showing the estimation of eye position according to one embodiment;
[0043] Figure 8 shows an eye geometry with an offset according to one embodiment;
[0044] Fig. 9 shows a model of an eye according to one embodiment;
[0045] Fig.10 illustrates an embodiment of an eye tracker functionality according to one embodiment;
[0046] Fig.11 A deterministic automaton for differences in spacing or distances according to one embodiment is shown;
[0047] Fig.12 illustrates a hidden Markov model according to one embodiment;
[0048] Fig.13 shows a normal distribution for conditional radiation probability according to one embodiment;
[0049] Fig.14 shows a matrix with transition probabilities according to one embodiment;
[0050] Fig.15 A flow chart of a method for determining a gaze direction of an eye according to one embodiment is shown. DETAILED DESCRIPTION
[0051] In the following description of advantageous exemplary embodiments of the invention, identical or similar reference numerals are used for similarly acting elements which are illustrated in the various figures, wherein a repeated description of these elements is omitted.
[0052] Figure 1 A section of data glasses 1 as an embodiment of a portable device is shown. A plurality of laser sensors 2, 3, 4, 5, 6 are shown by way of example, which are arranged on the data glasses 1. The number of laser sensors 2, 3, 4, 5, 6 can be freely selected, wherein, according to one embodiment, at least three laser sensors 2, 3, 4, 5, 6 are required. Optionally, a device 1a is arranged on the data glasses 1, which is used to determine the line of sight direction of the eye, at which the laser beam of the laser sensor 2, 3, 4, 5, 6 is directed, using the sensor values of the laser sensors 2, 3, 4, 5, 6. The device 6a is therefore a signal processing unit, which is coupled to the laser sensors 2, 3, 4, 5, 6 in a signal-transmittable manner. According to one embodiment, the laser sensors 2, 3, 4, 5, 6 are implemented as static sensors, which are rigidly fastened, for example, to the frame of the data glasses 1.
[0053] According to one embodiment, each of laser sensors 2, 3, 4, 5, 6 is designed to emit a single laser beam. Furthermore, according to one embodiment, laser sensors 2, 3, 4, 5, 6 are realized individually and are arranged spaced apart from one another, for example at a spacing of at least 5 mm.
[0054] With the solution described here, eye tracking, i.e. line of sight detection or line of sight tracking, can be achieved by means of a data-based solution with the aid of an arrangement of multiple static laser sensors. For this purpose, an estimation can be performed, which is performed mainly or only based on the data provided by the laser sensors 2, 3, 4, 5, 6 themselves and not based on a geometric model for which high-quality parameters must be known in advance or the geometric model must be solved very accurately during runtime. Instead, the proposed solution, which can be implemented, for example, in the form of an algorithm, does not require or hardly requires basic knowledge about the eye size of an eye for which the line of sight direction can be determined. In addition, the solution also works properly when the measuring device (here, for example, the data glasses 1) slips off. Therefore, the solution can achieve eye tracking based on static laser sensors 2, 3, 4, 5, 6 with uncertain (unsicher) positions. Advantageously, a camera for detecting the eye is also not required.
[0055] According to one embodiment, the proposed solution only requires a small number of static lasers in hardware, rather than a complete camera or laser scanner. The corresponding lasers are integrated in the laser sensors 2, 3, 4, 5, 6, for example. Therefore, the proposed solution is cost-effective due to the lower hardware requirements and is more robust than the device for scanning, because only static lasers are required. According to one embodiment, the laser sensors 2, 3, 4, 5, 6 are capable of measuring distance and speed at an update rate of at least 1 kHz, so that the update rate of the eye position based on the proposed algorithm is greater than 1 kHz. In addition, the sensor is extremely small, for example greater than or less than 2 mm, so that the sensor can be seamlessly integrated into the glasses frame, glasses and glasses legs. In addition, the algorithm is very streamlined, so that the algorithm can be implemented on an integrated device with low current consumption.
[0056] From an algorithmic point of view, the approach does not rely on a large number of higher quality parameters compared to model-based estimation approaches. Instead, the sensor positions of the individual laser sensors 2, 3, 4, 5, 6 and the eye position itself are estimated during runtime. The algorithm itself can be easily adapted to a different number of sensors. This is advantageous for different hardware setups with a variable number of sensors and for situations where the individual sensors either work with errors or are not accurately aimed at the eye. In addition to this, the time until convergence is short (in the sub-second range), which is advantageous in the case of slips or blinks.
[0057] The working principle for the laser sensors 2, 3, 4, 5, 6 is based on an optical resonator. In the resonator, electrons are excited by an external energy supply. The radiation generated by spontaneous emission is reflected back and forth in the optical resonator and leads to stimulated emission, thereby enhancing the resonance mode and generating coherent radiation. On one side of such a laser resonator, the laser radiation can be coupled into free space via a semi-transparent reflector. According to one embodiment, a surface emitter, a so-called vertical cavity surface emitting laser (VCSEL), is used for the laser sensors 2, 3, 4, 5, 6, in the case of which the reflector structure is implemented as a Bragg reflector, a so-called distributed Bragg reflector (DBR). The basic physical effect used in the scheme described here is the so-called "laser self-mixing". If the laser radiation scattered from the outside of the reflected beam is coupled back into the laser resonator, the stimulated emission is modulated according to the phase of the scattered back photons. For example, such scattered laser radiation can be a reflected beam, which can represent a portion of the laser beam emitted by one of the laser sensors 2, 3, 4, 5, 6 that is reflected at the surface intersection of a part of the eye. If the double spacing (out and back) between the laser resonator and the external scattering surface corresponds to an integer multiple of the wavelength, the scattered radiation and the radiation inside the laser resonator are in phase. This leads to positive interference, the laser threshold is lowered, and the laser power is slightly increased, which is detected by a photodiode, which is integrated in the laser sensors 2, 3, 4, 5, 6 according to one embodiment. In the case of slightly larger spacings, the two radiation waves are phase-shifted and at a certain point negative interference occurs, which reduces the laser output power. If the spacing to the surface that scatters changes at a constant speed, the laser power fluctuates between a maximum value in the case of constructive interference and a minimum value in the case of destructive interference. The resulting oscillation is a function of the speed of the scatterer and the laser wavelength. The same effect (oscillating laser output power) can be observed when the spacing between the laser resonator and the surface that scatters remains constant, but the laser wavelength changes. Now, the phase of the radiation between the inner hollow space (in the laser) and the outer hollow space (between the laser and the scattering) depends on how much wavelength "matches" the outer hollow space. However, the vibration frequency is related to the distance between the laser and the surface where the scattering occurs, because the wavelength in the near-infrared range is 850nm and the distance from the external hollow space is several tens of millimeters, so that a slight change in the laser wavelength can lead to a complete rotation of the laser phase in the external hollow space. The larger the distance, the smaller the wavelength change that leads to a complete change in the phase of the laser in the external resonator. If the fluctuation of the laser output power is observed, it is shown that, when the change of the laser wavelength is constant, the larger the distance from the scatterer, the higher the frequency of the power fluctuation. That is, if the power monitoring of the photodiode is mapped in the frequency range, the peak frequency is associated with the distance from the scatterer.The variation of the laser wavelength can be achieved by power modulation of the VCSEL. For example, linear modulation can be achieved by means of triangular laser current. Other known modulation methods, such as quadrature and sinusoidal, can also be used.
[0058] If both effects are superimposed, a difference frequency results, as is known from frequency modulated continuous wave radar systems (FMCW). Due to the Doppler shift of the frequency, the resulting difference frequency for a target moving towards the sensor is lower when the frequency is increased and higher when the frequency is decreased. The difference frequency for the rising and falling modulation segments can therefore be calculated individually. The average of the two frequencies is an indicator of the distance to the target, while the difference is associated with half the Doppler frequency.
[0059] exist Figure 1 In the exemplary data glasses 1 shown, the temples and the spectacle frames are in this example laser sensors 2, 3, 4, 5, 6 in the form of so-called ViP components. The laser beams of the laser sensors 2, 3, 4, 5 are arranged such that they focus four points on the surface of the eye.
[0060] The laser sensors 2, 3, 4, 5 can also be integrated into the eyeglasses or nose pads. In another possible embodiment, the laser sensors 2, 3, 4, 5, 6 are arranged in the temples of the glasses, but radiate towards the lenses of the data glasses 1. According to one embodiment, the holographic optical element (HOE) embedded in the lens is designed in this case to deflect the beam towards the eye. The arrangement can be operated in different modes. First, the laser is operated with a continuous wave (constant frequency), i.e. a constant wavelength. Secondly, the frequency of the laser is modulated in time, i.e. according to a triangular modulation scheme, as described above, so that there is a changing wavelength.
[0061] Figure 2 The spectrum of an unmodulated sensor in constant motion is shown. This is therefore a spectrum in the frequency range for the first case described above with a constant frequency and a constant speed of the scattering object. The scattering object is the eye or a part of the eye at which the laser beam is reflected or scattered. An amplitude axis 8 and a frequency axis 7 are shown. The Doppler frequency shift is the peak frequency 9 of the spectrum.
[0062] and Figure 2 different, Figure 3a and 3bAn exemplary spectrum for the movement of a scattering object while modulating the sensor is shown. This is therefore a spectrum for the second modulated operating type mentioned above, which again has a constant velocity of the scatterer. Figure 2 , which is shown with the aid of an amplitude axis 8 and a frequency axis 7. In this case, the scatterer is the eye or a part of the eye at which the laser beam is reflected or scattered. In the absence of movement, the spectrum is obtained with Figure 2 , but now the peak frequencies 10, 11 determine the distance between the sensor and the scattering surface. In the case of additional movement, the peak frequencies 10, 11 shift upwards or downwards, depending on the modulation ramp sequence (upwards or downwards) and the velocity vector (towards or away from the sensor). Figure 3a and 3b The resulting spectra for both ramp sequences are shown in . By separating the peak frequency 12 between the increasing and decreasing slopes, the direction and absolute speed of motion can be estimated.
[0063] Figure 4 A model of glasses is shown, which are, for example, data glasses 1, which have six laser sensors 2, 3, 4, 5, 6, 21 directed at the eye 20. The described solution requires multiple static lasers, which are mounted on a portable device, such as glasses. The laser can simultaneously measure the distance to a fixed object and the surface speed of the object along the laser line. It is assumed in the following that all lasers of the laser sensors 2, 3, 4, 5, 6, 21 are directed at the sclera, iris or retina of the eye 20. For the sake of simplicity, other situations are excluded, such as lasers that are not directed at the surface of the eye, eyelid blinking, etc., but the solution can be extended for such situations. It is also assumed that all laser sensors 2, 3, 4, 5, 6, 21 are firmly connected to the portable device so that the relative positions between the lasers and their orientation relative to the base frame (here, data glasses 1) remain constant (e.g. "glasses fixed coordinates").
[0064] The task of estimating the eye position when the laser position is unknown can be divided into three steps:
[0065] 1. Classify the surface
[0066] 2. Absolute sensor position
[0067] 3. Estimation of eye position
[0068] For the first task, it is assumed that the relative position and orientation of the laser sensors 2, 3, 4, 5, 6, 21 relative to the eye 20 are unknown. However, the evaluation of the distance measurement can indicate whether the laser is pointed straight at the sclera and therefore provides a relatively constant scleral radius value, whether there is a drop onto the iris disk or an even stronger drop onto the retina. The classifier function acting on the distance signal thus provides important first information about the position between the laser and the eye 20.
[0069] As an additional feature, scattering properties that cause signal quality variations can be exploited by different parts of the eye 20 (e.g., iris, sclera, retina, eyelids) in order to determine on which part the laser beam is located. An initial calibration gesture of the eye 20 can be used to classify the part, such as turning the eye 20 and blinking, and a multimodal distribution can be adapted to the detected SNR features and spacing features in order to train a classifier and obtain a corresponding decision boundary.
[0070] The second task is more computationally complex: Based on the above classification, a series of measurement points located on the sclera of the eye 20 is used in order to determine the absolute position of the laser sensors 2, 3, 4, 5, 6, 21. The number of samples used, i.e. the batch size, is a design parameter. The sclera measurement points are known in eye-fixed coordinates, i.e.
[0071] x g,i =s g,i +d i p i (1)
[0072] Among them, s g,i describes the laser position in eyeglass coordinates, d i describes the measured distance between the laser and the sclera, and p i Describes the orientation of the laser, which is also known in glasses-fixed coordinates.
[0073] To determine the sensor position in head-fixed coordinates, try transforming to head-fixed coordinates
[0074] x h =R(θ,φ,ρ)x g +t (2)
[0075] The transformation contains the parameters θ, φ, ρ, t to be determined x ,t y ,t z Since only scleral points are detected in the stack, the necessary conditions need to be met:
[0076] ∥x h ∥=r 巩膜(3)
[0077] Where, the scleral radius r 巩膜 is a relatively constant parameter that is assumed to be known. The equation can be rewritten in quadratic form:
[0078]
[0079] Since R is a one-dimensional matrix, the equation can be further simplified to
[0080]
[0081] Depending on the size of the stack, this is a nonlinear system of equations. It can be solved locally using standard methods, such as the "Levenberg-Marquardt" algorithm. The parameters for the affine transformation from eyeglass coordinates to head-fixed coordinates and thus the absolute sensor position in head-fixed coordinates are thus obtained. The practical problem of eye pose estimation is thus solved.
[0082] Figure 5 A corresponding classifier 30 is shown for classifying the surface for each of the laser sensors.
[0083] Figure 6 A corresponding estimator 32 for estimating the absolute sensor position and the intersection point in head-fixed coordinates is shown.
[0084] Figure 7 A flow chart of a corresponding estimation of the eye position according to one embodiment is shown. In a first block 40, the angular velocity is calculated with the aid of a possibly consistent set of linear equations, which can also be referred to as a means for determining the angular velocity.
[0085]
[0086] The solution to this system of equations is the angular velocity The angular velocity can be further integrated and given by These values are not reliable estimates for the true viewing angle, since the starting value of the integration is unknown. Therefore, an offset can at least be expected.
[0087] To calculate the estimated error, in block 42, using In the case of , the measured intersection point is transformed into eye-fixed coordinates. By storing data points or by training an approximate surrogate model, the following Figure 6The outer eye geometry is shown as a simplified eye model. From this shape, the offset angle (Versatzwinkel) that occurs during integration can be derived, since the iris disk of the eye can be used to determine the original gaze direction. This offset θ0, φ0 is then further used to compensate for the initial angle estimation error that occurs through integration. For this purpose, a block 44 representing means for learning a model from raw data and a block 46 representing means for estimating the gaze direction (offset) are provided.
[0088] Figure 8 The corresponding eye geometry 50 with an offset θ0 is shown. Based on the integration of the angular velocity, the offset θ0 is obtained, as shown according to Figure 5 Thus, an estimation of the eye shape can be achieved in eye-fixed coordinates.
[0089] According to one embodiment, the described approach is implemented using software. Glasses with more than two static laser sensors for each eye are suitable as hardware prerequisites. Such glasses contain a signal processing unit, for example a microcontroller. No camera or laser scanner is required to implement this function. According to one embodiment, the eye position is returned as a numerical value or is used to correct the projected retinal image depending on the eye position. Eye tracking also works properly if the glasses slip off.
[0090] According to the following figures, an algorithm for classifying laser measurements in eye applications and tracking applications which may encounter problems with slipping, as for example in the case of data glasses, is described.
[0091] When transforming the coordinates into head coordinates, a spherical sclera is assumed. Other models, such as a spheroid, are also conceivable, but they require more parameters to be estimated or known. However, the principle remains unchanged.
[0092] Fig. 9 A model of the eye 60 is shown. A laser beam is used to measure the distance and surface velocity of the eye. Figure 4 , showing the fixed coordinate y of the glasses g ,x g and the head-fixed coordinate x h ,y h In addition, the radius r of the retina is shown 视网膜 and the radius r of the sclera 巩膜 .
[0093] According to one embodiment, the problem of classifying the measured distance and surface velocity of the eye is solved with the help of static laser interferometry. This function can then be used as a building block in an eye tracker function.
[0094] Fig.10 An exemplary embodiment of a corresponding eye tracker function is shown. Here, only one possible design of a method 70 for determining the viewing direction of an eye or of a corresponding device is shown.
[0095] It is assumed that a laser sensor directed at the eye can provide the distance between the sensor and the surface intersection point (also referred to as intersection point for short) and the surface velocity at the focus in the form of a projection along the laser beam. Starting from these two signals, in step 72, a classification is made using a suitable classifier as to which part of the eye the laser intersects. Obvious classes and therefore parts of the eye are the sclera, iris and retina of the eye. According to different embodiments, other classes are added, such as eyelids or even error classes.
[0096] In step 74 , the position and orientation of the laser sensor is estimated using a suitable estimator, and in step 74 , the viewing direction is ascertained, for example likewise estimated using a suitable estimator.
[0097] The method 70 enables the use of an eye tracker based solely on a static laser sensor. Such a solution is significantly more streamlined than, for example, a camera-based system, which is reflected in lower current consumption, less weight and lower material costs. Such a system is advantageous compared to camera-based systems (Video-Okulographie-System, VOG), which typically have a much lower refresh rate, while eye tracking requires a high refresh rate of at least 250 Hz.
[0098] The described classification method enables Figures 1 to 9 An eye tracker system is described which, due to the self-calibration property, is robust against a slipping of a portable device arranged with a laser sensor.
[0099] According to one embodiment, the task of classifying the intersection area of the laser beam on the eye surface is solved with the aid of a simple solution.
[0100] 1. performed with a simple threshold classifier for spacing measurement, and
[0101] 2. Execution with the aid of an automatic solution with a threshold value for the difference in the distance signal, which indicates the distance between the sensor and the surface intersection point.
[0102] The first scenario requires an exact knowledge of the position and orientation of the laser relative to the eye in order to deduce the threshold parameters. Classification can then take place using "if-then" instructions (Wenn-Dann-Anweisung), for example
[0103] If d1 < d < d2, the laser is aligned with the sclera.
[0104] Such a scenario may be prone to slippage and may require time-consuming recalibration.
[0105] The second simple solution listed is based on an automaton that is based on the difference in spacing.
[0106] Fig.11 A corresponding deterministic automaton for the difference in spacing or distance is shown. Schematically, the iris 80, sclera 82, and retina 84 are shown as parts of the eye. Arrow 86 marks the start of the automaton.
[0107] According to one embodiment, the described scenario consists in using an edge detector (Flankendetektor) for the spacing difference signal, with which discontinuities are recognized when the laser transitions from one part of the eye to another. Although this scenario is more robust to slippage compared to the first scenario, it lacks the use of absolute distance information. In addition, both scenarios are deterministic and thus inherently unable to cope with uncertainty.
[0108] Therefore, according to one embodiment, a probabilistic scenario, namely a hidden Markov model (HMM), is used, in which the characteristics of the eye geometry are explicitly used.
[0109] Fig.12 According to one embodiment, such a hidden Markov model 90 is shown as a series of hidden states (sclera, iris, retina) evolving over time, where there is a laser measurement for each time step. States s(0), s(1), s(2) are shown for the sclera, states i(0), i(1), i(2) are shown for the iris, and states r(0), r(1), r(2) are shown for the retina.
[0110] The model uses transition probabilities for the development of the hidden states and emission probabilities for the actual measurements. All probabilities in the range from 0 to 1 are modeled with reference to a specific eye geometry, and during runtime, certain parameters related to the position of the glasses, which are a priori unknown, can be deduced.
[0111] Another advantage of this approach is that, although the solution is based on a theoretically created framework, the actual implementation can be done in a computationally efficient way, enabling real-time implementation on hardware with limited computing speed. This is consistent with the need for algorithms with low energy consumption.
[0112] The proposed classifier can be a part of an eye tracker such as Fig.11 For a detailed description of the measuring principle itself, reference is made to Figures 1 to 9 Description.
[0113] Assume that, for the laser sensor i aimed at the eye, Fig.10 As shown, the measured value d i and v i Available for use if used as a basis Fig.11 The input values of the method shown are the same.
[0114] The typical distance between the laser sensor and the surface intersection point of the laser beam on the eye is in the range of a few centimeters. The wavelength of the laser can be in the range of 800-1050 nm, typically in the range of 850-940 nm.
[0115] According to one embodiment, it is identified which part of the eye (sclera, iris, retina) the laser is aimed at. These markers are needed for further processing of the measurements and correspond to the hidden states z of the Hidden Markov Model, such as Fig.13 The probability of radiation in the range 0,…,1 is modeled as a mixture of Gaussian distributions.
[0116] Fig.13 Exemplary normal distributions for the corresponding conditional radiation probabilities are shown: A normal distribution 92 is associated with the iris, a further normal distribution 97 is associated with the sclera, and a further normal distribution 96 is associated with the retina.
[0117] Fig.14 A matrix 100 is shown in which the transition probability from a hidden state z(k) to a subsequent state z(k+1) is modeled as a constant matrix A(d).
[0118] In this case, the top row is assigned to the iris, the middle row to the sclera, and the bottom row to the retina. The left column is assigned to the transition to the sclera, the middle column to the transition to the iris, and the right column to the transition to the retina.
[0119] The unknown parameters of the hidden Markov model, i.e. the parameters of the radiation probability, are iteratively updated by means of the following update equations taking into account the last N data points:
[0120]
[0121] N i =λ i N′ i +γ(z i (T)) (12b)
[0122]
[0123] λ i is the forgetting factor in the range of 0,…,1, and N i is the number of valid points that support clustering. i represents the positive weight of the cluster within the Gaussian mixture distribution, and μ i represents the mean value of the cluster, typically in the range of a few centimeters, and σ i represents its corresponding variance.
[0124] Based on these statistics, the following equation can evaluate the clustering z i The responsibility for a data point d(k), i.e., the most likely cluster to which the laser measurement belongs, can be determined:
[0125]
[0126] The scheme is a typical expectation-maximization scheme and can be summarized as follows:
[0127] Algorithm 1 Classification algorithm.
[0128] Initialization: N*, z0
[0129] For k=1,…,T execute
[0130] - Get the measurement d(k); Δd(k)
[0131] -Update the transition probability matrix A(Δd(k))
[0132] - Updated statistics (12)
[0133] - Arrange clusters in ascending order according to μi
[0134] - Classification of the measurement d(k) using (13), where d is confined to the interpolation region.
[0135] Finish
[0136] Advantageously, the algorithm can be performed online and is therefore suitable for sequential data. In addition, the algorithm contains certain exponential forgetting properties and only considers the latest and therefore most relevant data points.
[0137] This property is important so that the classifier can adapt in case the glasses slip off.
[0138] Fig.15 A flow chart of a method 70 for determining a gaze direction of an eye 20 according to one embodiment is shown.
[0139] In step 110, the distance value (d i ) and speed value (v i ), and in step 76, the spacing value and the speed value are used to obtain a sight direction value (θ, Φ) representing the sight direction, such as has been described according to Fig.11 Just as described.
[0140] To this end, according to one embodiment, in step 72 the distance value (d i ) and speed value (v i ) to classify in order to obtain the classification value (l i ), the classification value can be used to obtain the sight direction value (θ, Φ) in step 76. To this end, according to one embodiment, the position value (s h,i ), and in step 76 the position value is used to obtain the sight direction value (θ, Φ).
[0141] According to one embodiment, at least three laser sensors are used for determining the absolute position (trilateration).
[0142] Optionally, in step 112, a reflected beam is received, which constitutes a component of the laser beam emitted by the laser sensor that is reflected or scattered at a part of the eye. In step 114, using fluctuations in the radiation intensity of the reflected beam, a distance value (d i ) and speed value (v i ).
[0143] Optionally, in step 116, the aforementioned laser beam is emitted at a constant wavelength, and in step 114, the velocity value (v i ).
[0144] Therefore, when the wavelength of the radiation generated by the laser is constant, the fluctuations in the radiation intensity or radiation power of the laser, caused by constructive and destructive interference of the radiation reflected from the moving object with the radiation back-coupled into the resonator of the corresponding laser sensor, are used to calculate the speed of the object, which is here the corresponding part of the eye.
[0145] According to one embodiment, in step 116, the laser beam is emitted in a different time interval not with a constant wavelength but with a variable wavelength, and in step 114, the distance value (d i ).
[0146] Thus, in the case of wavelength variations of the radiation generated by the laser, for example by laser current modulation, the fluctuations in the radiation intensity or radiation power of the laser due to constructive and destructive interference of radiation reflected from an object that does not move relative to the laser with radiation coupled back into the resonator can be used to calculate the distance to the object. For example, the calculation can be made from the frequency of the peak amplitude of the radiation power.
[0147] If both effects are used simultaneously, fluctuations in the radiation power at a first frequency are generated by the movement of the object and fluctuations in the radiation power at a second frequency are generated by a change in the wavelength of the laser radiation, so that a beat frequency is generated.
[0148] From the average value of the beat frequency, for example in the case of a triangular modulation of the laser current with a rise and fall of the laser current, the distance to the object can be determined. From the difference in the beat frequency in the case of a triangular modulation of the laser current, the direction of movement and the speed of the object can be determined. This mixed use of effects is the basis for a number of exemplary embodiments.
[0149] According to one embodiment, in optional step 118, when emitting laser beams at constant and varying wavelengths, the corresponding beat frequency is determined using the fluctuations in the radiation intensity of the reflected beam. In optional step 120, using the average value of the beat frequency, a movement direction value representing the direction of movement of the eye and a movement speed value representing the movement speed of the eye are determined. In this case, in step 76, using the movement direction value and the movement speed value of the eye, the gaze direction value (θ, Φ) can be determined.
[0150] The two effects mentioned can therefore be used together or separately. According to one embodiment, both effects are necessary, since in order to determine the position of the eye, not only a distance measurement but also velocity measurement data is required. As a further embodiment, the distance measurement can be switched on or off by switching the modulation of the laser on or off, for example, by means of a triangle. As a result, the velocity can be determined better or more accurately for the case in which the modulation is switched off.
Claims
1. A method (70) for determining the viewing direction of an eye (20) for data glasses (1), in, The method (70) comprises the following steps: A first distance value and a first speed value are read in, wherein the first distance value represents the distance between a first laser sensor (2) for emitting a first laser beam and a first surface intersection point of the laser beam on the eye (20), and wherein the first speed value represents the surface speed at the first surface intersection point, reading in a second distance value and a second speed value, wherein the second distance value represents the distance between a second laser sensor (3) for emitting a second laser beam and a second surface intersection point of the laser beam on the eye (20), and wherein the second speed value represents the surface speed at the second surface intersection point, reading in at least one third distance value and a third speed value, wherein the third distance value represents the distance between a third laser sensor (4) for emitting a third laser beam and a third surface intersection point of the laser beam on the eye (20), and wherein the third speed value represents the surface speed at the third surface intersection point; classifying (72) the distance value (di) and the speed value (vi) so as to obtain a first classification value, a second classification value, and a third classification value, wherein the first classification value shows the part of the eye (20) assigned to the first surface intersection point, the second classification value shows the part of the eye (20) assigned to the second surface intersection point, and the third classification value shows the part of the eye (20) assigned to the third surface intersection point; determining (74) a first absolute value representing a pose of the first laser sensor in head-fixed coordinates of a head-fixed coordinate system, a second absolute value representing a pose of the second laser sensor in the head-fixed coordinates, and a third absolute value representing a pose of the third laser sensor in the head-fixed coordinates using the spacing value and the classification value (li); Using the surface intersection and the relationship between the head-fixed coordinate system and the eyeglass coordinate system, determine (40) the angular velocity The angular velocity represents an eye surface velocity of the eye (20); Using the angular velocity Determine the sight direction with offset When using the sight direction with offset and for correcting the sight line direction with an offset The deviation (θ 0 ,φ 0 ), obtain (76) the sight direction value (θ, Φ) representing the sight direction.
2. The method (70) according to claim 1, comprising the step of determining parameters (θ, φ, ρ, t , θ ... x ,t y ,t z ).
3. The method (70) according to any one of the preceding claims, in, By the angular velocity The integral of 4. The method (70) according to any one of the preceding claims, in, In the case of using an eye surface model, the offset (θ 0 ,φ 0 ), the eye surface model is determined by the spacing value (di), the speed value (vi) and the surface intersection point.
5. The method (70) according to claim 4, comprising the step of learning the eye surface model continuously over time from the detected data points.
6. The method (70) according to claim 1, the method comprising the following steps (112): receiving a first reflected beam, a second reflected beam and a third reflected beam, the first reflected beam representing the reflection or scattering of the first laser beam at the first surface intersection, the second reflected beam representing the reflection or scattering of the second laser beam at the second surface intersection, the third reflected beam representing the reflection or scattering of the third laser beam at the third surface intersection, and the method comprising the following steps (114): determining the first spacing value and the first speed value using fluctuations in the radiation intensity of the first reflected beam, determining the second spacing value and the second speed value using fluctuations in the radiation intensity of the second reflected beam, and determining the third speed value using fluctuations in the radiation intensity of the third reflected beam.
7. The method (70) according to claim 6, comprising the steps (116) of emitting the first laser beam at a constant wavelength, emitting the second laser beam at a constant wavelength, and emitting the third laser beam at a constant wavelength, in, In the determining step (114), the first velocity value is determined using fluctuations in the radiation intensity of the first reflected beam during emission of the first laser beam at the constant wavelength, the second velocity value is determined using fluctuations in the radiation intensity of the second reflected beam during emission of the second laser beam at the constant wavelength, and the third velocity value is determined using fluctuations in the radiation intensity of the third reflected beam during emission of the third laser beam at the constant wavelength.
8. The method (70) according to claim 7, in, In the emitting step (116), the first laser beam is emitted at a changed wavelength during a further first time interval, the second laser beam is emitted at a changed wavelength during a further second time interval, and the third laser beam is emitted at a changed wavelength during a further third time interval, wherein in the determining step (114), during the emission of the first laser beam at the changed wavelength, the first spacing value is determined using fluctuations in the radiation intensity of the first reflected beam, during the emission of the second laser beam at the changed wavelength, the second spacing value is determined using fluctuations in the radiation intensity of the second reflected beam, and during the emission of the third laser beam at the changed wavelength, the third spacing value is determined using fluctuations in the radiation intensity of the third reflected beam.
9. The method (70) according to claim 8, comprising the steps (118) of determining a first beat frequency using fluctuations in the radiation intensity of the first reflected beam during emission of the first laser beam at the constant and the varied wavelength, determining a second beat frequency using fluctuations in the radiation intensity of the second reflected beam during emission of the second laser beam at the constant and the varied wavelength, and determining a third beat frequency using fluctuations in the radiation intensity of the third reflected beam during emission of the third laser beam at the constant and the varied wavelength, and comprising the steps (120) of determining a movement direction value and a movement speed value using average values of the beat frequencies, the movement direction value representing the movement direction of the eye (20) and the movement speed value representing the movement speed of the eye (20), in, In the determining step, the sight direction value (θ, Φ) is determined using a movement direction value and a movement speed value of the eye (20).
10. A device (1a) for determining the direction of sight of an eye (20) for data glasses (1), in, The device is designed to carry out and / or control the steps of a method (70) according to any of the preceding claims in corresponding units.
11. Data glasses, comprising at least three laser sensors for emitting a laser beam in the direction of an eye (20) and for receiving a reflected beam of the laser beam reflected at the eye (20), the data glasses comprising a device according to claim 6 for determining the direction of sight of the eye (20). 12 . A computer program configured to carry out and / or control the steps of the method ( 70 ) according to claim 1 .
13. A machine-readable storage medium on which the computer program according to claim 12 is stored.
